MID-TERM PERFORMANCE EVALUATION OF LONG-TERM ASSISTANCE AND SERVICES FOR RESEARCH (LASER) EVALUATION REPORT March 13, 2023 PHOTO: INDIANA UNIVERSITY AND THE LASER PROGRAM LINC LLC 1201 Connecticut Ave NW, Suite 200J, Washington, DC 200036 Email: info@linclocal.org Website: www.linclocal.org The LASER Mid-Term Performance Evaluation was conducted by Dr. Heather Huntington (PDRI DevLab at the University of Pennsylvania), Ms. Aleta Starosta (The Cloudburst Group), Dr. Miriam Counterman (The Cloudburst Group), and Ms. Noelle Wyman Roth (Duke University), with support from Patrick Lohmeyer (LINC LLC), Simon Conté (The Cloudburst Group), Makayla Baker (The Cloudburst Group), and Ryan Hatano (The Cloudburst Group). The evaluation team would like to thank the USAID staff, researchers, and implementing partners who shared their experiences and expertise with us. CONTENTS ABSTRACT 1 EXECUTIVE SUMMARY 2 1. INTRODUCTION 5 2. EVALUATION QUESTIONS AND METHODOLOGY 7 3. EQ1: WHAT ARE THE STRENGTHS AND WEAKNESSES OF THE LASER DESIGN? 11 4. EQ2: WHAT ARE THE CHALLENGES AND SUCCESSES OF LASER M/B/IO BUY-INS? 18 5. EQ3: TO WHAT EXTENT HAVE CSFA AND ERT BEEN EFFECTIVE? 23 6. EQ4: WHAT ARE THE MOST SIGNIFICANT EARLY RESULTS OF LASER? 27 7. EQ5: TO WHAT EXTENT HAVE ADAPTATIONS ADDRESSED CHALLENGES? 33 ANNEX 1. LIST OF EVALUATION QUESTIONS 38 ANNEX I1. KII SUMMARY TABLE 40 ANNEX III. LASER-PULSE NETWORK SURVEY DEMOGRAPHICS 41 ANNEX IV. LASER-PULSE USAID M/B/IO SURVEY DEMOGRAPHICS 43 ANNEX V. COMPARISON MECHANISM SELECTION CRITERIA 44 ANNEX VI. FIDELITY ANALYSIS 45 ANNEX VII. EVALUATION SCOPE OF WORK 46 ANNEX VIII. EVALUATION DESIGN REPORT 65 ANNEX IX. EVALUATION BRIEF 152 ACRONYMS AOR Agreement Officer’s Representative CDCS Country Development Cooperation Strategy COP Chief of Party CSFA Comprehensive Success Factor Analysis DCOP Deputy Chief of Party DDI Bureau for Development, Democracy, and Innovation EQ Evaluation question ERT Embedded Research Translation ET Evaluation team HEI Higher education institutions HIC High-income country IR Intermediate result ITR/R Innovation, Technology, and Research Hub/Research Division KII Key informant interview LASER Long-Term Assistance and Services for Research LASER-PULSE LASER Partners for University-Led Solutions Engine LMIC Low-/middle-income country M/B/IO Missions, Bureaus, and Independent Office MCSIE Multi-Country Study on Inclusive Education QA/QC Quality assurance/quality control R4D Research for Development RFA Request for application RFP Request for proposal RTAC The Research Technical Assistance Center SLA Systems-level analysis ToC Theory of change USAID United States Agency for International Development 1 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV ABSTRACT This mid-term performance evaluation report on the USAID Long-Term Assistance and Services for Research (LASER) award presents findings on how LASER has been used in practice, identifying the program’s strengths and weaknesses, and addresses how USAID could improve future program designs. It also recommends actionable changes that USAID and the implementing partner, Purdue University, can make in the remaining years of implementation to ensure that the program meets its goals and objectives. The mixed-methods evaluation employed two quantitative surveys with a total of 303 respondents and 61 qualitative interviews with seven stakeholder groups, as well as a desk review, to answer the evaluation questions. USAID commissioned the evaluation of LASER in September 2021. LINC, together with its partners, The Cloudburst Group, PDRI DevLab at the University of Pennsylvania (formally DevLab@Duke), and Duke University, conducted the 19-month evaluation. The evaluation found that LASER has an overly ambitious design that has made it difficult for the implementing partner to successfully meet the program’s goals and objectives. Early program and policy results are limited despite a high volume of research products, and stakeholders reported management challenges. LASER has made numerous adaptations in response to these challenges and the quality of the work has improved over time. To meet the award objectives for the remainder of the program, the evaluation team recommends a focus on (1) producing fewer outputs but increasing the policy and program impacts of what is produced and (2) increasing opportunities for low-/middle-income country (LMIC) researcher engagement in each stage of buy-in. The evaluation team also recommends that future iterations of the program be streamlined to focus on research translation and capacity-strengthening for LMIC academic institutions and researchers, two areas where the LASER program excels. 2 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV EXECUTIVE SUMMARY The United States Agency for International Development’s (USAID) Innovation, Technology, and Research Hub/Research Division (ITR/R) launched the Long-Term Assistance and Services for Research (LASER) program in 2018 to improve development outcomes through research engagement. LASER’s purpose is to support international and U.S.-based higher education institutions (HEIs) and networks to improve development research opportunities, evidence generation, and uptake by development actors and policy makers. USAID commissioned a mid-term performance evaluation of LASER to take place from September 2021 to April 2023 to improve USAID’s understanding of LASER’s strengths and weaknesses and to gain a deeper understanding of the LASER research network model to improve current and future programming implemented by HEIs and other development practitioners. LINC, together with its partners The Cloudburst Group, PDRI DevLab at the University of Pennsylvania, and Duke University, conducted the 19-month evaluation. METHODOLOGY The evaluation team employed a mixed￾methods approach to allow for triangulation of findings using data from multiple sources and to ensure rich, comprehensive data that thoroughly explore the research questions. This is not a causal analysis of outcomes and impacts and the evaluation team cannot attribute any changes specifically to LASER. Exhibit 1 shows the qualitative, quantitative, and secondary data sources included in the methodology. The research questions were: 1. What are the strengths and weaknesses of the LASER design? 2. What are the challenges and successes of LASER Mission, Bureau, and Independent Office (M/B/IO) buy-ins, particularly in terms of scope development, Mission engagement, management, and collaboration with academic partners? 3. To what extent have Comprehensive Success Factor Analysis (CSFA) and Embedded Research Translation (ERT) been effective in helping stakeholders build relationships with USAID and each other, generate locally relevant research, and set the foundation for policy uptake? 4. What are the most significant early results of LASER? How have research results differed depending on whether activities were core-funded or buy-in investments? 5. To what extent have the adaptations made by the LASER program to date, particularly those made in response to COVID-19, addressed challenges and increased the likelihood of meeting program goals and objectives? FINDINGS Exhibit 1. LASER mid-term evaluation data sources 3 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV LASER’s greatest value add lies in capacity-strengthening for researchers and research translation. There is appetite for research translation from both USAID and researchers and LASER is well￾positioned to contribute to best practices throughout the Agency. The resources LASER has developed around ERT are high quality and have the potential to be shared and adapted across the Agency. The inclusion of a dedicated research translation partner is the most unique element of ERT and sets it apart from the dissemination and utilization tools most researchers and USAID awards currently employ. The emphasis on strengthening the capacity of researchers, especially LMIC and women researchers, is also a strength of LASER. RFA researchers received support to refine their research proposals, navigate the USAID compliance process, adapt their research because of COVID-19, and translate their research products into policy-friendly formats. LASER has been successful at engaging and growing institutions, networks, and researchers. Institutional and individual capacity-strengthening of funding recipients has successfully helped researchers and universities partner with USAID for the first time. LASER has excelled at applying adaptive management to address challenges. LASER’s adaptive management processes were praised by USAID M/B/IO staff who engaged with LASER across both core- and buy-in-funded activities. Key informants made it clear that LASER struggled with management in the first two years of the activity, and M/B/IOs reported negative experiences with LASER management, the quality of the research products, and the application of research translation. However, through adaptive management practices—including new LASER leadership at Purdue (which began in 2021), stronger quality assurance/quality control (QA/QC) protocols, the evolution of research translation from Comprehensive Success Analysis to CSFA and then CSFA/systems-level analysis (SLA), as well as adaptations to ERT and improvements to the request for application (RFA) process—LASER’s performance is widely seen to have improved over time. LASER has produced a vast number of research outputs, but few policy or program outcomes. At the end of Year 4 (September 2022), LASER had produced 101 research products, engaged 604 development actors, translated 92 research products, developed 119 research translation materials, held 96 convenings with decision makers to disseminate research, and trained 185 development actors on research translation. However, this high number of outputs has not translated into a large number of policy or program outcomes. The LASER team identified nine program or policy changes, which include outputs like dissemination events and outcomes such as one government-level policy change. This may be because of delays in research implementation and uneven application of ERT. LASER’s ambitious design has hindered implementation. The large number of components that make up the LASER program (seven in total) have contributed to significant implementation challenges for some activities. Some components, like the RFA and research for development (R4D) convenings, were delayed in the first year. Other components, like the LASER network, lacked the resources to be implemented to their fullest extent. Some elements, such as buy￾ins, contradicted some program goals while accomplishing others. The program also struggled to maintain the quality of the research across the various components. Many of the struggles the program faced are more likely to stem from flaws in the design and limitations in what USAID as an agency is built to accomplish than shortcomings on the part of the implementing partner. While the components of the theory of change have supporting evidence, the ultimate outcome has not been realized. 4 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV The evaluation team found some evidence to support all three of the hypotheses that lead to the theory of change. However, the evaluation team did not find evidence that LASER has led to useful policies, products, and practices, which the theory of change would predict. This suggests that while closer collaboration did occur, that alone was not enough to overcome the barriers to implementing evidence￾based solutions to development challenges. The theory of change should be modified to include a greater emphasis on elements that increase the depth and quality of research, including prospective research, that are more likely to promote meaningful policy and program changes. LASER Management, particularly of buy-ins, has been a significant challenge, and has affected the quality of LASER research products. The decentralized LASER consortium was not set up to effectively manage a USAID award, and USAID, researchers, and the LASER management team reported confusion about roles, responsibilities, quality control, budgets, and timelines. The buy-in process and research product quality have improved over time as the LASER implementing partner developed standard operating procedures and adapted its research processes, including quicker startup, stronger QA/QC of research products, and new ERT processes. Some of these management challenges are inherent to awarding funds to an academic institution rather than a traditional USAID implementing partner. KEY RECOMMENDATIONS The evaluation team made a total of 16 recommendations for the LASER implementing partner and USAID/ITR/R across the five evaluation questions. The highest priority recommendations are presented below. RECOMMENDATIONS FOR THE IMPLEMENTING PARTNER FOR THE REMAINDER OF THE PROGRAM  Focus on producing fewer outputs, but increasing the policy and program impacts of what is produced, including prospective research when possible.  Prioritize increasing the number of buy-ins led by local universities and incorporate LMIC researchers into the co-creation process during scope development.  Continue to refine ERT and develop training programs and protocols for implementation, including developing a “menu of services” for buy-ins to ensure each M/B/IO has a similar translation experience.  If LASER holds future R4D convenings, it should hold the events after awards have happened and research has begun and focus on ERT rather than CSFA/SLA.  Find ways to share lessons learned on HEI capacity-strengthening, especially LMIC HEI capacity￾strengthening, across the Agency. RECOMMENDATIONS FOR USAID/ITR/R FOR FUTURE PROGRAM DESIGNS  The LASER theory of change should be modified to reflect an emphasis on quality of research outputs rather than the volume of research produced and translated, as well as an emphasis on increasing demand for translated research by USAID and other stakeholders, including the private sector. 5 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV  Future iterations of LASER should be narrower in scope and focus on the program’s strengths to allow the implementing partner a better opportunity to accomplish the program objectives. These strengths include research translation and LMIC researcher capacity-strengthening. Eliminating or reducing buy-ins, which by definition are prescriptive rather than local demand-driven and are often retrospective evaluations rather than higher quality prospective research, would aid in this goal. USAID should prioritize a strong centralized management structure when designing future procurements, and explore prime awardees outside of HEIs for more efficient management. 1. INTRODUCTION 1.1 PURPOSE USAID’s Innovation, Technology, and Research Hub launched the LASER program in 2018 to improve development outcomes through research engagement. Based on a gap identified by researchers and practitioners, LASER’s purpose is to support international and U.S.-based HEIs and networks to improve development research opportunities, evidence generation, and uptake by development actors and policy makers. LASER aims to achieve its purpose by identifying new research questions, funding research activities, translating research results into development impact, and building the capacity of local HEIs and researchers. USAID commissioned a mid-term performance evaluation of LASER in September 2021. LINC, together with its partners The Cloudburst Group, PDRI DevLab at the University of Pennsylvania, and Duke University, conducted the 19-month evaluation. This document presents the findings from the evaluation. The evaluation objectives are: 1. To improve the USAID Bureau for Development, Democracy, and Innovation (DDI) ITR/R’s understanding of LASER’s strengths and weaknesses in terms of design, partnership choices, implementing mechanism, benefits to stakeholders, and Mission engagement. 2. To gain a deeper understanding of the strengths and weaknesses of the LASER research network model to improve current and future programming implemented by HEIs. The mid-term performance evaluation presents findings on how LASER has been used in practice, identifies its strengths and weaknesses, and addresses what USAID should have done differently in the design of LASER. It also answers research questions about LASER’s outputs and outcomes to date and provides actionable recommendations for changes that USAID and the implementing partner can make in the remaining years of implementation to ensure that the project meets its goals and objectives and for future iterations of the mechanism. 1.2 LASER BACKGROUND LASER funds researchers to explore development questions that span regions, sectors, and universities. It supports USAID M/B/IOs through core research grants and R4D convenings and through buy-in agreements to conduct research and deploy evidence-based solutions that benefit M/B/IO needs.1 Launched in 2018, LASER began as a five-year cooperative agreement that was extended in 2022 to a 1 A buy-in is when a USAID M/B/IO uses its own funding to “buy” the services of a project or activity originally procured/contracted by another USAID office. 6 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 10-year award. LASER is implemented by Purdue University with four additional partners: Catholic Relief Services, Indiana University, Makerere University in Uganda, and the University of Notre Dame. LASER’s theory of change is that “closer collaboration between academic researchers, development practitioners, policymakers, and donors results in new research that is readily translated into useful policies, products, and practices as evidence-based solutions to development challenges.” Its objective and four intermediate results (IRs) are shown in the results framework in Error! Reference source not found.. The primary categories of LASER’s services are: 1. LASER RESEARCH GRANTS (RFAs): Grants of up to $250,000 for researcher-practitioner teams to generate research-driven solutions to field-sourced challenges that translate into policy and practice. These are competitively awarded through an RFA process. They are generally funded by USAID/ITR/R funds to support the core activities of the award. 2. R4D CONVENINGS: The R4D convenings were originally planned as semi-annual three-day convenings of regional researchers, national and local government officials, non-governmental organization and private sector partners, and USAID representatives, but have taken on alternative forms in response to the COVID-19 pandemic. In this forum, LASER applies CSFA, a Purdue￾developed methodology, to screen and mine a large volume of documentation to discover development priorities across a large number of stakeholders and iteratively refine that into priority research topics and, eventually, research questions. 3. CSFA/SLA: CSFA is an innovative systems-level approach for gathering information from diverse stakeholders for defining priority research areas. LASER sometimes uses the term “comprehensive issue analysis.” In 2021, CSFA evolved into SLA, which is a more portable, streamlined version that provides researchers and practitioners with tools to identify “structural gaps in knowledge in their work.”2 4. ERT: ERT is an iterative co-design process among academics, practitioners, and other stakeholders in which research is intentionally applied to a development challenge. This approach has four core pillars: partnership, process, product, and dissemination. 5. RESEARCH CAPACITY-STRENGTHENING: LASER works directly with the RFA awardees, buy￾in teams, and its wider network of researchers and development actors to strengthen and sustain their capacity for development research and research translation through training, webinars, and mentorship. 2 LASER Year 3 Bi-Annual Report, pg. 6. Exhibit 2. LASER theory of change 7 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV 6. SUSTAINABLE NETWORK: The LASER Partners for University-Led Solutions Engine (LASER￾PULSE) network convenes 3,013 researchers, practitioners, and policy makers in 74 countries, including all USAID partner countries.3 7. USAID BUY-INS: LASER provides specific collaboration with USAID M/B/IOs through buy-ins, which are modifications to the LASER cooperative agreement with USAID/ITR/R that enable expanded M/B/IO engagement and funding.4 The LASER program has just completed its fourth year of implementation and received a five-year, no￾cost extension in 2022 to continue work through July 2028. Work during this time period will focus primarily on buy-in activities. 2. EVALUATION QUESTIONS AND METHODOLOGY This evaluation reviews the strengths and weaknesses of LASER’s services and approach—including an analysis of LASER research grants, R4D convenings, research capacity-strengthening, and USAID buy￾ins—with special attention to differences in the performance of core-funded or buy-in investments, as well as how the LASER program adapted to the COVID-19 environment. 2.1 Evaluation Questions In consultation with USAID, the evaluation team identified the following five core evaluation questions (EQs) that guided the evaluation. For a full list of questions and sub-questions, refer to Annex 1. 1. What are the strengths and weaknesses of the LASER design? 2. What are the challenges and successes of LASER M/B/IO buy-ins, particularly in terms of scope development, Mission engagement, management, and collaboration with academic partners? 3. To what extent have CSFA and ERT been effective in helping stakeholders to build relationships with USAID and each other, generate locally relevant research, and set the foundation for policy uptake? 4. What are the most significant early results of LASER? How have research results differed depending on whether activities were core-funded or buy-in investments? 5. To what extent have the adaptations made by the LASER program to date, particularly those made in response to COVID-19, addressed challenges and increased the likelihood of meeting program goals and objectives? 3 LASER-PULSE network numbers as of January 3, 2023. The current number is on the network website. 4 These buy-ins must be within the scope of the LASER cooperative agreement and are co-created between LASER and the interested USAID entity, as approved by the LASER Agreement Officer’s Representative (AOR) to provide specific research and other activity support. 8 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 2.2 METHODOLOGY The evaluation team employed a mixed￾methods approach to allow for triangulation of findings using data from multiple sources and to ensure rich, comprehensive data that thoroughly explore the research questions. This is not a causal analysis of outcomes and impacts, and the evaluation team cannot attribute any changes specifically to LASER. Exhibit 3 shows the qualitative, quantitative, and secondary data sources included in the methodology. The evaluation team collected data in two rounds to maximize time for research to be completed and outcomes to mature. The first round of data collection included quantitative surveys and qualitative interviews with the LASER leadership team and key informant interviews (KIIs) associated with (a) the East Africa and Colombia R4D convenings and RFAs, (b) the Tusome Case Study of Kenya’s Early Grade Reading Program, and (c) Support to Traditional Cultural Practices in Northern Iraq buy-in case studies, which took place in April and May of 2022. The second round of data collection included KIIs associated with the Global RFA round, Ethiopia RFA, and post-award convening, as well as two additional buy-in case studies: the Multi￾Country Study on Inclusive Education (MCSIE) and the Trafficking in Persons in South Africa study. The evaluation team adapted the evaluation design to include two additional RFA case studies, the Global Round and Ethiopia, and an additional R4D convening in Ethiopia to capture adaptations to RFA, CSFA/SLA, and ERT. All interviews for the second round took place in September and October 2022. A majority of the 24 USAID M/B/IO survey respondents were involved in buy-ins (75 percent), while fewer had participated in either an RFA (25 percent) or an R4D convening (16.7 percent). The LASER￾PULSE Network survey respondents were largely researchers (69.3 percent), with some development professionals (19.5 percent) and few donor representatives (3.5 percent). Additional details of the survey demographics are available in Annexes III and IV. Exhibit 4. USAID M/B/IO respondents (n=24) Exhibit 3. Evaluation data sources 9 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV Exhibit 5. LASER-PULSE Network survey respondents by role (n=235) Quantitative data were analyzed with descriptive statistics to answer the research questions. The evaluation team generated descriptive statistics from the survey results to show the distribution of responses across questions. The evaluation team compared and contrasted the response tabulations with the results of qualitative data collection and, when possible, disaggregated responses by gender and LMIC and high-income country (HIC) respondents. As discussed in Section 2.3, the evaluation team planned to use statistical tests to examine significant differences between these groups; however, it was not feasible to apply this method because of lower than expected response rates. The evaluation team explored qualitative data sources through content analysis and thematic analysis organized by research questions. The evaluation team triangulated findings from the qualitative analysis with the findings from other data sources. Through thematic analysis organized by research question, the evaluation team identified instances where findings coincide and instances where findings diverge; divergences are noted by type of respondent or by type of LASER program. 2.3 LIMITATIONS LOW QUALITATIVE AND QUANTITATIVE RESPONSE RATES 10 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV The original study design called for 65 KIIs with researchers, USAID M/B/IOs, boundary partners, and the LASER implementing partner. While the team was able to get representation from researchers and USAID for all the buy-in, RFA, and R4D case studies, the evaluation team was only able to interview one boundary partner, despite the LASER team, USAID, and the evaluation team reaching out to them. The evaluation team did speak with one translation partner from the USAID/Ethiopia RFA round who attended the R4D. The lack of boundary partner representation makes it more challenging to speak to broader policy or program impacts of research translation, or the perception of CSFA/SLA and ERT from stakeholders who are less involved in the process. Researchers were the easiest group to speak with, which is unsurprising since they received funding from LASER. The major barriers to participation included staff turnover (particularly among USAID M/B/IO officers), participants not remembering their interactions with LASER, and lack of time to participate. To reach respondents, the LASER implementing partner or USAID facilitated email introductions to the evaluation and the evaluation team, and the evaluation team followed up directly to schedule discussions and send 1–2 reminders. If a respondent group was very difficult to confirm, USAID or LASER would also send a reminder. To increase the total number of KIIs and capture later program activities, the evaluation team added two additional RFA rounds, Ethiopia and Global, to the sample, as well as the post-award R4D in Ethiopia. The quantitative survey also had lower-than-expected response rates. For the USAID M/B/IO survey, the evaluation team anticipated a 30 percent response rate, depending on the role of the individual; however, the final response rate was approximately 12 percent. For the network survey, the evaluation team anticipated a response rate of 16 percent and the actual response rate was 10 percent. Details about the survey demographics can be found in Annex III. To improve the low response rate, USAID shared the M/B/IO survey with USAID staff and LASER shared the researcher survey with its network. The LASER mid-term evaluation Contracting Officer’s Representative and the evaluation team sent at least two reminders to participate in the survey, and USAID staff who had participated in LASER activities sent personalized requests. The low response rate across instruments leaves the evaluation vulnerable to nonresponse bias, as those who made time to speak with the evaluation team may have had more positive or more negative experiences with LASER than the typical respondent. LONG RECALL PERIODS FOR EARLY LASER ACTIVITIES Respondents involved in the East Africa and Colombia RFA and R4D case studies had recall periods of two to three years, and respondents struggled to remember details of their involvement. USAID M/B/IO respondents, in particular, may not have had any additional interaction with LASER since the events in 2019. The evaluation team added new interviews about the Ethiopia RFA and R4D case study and the Global RFA in the second round of data collection to capture more recent experiences with LASER. LACK OF EARLY RESULTS AND EVOLUTION OF CORE FUNDING COMPONENTS MADE SOME PLANNED EVALUATION ELEMENTS DIFFICULT TO CAPTURE The evaluation team designed the evaluation around speaking to program and policy impacts of research findings. The case study and site selection therefore skew heavily to Year 1 and 2 LASER activities, under the assumption that the completed research would have the greatest ability to trace the impacts of research translation across USAID M/B/IOs, academic researchers, and boundary partners. However, LASER’s slow start-up, followed by the COVID-19 pandemic, left early research with limited or no 11 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV results when the evaluation team collected data in the spring of 2022. The lack of concrete changes also limited the evaluation team’s ability to employ Most Significant Change methodology. When asked, respondents were unable to point to changes in policies or program design as a result of the LASER program.5 The evolution of ERT presented another challenge for the evaluation design. The evaluation team was tasked with examining the differences between “light” and “deep” ERT, and worked with the LASER team to identify buy-in case studies that employed light or deep ERT. However, in practice, USAID and researchers were not able to differentiate between light and deep ERT, and there was no standard implementation of either that would allow for easy comparison, and no standard definition of ERT until Year 3. The evaluation team therefore ultimately defined ERT broadly to mean any research translation efforts and examined changes from early research to later research. 3. EQ1: WHAT ARE THE STRENGTHS AND WEAKNESSES OF THE LASER DESIGN? 1.1. What evidence exists that the development hypothesis underlying LASER’s theory of change is valid in practice? How might the theory of change be adapted to account for early evidence on these development hypotheses? 1.2. What elements of the design of LASER have contributed to versus hindered implementation? For example, to what extent is combining buy-ins for USAID M/B/IOs and more open-ended core funding under one cooperative agreement hindering or helping implementation? How did integrating attention to gender sensitivities and/or local stakeholders hinder or support implementation? 3.1 FINDINGS The LASER theory of change states: Closer collaboration between academic researchers, development practitioners, policymakers, and donors results in new research that is readily translated into useful policies, products, and practices as evidence-based solutions to development challenges. According to the evaluation scope of work (see Annex VII), the LASER theory of change is based on three primary hypotheses: (1) involving diverse partners in research question identification leads to contextually relevant studies; (2) building the capacity of researchers to identify research needs and find appropriate partners will result in stronger research proposals; and (3) providing researchers with a strong and active network of peers to collaborate with will give them a body of knowledge and resources to leverage for submitting strong research plans/proposals and generating research that is relevant for development. The evaluation team found some evidence, detailed below, to support all three of these hypotheses that lead to the theory of change. However, the evaluation team did not find evidence that LASER has led to 5 "Program or policy changes" are defined as alterations to institutional practices and procedures that are attributable, in part or in entirety, to information transmitted through lab-funded research efforts. These changes can occur in a variety of development-related institutions, including: local, national, or regional governments; non-state actors; and USAID or other development actors. This is the same definition LASER uses in its Performance Indicator Reference Sheet for Indicator L3.S.2_in1, “Number of program or policy changes made by private sector, public sector, or other development actors.” 12 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV useful policies, products, and practices, which the theory of change would predict. This suggests that while closer collaboration did occur, that alone was not enough to overcome the various barriers to implementing evidence-based solutions to development challenges. The evaluation team also found that the buy-in components of LASER sometimes worked against achieving the theory of change. This is because of the prescriptive nature of the scopes of work, developed by Missions with little room for local input or engagement, and the tendency toward retrospective evaluations, which are useful for USAID programming changes but are unlikely to lead to greater program or policy changes with a wider group of boundary partners. Diverse partners lead to contextually relevant studies. There is some evidence that bringing diverse partners from academia, the private sector, civil society, and government together to develop research questions during the R4D process and to translate research produced through RFAs helped make the research more contextually relevant. Half of the RFA key informants (n=4) identified the translation partner element of the RFA process as critical to the relevance and sustainability of their work, which speaks to how involving diverse partners leads to contextually relevant studies. However, although the R4D convenings in East Africa and Colombia also involved diverse partners, there was less evidence that this led to contextually relevant research questions.6 Capacity-strengthening supports stronger research proposals. The LASER team has provided capacity-strengthening and worked with researchers and institutions to identify research needs and find appropriate partners. This has translated into a body of knowledge and resources to support the creation of strong proposals and development-relevant research. This includes working directly with LMIC universities to build understanding of the rules around accepting U.S. Government financing and the development of a course on “21st Century Development Research Leadership for LMIC Higher Education Institutions” that was taken by 52 participants. It also includes the mentoring provided to researchers in buy-ins and RFAs to help navigate USAID compliance issues, translate their work to policy audiences, and provide QA/QC of their final products. A strong peer network promotes stronger proposals and research products. The evaluation found some evidence that providing researchers with a strong and active network of peers to collaborate with will support stronger proposals and better development-relevant research. However, it is difficult to fully validate this finding given the current state of the LASER network. An RFA researcher who used network resources to solve a research problem expressed appreciation for the breadth and depth of expertise covered by the LASER-PULSE network, including communication and ERT. However, this is the only concrete example from the available evaluation data. To truly achieve this objective, the LASER-PULSE network would require greater investment to transform it from a database of researchers to a true “development bazaar,” as a LASER implementing partner member called his vision for the network. To sustain a community of practice, the implementing partner told the evaluation team, would require a full-time program with a significant level of investment, something that will not be possible under the current no-cost extension. ELEMENTS OF THE LASER DESIGN THAT HELP AND HINDER IMPLEMENTATION 6 For more details on the R4D convenings, see Section 5 on EQ 3. 13 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV Error! Reference source not found. shows the major LASER design elements that help and hinder implementation. Each element is discussed below. Elements that Help Implementation Elements that facilitate implementation include effective adaptive management, an emphasis on research capacity-strengthening (especially of women and LMIC researchers), the co-creation process for RFAs and buy-ins, an emphasis on research translation, and the ability to use the LASER research network to recruit qualified researchers for buy-ins. USAID stakeholders who engaged with LASER across both core and buy-in funding activities widely praised LASER’s adaptive management processes. Key informants made clear that LASER struggled with management in the first two years of the award, and M/B/IOs reported many negative experiences, particularly related to buy-in research. These challenges are detailed in Section 4, EQ 2 and Section 7, EQ 5. However, through adaptive management practices—including new LASER leadership at Purdue, stronger QA/QC protocols, the evolution of research translation from Comprehensive Success Analysis to CSFA and then CSFA/SLA, as well adaptations to ERT and the improvements to the RFA solicitation process—LASER’s performance is widely seen to have improved over time. More details about the adaptive management changes LASER made can be found in Section 7 on EQ 5. There was evidence from both the Global and Ethiopia RFA case studies that stakeholder engagement through the co-creation and ERT processes improved study design and implementation. Specifically, respondents provided concrete examples of how co-creating the study with research translation partners adjusted their study design, including working in different locations and/or with different populations. In addition, respondents felt that the ERT process required them to think about dissemination throughout the lifecycle of the project. Global RFA respondents described how the co￾creation process helped the team get more specific when developing their research question and dissemination plan. One respondent explained that this process “helped academics get an eye on reality.” One RFA respondent indicated that they would continue using ERT going forward and that it complements tools already in use. One respondent felt that “[ERT] will strengthen research work within the community”; another said, “ERT is important to effectively disseminate our research products.” This speaks to the value add of the ERT process. The emphasis on research translation is an important component of the LASER design. Both USAID program managers and the IP see this aspect as critical to support the program to reach its objectives; however, the implementing partner did not successfully deploy a uniform methodology for research translation in the first three years. The process of research translation that did occur was ad hoc and Exhibit 6. Elements of LASER design that help and hinder implementation 14 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV varied greatly by research project.7 As implementation improves, the unique elements of ERT and the high demand for translated research will help LASER reach its objectives.8 Strengthening the capacity of researchers, especially LMIC and women researchers, was unique to LASER among the three comparison mechanisms reviewed as part of the evaluation. RFA researchers spoke to receiving support refining their research proposals, navigating the USAID compliance process, adapting their research because of the COVID-19 pandemic, and translating their research products into policy-friendly formats. These types of capacity-strengthening efforts were all highly personalized and time-intensive. Forty-three percent of survey respondents used capacity￾strengthening tools from the website, including training and resources. “The way we were looking at research before is not the way we are doing it now. We are able to collaborate with universities outside of Kenya, international university partners are not looking down on us, they value our contributions. We also involve other stakeholders in the private sector, the CSO [civil society organization] sectors; the kinds of stakeholders coming to their workshops are different than they would usually seek, and see that the utilization of the research will be higher.” – LMIC buy-in researcher According to USAID R4D participants, this emphasis on researcher capacity-strengthening helped bring USAID staff in contact with researchers they would not otherwise have been able to work with. Requiring all researchers to take a gender training and incorporate a gender perspective into all research topics helped make the research products more inclusive.9 The LASER-PULSE network is another design element that is very helpful for implementation. In interviews with the LASER team, the evaluation team was told that early in the program, USAID M/B/IOs asked for a list of researchers in USAID partner countries who do research on particular sectors. Without the LASER network, there would be no way to meet this need. As a member of the implementing partner management team described it, “The notion that a database can be created voluntarily is phenomenal,” particularly one with such high coverage of country distribution, sectors, and engagement with LMICs. The LASER network has not evolved into a community of practice as some LASER team members hoped, but the existence of the database meets an important need for widely disseminating funding opportunities for core funding and for buy-in research. ELEMENTS THAT HINDER IMPLEMENTATION An element of LASER that hinders its implementation is the award’s mandate to serve universities and USAID simultaneously. USAID/ITR/R initially designed LASER as a follow-on to the Higher Education Solutions Network awards to address university desires for longer commitments and USAID’s need for quick turnaround of deliverables. LASER was designed to be dynamic and to work across all regions and sectors, to develop a network of researchers, and to have university engagement, but also to allow for buy-ins that allow USAID M/B/IOs to have access to the network of researchers, with an end goal of research generation and translation. USAID’s goal in designing LASER was to provide all the benefits of both the Research Technical Assistance Center (RTAC), which provides rapid, actionable, and on-demand research for USAID M/B/IOs through a global network of academic researchers, and the Science, Technology, Innovation, 7 The evaluation team recognizes that the products from research translation should be customized for each research project, so the ultimate translation should look different. However, there should be a uniform methodology underlying ERT so that the process of translation is similar across projects. For example, a translation plan should be a component of each research design, and researchers and Missions should receive similar information and training on ERT. 8 These findings are discussed further in Section 5 under EQ 3. 9 These findings are discussed further in Section 6 under EQ 4. 15 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV and Partnership (STIP) Annual Program Statement (APS), which allows USAID M/B/IOs to collaborate with global HEIs to conduct larger value, longer term, forward-thinking scientific research and innovation. LASER was meant to be fast and responsive while also producing exceptionally high-quality products that would improve programs and policy. This ambitious scope proved difficult to execute. According to an interview with a LASER program manager, USAID/ITR/R designed the program to identify problems and address gaps through core funding, but restrictions on Mission concurrence would make this difficult: “The function of the [LASER] design is limited by USAID… even the core funds are limited by what USAID Missions and OUs [Operating Units] are willing to approve. So [the LASER team] had to pivot very hard in the beginning to be more responsive to USAID than what the design had intended.” During this pivot, LASER struggled to create a management structure that could meet the needs of both researchers and USAID, as described in more detail in Section 4 on EQ 2. The focus on buy-ins forced LASER to focus on retrospective program evaluations, rather than prospective research that focused on forward-thinking and “bigger picture” questions envisioned in the original design. Doing retrospective mid-term or final program evaluations have the potential to influence USAID program design and the adaptive management of existing programs. However, these evaluations are unlikely to represent the gold standard for innovative scientific research envisioned in the LASER objective. “Everyone [at USAID] comes to LASER for evaluations, so instead of looking forward with research, we are going backwards with evaluations… if we could be forward-looking, it would be so much more dynamic.”– USAID program manager Another major hindrance to LASER implementation was the decentralized consortium design. LASER key personnel are spread across the five-university consortium, and decision-making is decentralized. According to a LASER program manager, “previous meetings had over 50 people, including representatives from each consortium member. This was unwieldy, and restricted the [Chief of Party’s] ability to move.” This was later streamlined, and in 2021, LASER brought in a new program manager. The implementing partner team shared that having such a decentralized structure makes it challenging to shift resources and adapt to shifting priorities. A research institution model more similar to the RTAC model, implemented by NORC at the University of Chicago, would have made implementing LASER easier. RTAC works better in part because it is implemented by an experienced USAID contractor with all the processes in place to manage USAID funding, as well as full-time key personnel who all work for the prime organization. Purdue has recently established the Purdue Applied Research Institute LLC, a nonprofit arm of the university, and LASER will be managed out of the Purdue Applied Research Institute going forward. This may reduce some of the friction inherent in contracts with universities and assist with centralizing the management of the program, which would improve the likelihood of LASER meeting its program objectives. ELEMENTS THAT BOTH HELP AND HINDER Two elements contribute in some ways to LASER implementation but hinder in others. The first of these is the combined buy-in and core funding model. Discussed in greater detail in Section 4, EQ 2, buy-ins were a significant management challenge for the LASER implementing partner, and the number of buy-ins in Year 1 likely contributed to the slow rollout of other core-funded activities. The skills and personnel needed to manage a rapid and flexible buy-in mechanism are different from those needed to conduct the core funding activities, and combining them impeded the efforts of both. Buy-ins require the ability to be nimble, to quickly recruit and contract short-term consultants, to provide QA/QC, to manage a large number of independent projects simultaneously, and to adjust staffing up and down depending on demand, while a more traditional core-funded activity will have consistent staff and a reliable work plan and workflow. 16 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV The LASER implementing partner reported fewer average responses to a buy-in request for proposals (RFP) than an RFA, which it suggested stems from researchers being less interested in conducting research proposed by USAID than in conducting research they can propose themselves. Buy-ins might therefore be better served by research professionals in the development community who specialize in conducting policy-relevant research for USAID M/B/IOs, or a combination of academic and development researchers, than by academic researchers alone. There are, however, some important synergies between the core funding and buy-ins that may justify keeping the two combined, especially as buy-in management processes improve. The LASER implementing partner team spoke to how buy-ins provided an important testing ground for early co￾creation and ERT models, which in turn improved the core-funded activities. The LASER network, which was part of the core funding activities, was also an asset for USAID buy-in clients, and opened doors for USAID RFAs to reach thousands of researchers, many of them women and LMIC researchers. The second element that both contributes to and hinders LASER’s implementation is partnering with universities as prime award holders. Researchers and USAID LASER management cited several benefits to having universities as the implementers. These include access to world-class scholars and universities being best positioned to strengthen the capacity of other researchers and research institutions. HIC and LMIC researchers alike told the evaluation team that they liked working with a mechanism managed by an academic institution. Interviewees felt that this made things easier for them and that LASER could play a role in overcoming some of the challenges they faced with various points of conflict between their university and USAID systems. However, there are some challenges when USAID partners with universities, particularly as prime award holders. Large research universities typically have huge bureaucracies that are not designed to implement U.S. Government contracts or cooperative agreements, particularly related to international development. Universities typically do not have efficient processes in place to set up subcontracts with vendors like data collection firms, pay international vendors, or pay for international travel. There are important differences in administrative processes, such as challenges synchronizing budgets and thinking about billable hours, as well as bureaucratic challenges that interfere with rapid execution and streamlined communication. LASER faced many of these challenges in the first three years until it could build systems and hire staff geared toward implementing research for USAID. Working with universities as subcontractors would place the burden of navigating these bureaucratic challenges with the prime contractor, rather than USAID CORs/AORs. Contracting with the new Purdue Applied Research Institute may reduce these burdens. “The hard part of working with universities is the bureaucracy… Communication can be problematic. It is hard to get people in the university with USAID experience, who understand how the process works. This is particularly true if we work with universities around the world.” – LASER program manager Key informants also raised cultural differences between USAID and universities. Universities exist to generate knowledge, promote intellectual freedom, and train scholars. They are less in tune with fundamental elements of U.S. Government contracting work, such as timelines, QA/QC, and client communication compared to a traditional USAID implementing partner. “If we have a higher education institution as an implementer, we need to understand the university’s mission is not to implement USAID programing; it is to create knowledge for the greater good. These are sometimes aligned, but not always.” – LASER program manager 17 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV 3.2 CONCLUSIONS The LASER program strengths include collaborative and adaptive management, contributions to capacity development, and the stakeholder engagement facilitated by co-creation and ERT. LASER weaknesses include early management challenges within the consortium, a slow start-up that limited early results, and limited resources to create a more robust LASER network. The evaluation team found evidence to support each of the three major hypotheses that contribute to the theory of change, but the theory of change itself has yet to be proven. This suggests the theory of change may not be valid and could be modified to include components that would strengthen the depth and quality of the research produced, as well as demand for research by end users. However, these components should not be added on top of an implementation agenda that is already stretched too thin. Instead, elements of the LASER program that are not directly related to accomplishing the theory of change, such as the various online training sessions LASER produces, or that sometimes contradict the theory of change, such as buy-in research, should be eliminated from the scope. Elements of the LASER design that help successful implementation include capacity-strengthening for researchers, research translation, co-creation, and the LASER network. Elements that hinder successful implementation include the dual constituencies of USAID and researchers and the decentralized consortium model. Elements that help in some ways but hinder in others include combining buy-ins with core funding and partnering with academic institutions as prime award holders. 3.3 RECOMMENDATIONS The theory of change should be modified to account for obstacles beyond increasing the supply of quality evidence. Creating stronger research products, building research capacity, and increasing network connections between researchers may help produce higher quality research, but do not go far enough to ensure that the research is actually being used by development practitioners, policy makers, and donors. One component that could be emphasized in a revised theory of change is a bigger emphasis on the depth and quality of the research produced, rather than the quantity of new research. This could include a bigger emphasis on prospective research, which is better suited to findings that result in larger policy and program changes. Another missing piece in the theory of change is demand-side interventions that could ensure that decision makers have the desire and ability to prioritize research evidence in their policy and programming decisions. Future iterations of LASER should be streamlined to build on the program’s success. The large number of program components led to some elements being watered down, such as the LASER network; done slower than planned, like the RFA and R4D convenings;10 or done poorly, such as buy-in research.11 A follow-on that focuses solely on researcher capacity-strengthening, with an emphasis on training researchers and USAID M/B/IOs and implementing partners on research translation, would best play to LASER’s current strengths. Other ways to reduce the scope of the LASER core funding would be to reduce the number of RFAs and R4D convenings held to allow additional time and resources for each one, slowing the introduction of buy-ins until Year 2 to allow time for setting up systems and protocols, or eliminate buy-ins entirely, as discussed in Section 4. 10 This is discussed in greater detail in Section 7. 11 This is discussed in greater detail in Section 4. 18 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV One design change that could validate the current theory of change would be to move toward prospective evaluations and research. If USAID M/B/IOs want to produce useful research products with a wide audience outside of USAID for potential translation, the buy-in co-creation process needs to shift away from retrospective and toward prospective evaluations and research that are better positioned to inspire policy and program changes. If this cannot be accomplished, buy-ins should be eliminated to preserve bandwidth for other types of research that are more likely to accomplish the program objective. Though buy-ins have a built-in audience for the findings, the findings of retrospective evaluations are unlikely to lead to the types of larger policy changes and innovations envisioned by LASER’s objective and theory of change. 4. EQ2: WHAT ARE THE CHALLENGES AND SUCCESSES OF LASER M/B/IO BUY-INS? 2.1. What are the challenges and successes of LASER M/B/IO buy-ins, particularly in terms of scope development, Mission engagement, management, and collaboration with academic partners? 2.2 To what extent has the buy-in process and research product quality changed and/or improved over time? 4.1 FINDINGS Between October 15, 2018 and March 31, 2022, the LASER program awarded 17 buy-in research tasks, five of which will have been completed by the end of Q1 FY23. Buy-in activities total $24,978,389, and the average award amount is $1,469,317. The sections below explore the challenges and successes of the LASER M/B/IO buy-ins. SUCCESSES Collaboration with Academic Partners As of January 2023, the LASER website reported that 17 academic partners had been engaged as part of buy-in activities, 7 from HIC countries and 10 from LMICs. Exhibit 7 shows the various self-reported benefits to researchers for participating in buy-ins. Though the sample size is small (n=20), the two largest perceived benefits are related to collaboration—finding appropriate research partners and developing a strong and active network. HIC and LMIC researchers reported similar benefits. Exhibit 7. Benefits of buy-ins research reported by academics ALL RESEARCHERS (N=20) HIC (N=5) LMIC (N=15) Finding appropriate research partners 60% (12) 60% (3) 60% (9) Developing a strong and active network 55% (11) 80% (4) 46.7% (7) 19 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV Learning new research skills 50% (10) 40% (2) 53.3% (8) Learning new strategies for developing research based on local demands 25% (5) 20% (1) 26.7% (4) HIC researchers similarly reported that they were able to work with other researchers or practitioners who they would not have been aware of or able to work with otherwise. One HIC research team stated that it plans to continue working together. In another example, the research team has made connections with USAID and expanded its work with USAID beyond its buy-in research. Compared to LMIC researchers, HIC researchers were less likely to report that the LASER buy-in improved their research skills. There was one contradictory case to the otherwise strong reports of collaboration with academic researchers. One Mission reported a patronizing attitude by U.S. researchers toward the LMIC research team that led to an LMIC university quitting the project. Two USAID respondents told the evaluation team they had hoped to see non-U.S. universities in the lead positions with support from U.S. institutions, instead of the inverse. Though buy-in and RFA research opportunities are circulated the same way, the LASER team told the evaluation team that it receives fewer responses on average for a buy-in opportunity. According to a LASER staff member, “When USAID said they want this, that naturally reduces the pool of people who are interested in that versus when they get to propose something on their own.” CHALLENGES Scope Development Scope development varied depending on the USAID M/B/IO. In South Africa, USAID described the process as very collaborative, including site visits and collaboration with local research universities, while in Malawi the LASER team reviewed a scope that was already developed. The evaluation team found only one example of country partners being involved in scope development. Some buy-ins included an inception visit, others did not. According to the implementing partner and the USAID LASER management team, a USAID Mission often approaches the LASER AOR with a developed idea of what it would like to learn, rooted in the Mission’s CDCS priorities. There is little or no opportunity for country researcher or boundary partner input in the scope development process, unless a USAID Mission seeks it independently. The following quote from a buy-in client is representative of the scope creation process across the buy-ins the evaluation team reviewed. “[The scope] was largely developed internally by us at USAID, even the EQs. I don’t think the EQs changed much throughout the original concept to making the award, but it did involve a lot of consultation with the three missions, the first activity manager coordinated calls with all the missions together to talk through proposed EQs to see if they were useful, interesting, context based, etc.” – USAID buy-in respondent The top-down nature of this process means that the research questions are not influenced by CSFA/SLA, which the LASER implementing partner team identified as a missed opportunity. Mission Engagement 20 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Fifteen M/B/IOs have bought into the LASER program, and two have done so multiple times. USAID respondents lauded LASER’s collaborative research design process once the initial scope was developed. A USAID M/B/IO chose the LASER mechanism because of the co-creation aspect and the ability for non-U.S. and U.S. universities to complement one another. Another USAID M/B/IO representative felt that LASER helped them connect with the right local researchers who had strong relationships and local context that made the research successful. After a high level of engagement during the initial scoping and onboarding, Mission engagement then typically reduces to regular check-ins and readouts of the research findings. These check-ins vary by buy-in. Mission engagement then increases again upon the dissemination of results. Buy-ins have an inherent advantage for research translation—if an M/B/IO is dedicating its own funding for a research product, it has an incentive to disseminate and use the findings. The survey data show that USAID Missions and USAID/Washington comprised the largest audience for buy-in research, followed by in-country researchers or a larger academic audience. However, the evaluation team expected the proportion of researchers who identified USAID as the audience to be closer to 100 percent, given that each research product was produced for a USAID M/B/IO. This difference may be because of a lack of researcher awareness about how research results are disseminated at USAID. Exhibit 8. Perceived audience for buy-in research according to researchers AUDIENCE PERCENT (N=33) USAID Mission 27.8% (10) USAID/Washington 19.4% (7) HEIs in LMICs 11.1% (4) HEIs in HICs 5.6% (2) Host Country Government 8.3% (3) Implementing Partners 19% (3) Local Researchers/Larger Academic Audience 33% (4) Management Management was and continues to be a challenge for buy-ins. The Year 1 bi-annual report notes a steep learning curve, particularly because buy-ins began immediately upon the start of the program. This is supported by qualitative interviews. The AOR noted time spent copyediting early research products, and researchers noted issues such as incomplete guidelines, incorrect information, and services not available when requested. All USAID buy-in respondents reported issues with inconsistent or absent management during the buy-in process, and many relied heavily on the AOR. 21 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV “We had different expectations on the role of Purdue. Thought they would act more like a prime but that’s not what happened… An unusually large amount of money went to Purdue and we don’t think we got value. And research is still embargoed.” – USAID M/B/IO buy-in respondent “In terms of quality, we had a lot of concerns at the very beginning. There were some challenges because of COVID-19 which we understand. There were a lot of comments [on the report] at the beginning, we felt that some conclusions were not backed by robust data and some statements or conclusions were quite sensitive, but in the end, the final report is not extremely high quality but acceptable, they improved it a lot after we provided a lot of feedback and information on the context in Cambodia.” —USAID M/B/IO buy-in respondent One went on to note that management was difficult and that they would not consider using the LASER mechanism again. Every buy-in case study with USAID specifically mentioned issues with LASER management. One team expressed disappointment at the lack of research translation or dissemination offered for its buy-in and had to use other funds to take the dissemination of the research to another mechanism. Researchers also shared concerns about LASER management, particularly, confusion over roles and responsibilities and guidance on USAID processes and regulations. The LASER team has developed systems to improve buy-in management, including standard operating procedures to incorporate the different contexts and needs for managing buy-ins, and interviews with the AORs have noted improvements. The current systems involve detailed project trackers in Google Sheets with information about deliverables, ERT progress, reporting, and achievements. The Year 4 annual report also identifies new systems for increased QA/QC of research translation products by the LASER team prior to submission to the client. The following quote from a USAID MCSIE buy-in respondent is illustrative of the type of feedback on LASER’s performance that was consistent across Missions. “My perspective was that after getting over some initial hurdles where we were not as happy with some of the quality at the beginning, we were able to have open communication, and then we got to a better place and were more on track. Some of the questions about quality at the beginning, perhaps there was misalignment about the expectations.” – USAID M/B/IO buy-in respondent Some of these management challenges are linked to the structure of the LASER consortium, as described in Section 3, EQ 1, and to unclear expectations as to LASER’s role in QA/QC and managing client communications. Two USAID respondents on different buy-ins noted the value of having a mechanism like LASER to conduct research but also noted the need for strong management from the implementing partner. “[USAID] needs a research-oriented mechanism like this, but you need to make sure you have organizations that can handle QA/QC, translate research-speak into development-speak.” – USAID buy-in respondent 4.2 CONCLUSIONS MANAGEMENT OF BUY-INS HAS BEEN A SIGNIFICANT CHALLENGE, AND HAS AFFECTED THE QUALITY OF LASER RESEARCH PRODUCTS. The LASER Consortium was not set up to effectively manage buy-ins, and confusion about roles, responsibilities, quality control, budgets, and timelines was reported by all stakeholder groups. The buy-in process and research product quality have improved over time as the LASER implementing partner developed standard operating procedures and adapted its research processes, including quicker startup, stronger QA/QC of research products, and new ERT processes. 22 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV SCOPE CREATION AND MISSION ENGAGEMENT PROCESSES ARE UNEVEN ACROSS BUY￾INS. Some of the variation is rooted in USAID M/B/IOs and their interest in participating in the research. Other variation may be driven by lack of standard protocols, as well as LASER staff turnover, funding levels, and the personalities and interests of the research team. The evaluation team found a need to implement standardized processes and an opportunity to emphasize the benefits of co-creation to Missions through these standard processes to encourage them to seek local input into scope development. BUY-INS SERVED AS AN IMPORTANT TESTING GROUND FOR ERT PROCESSES AND RESEARCHER CAPACITY-STRENGTHENING TOOLS. This was particularly true in the early years of LASER when many core-funded opportunities were delayed and buy-ins were the primary tool for conducting research. ERT, in particular, used lessons from buy-in research to inform its design. Core￾funded activities benefited from the buy-in component of LASER—but it is less clear if buy-in clients benefit from the core LASER activities in a large enough way to justify combining core funding and buy￾ins under a single mechanism. BUY-INS HIGHLIGHT A TENSION BETWEEN LASER’S LOCALLY DERIVED RESEARCH PRIORITIES AND THE TOP-DOWN NATURE OF THE SOW DEVELOPMENT PROCESS. The evaluation team found little opportunity for collaboration in setting the research priorities, which are often derived from the CDCS, but noted opportunity for collaboration in refining the research questions and methodologies. Given the incentives and priorities of USAID M/B/IOs, this is unlikely to be resolved by any action the LASER team can take. 4.3 RECOMMENDATIONS THE LASER TEAM SHOULD CONTINUE TO DEVELOP AND STRENGTHEN MANAGEMENT PROCESSES to focus on serving USAID M/B/IOs in a rapid, flexible, and cost-effective manner. This includes clean delineation and communication of roles and responsibilities, rigorous QA/QC protocols, responsive timelines, and a standardized onboarding for academic researchers. The time and cost of these activities should be incorporated into the scope of each buy-in award. LASER SHOULD PRIORITIZE INCREASING THE NUMBER OF BUY-INS LED BY LOCAL UNIVERSITIES. In line with USAID’s Locally Led Development approach and LASER’s program objectives, the evaluation team recommends prioritizing increasing buy-in research being led by LMIC universities and researchers, with HIC institutions in a capacity-strengthening or support role. By the end of the program, this number should be 50 percent or higher. To decrease the burden on the LASER team, capacity-strengthening should be part of the role of the partner HIC university, which is a growing priority. This will strengthen the local relevance of the research products, strengthen the capacity of LMIC researchers and institutions, and increase opportunities for USAID M/B/IOs to engage with researchers in their country. LASER SHOULD INCORPORATE LMIC RESEARCHERS INTO THE CO-CREATION PROCESS DURING SCOPE DEVELOPMENT. LASER should develop protocols to incorporate local researchers into the co-creation process with USAID to refine the SOW prior to the release of the RFP. This would alleviate the tension between the top-down nature of Mission priorities and the desire to reflect local priorities. This will be a challenge for buy-ins, since USAID typically knows what it wants before it commissions a buy-in, but if USAID is serious about achieving its localization goals, it can be done. This will require resources for recruiting and paying researchers for the co-creation period prior to the award of the RFP. This could be set up as a standing task order under LASER that would allow LASER to recruit and pay local researchers on behalf of M/B/IOs to assist in scope development. If this is 23 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV successful, there is an opportunity for LASER to socialize this service through USAID’s Local, Faith, and Transformative Partnership Hub. IN FUTURE ITERATIONS OF LASER, BUY-INS SHOULD BE CONDUCTED THROUGH A SEPARATE MECHANISM. By nature, buy-in research, which focuses on Mission priorities and often involves retrospective evaluations, is unlikely to help the LASER program achieve its overarching program objective of innovative program and policy changes targeting a diverse group of stakeholders. Eliminating buy-ins would allow more bandwidth for conducting the types of locally driven and prospective research the program theory of change and objective envisions. If buy-ins remain as part of the next iteration of LASER, the theory of change should be adapted to reflect the constraints and opportunities of buy-in research. Eliminating the buy-in component would not require a new USAID program to replace it because many regional and country-specific research mechanisms already exist to meet this need, including RTAC and the Monitoring and Evaluation Learning Services (EVAL-ME) IDIQ. 5. EQ3: TO WHAT EXTENT HAVE CSFA AND ERT BEEN EFFECTIVE? 3.1 To what extent have CSFA/SLA and ERT been effective in helping stakeholders to build relationships with USAID and with each other, generate locally relevant research, and set the foundation for policy uptake? 3.2 What lessons can be drawn from LASER’s experience with CSFA and ERT to improve the design of future research activities? 3.3 What can be learned from differences in the success of projects that received or did not receive deep ERT assistance and integration? 5.1 FINDINGS EFFECTIVENESS OF CSFA/SLA The evaluation team found limited evidence of the early CSFA being effective in helping stakeholders build relationships, generate locally relevant research, or aid in policy uptake. No researchers from the RFAs where CSFA took place claimed to draw from the CSFA results when designing their research. Two USAID M/B/IO representatives had no memory of the CSFA process, while a third believed that CSFA was not asking the right questions. They noted that the Mission “needed a symposium rather than a conference. [We] needed people to bring key questions and discussion, not just speakers and conversations about what LASER does.” 24 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV In one case study, USAID respondents remembered the process and noted that the R4D convening where CSFA was conducted generated good discussion and helped build relationships, but was not translated into the RFA. Respondents from another case study expressed a strong preference for identifying their own research questions based on their CDCS priorities, rather than identifying gaps through CSFA. USAID respondents from both case studies thought the process moved too slowly. A USAID staff member who attended an R4D convening was critical of bringing an “American tool into a sub-Saharan conference, [and] presenting it as the foundation of discussion,” and recommended the conference open the discussion more broadly and better integrate local stakeholders. He continued “[CSFA] came across as ‘an American used sophisticated tool [and] datasets independent/non-relevant to Africa, here are questions to discuss among yourselves.’” “When [LASER was] first presented, the reaction [at the Mission] was we don’t have time for this — we wanted fast and we wanted solutions… In the six months after they started, the Mission would have said [R4D/RFA] is maybe not a good use of time and energy.” – USAID M/B/IO The number of quantitative respondents who identified that they had participated in a CSFA process is small, but their reported experiences are consistent with the qualitative data. Only 6 of 36 USAID survey respondents thought that the CSFA process during the R4D convening was useful or very useful for developing locally led research priorities, and just 4 of the 36 felt it was useful or very useful for developing new research questions. The numbers are even smaller for usefulness during the RFA process, with only two respondents responding that the CSFA process was useful for both developing locally led research priorities and developing new research questions. Respondents to the researcher survey reported more benefits; five researchers felt CSFA helped develop new partnerships, two felt it helped influence USAID policy, and two felt it helped influence government policy. The LASER team told the evaluation team that a lot of what made CSFA valuable was the process of getting a large group of stakeholders committed to a process by bringing them together for a day and a half to discuss the different questions and hash out details, parts that “had huge value, but are hard to capture with MEL.” However, even the LASER team said that the original CSFA process was not nimble, adaptive, or portable. Because of these shortcomings, the use of CSFA was substantially adjusted and evolved into CSFA/SLA. SLA encourages researchers to take a systems-level view of a complex problem in a particular context and apply it to a specific challenge, ultimately applying the insights gained to articulate new research questions. LASER used this process for the Ethiopia RFA round. Respondents from the Global RFA indicated that the CSFA/SLA process assisted with identifying a key gap, explaining that it was a self￾reflective process for researchers and assisted practitioners with understanding where knowledge gaps are; they also described it as “enjoyable” and “loved the way that it was a very detailed, stepwise process.” Ethiopia RFA respondents indicated that it was useful to start from a gap and learn, particularly when trying to create locally relevant research. The LASER implementing partner identified that the greatest opportunities for CSFA/SLA are for setting strategies and developing the right research questions, which can conflict with USAID’s pre-determined CDCS priorities. “It’s hard to convince M/B/IOs to work where the gaps are,” said one LASER team LASER’s core innovation is Comprehensive Success Factor Analysis, previously called Comprehensive Issue Analysis, and later adapted to CSFA/Systems Level Analysis. CSFA is a highly technical approach that uses machine learning to identify research gaps and inform research questions. The SLA approach is a less technical, though still heavily theoretical, approach to developing research questions that involves taking a systems-level view of a context or challenge to identify new activity focal points, collaboration opportunities, and/or research questions. 25 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV member, if those gaps are not under the CDCS priority areas. A USAID M/B/IO respondent expressed the flip side of this argument, telling the evaluation team that the Mission already knew what questions it wanted to explore, so the CSFA process was not helpful. EFFECTIVENESS OF ERT Data on ERT effectiveness varied greatly from source to source. ERT has evolved from the beginning of LASER; only in Year 3 did the annual report include a standardized definition, and in Year 4 it began to resemble a standardized process. This uneven application of research translation over the past four years likely explains the variance in experiences. It also makes it impossible to compare LASER research that received “deep” ERT support to projects that did not, as there was no uniform ERT protocol across research. Instead, the evaluation team compares ERT support on early research to ERT support on later research. When LASER conducted ERT, researchers were responsive. Three RFA winners and four buy-in researchers told the evaluation team in qualitative interviews that they learned a great deal about producing dissemination materials that were tailored to the audience, especially a non-academic audience. Respondents also mentioned learning how to create new types of dissemination products, such as videos and webinars, beyond the PowerPoint presentations and executive summaries they were used to making. Three key informants specifically mentioned the value of having a translation partner as part of the research design. Two other researchers, both from HICs, told the evaluation team that ERT was similar to what they typically do in their work. Researcher respondents to the quantitative survey responded that the leading benefits to participating in ERT were developing new partnerships (n=18), followed by producing an academic presentation (n=10). From qualitative interviews, researchers were most likely to note that the idea of a translation partner was innovative and benefited their research. Qualitative respondents mentioned that the presentation on ERT at the Ethiopia R4D convening was a valuable part of the convening. One of the strongest pieces of evidence of the perceived value of ERT is the sharing of the ERT model at the USAID Higher Education Global Evidence Summit as an innovative approach to collaborative research. The same summit included links to the ERT model and ERT resources as references. “The idea of ERT is very innovative... most research outputs are in the library. Researchers are not equipped to expedite translation of findings into action or policy. [I] find ERT component very good one, [and] definitely would like to use further in the future.” – RFA researcher “It was interesting to think about having a translation partner because you can’t always guarantee the sustainability of your project, but the translation partner in the field helps think through and ensure sustainability.” – RFA researcher The LASER team’s support of translation activities is deeply uneven. In the Tusome case study of Kenya’s Early Reading Program, LASER provided research translation support and the research team was able to work with a translation partner, resulting in a four-page briefer and a webinar to make the research results more accessible to a wider audience. It did this before the formal introduction of the ERT process. Two later buy-in cases reported that their translation activities have evolved naturally LASER-PULSE defines ERT as an iterative co-design process among academics, practitioners, and other stakeholders in which research is intentionally applied to a development challenge. Core to this approach are four pillars: partnership, process, product, and dissemination. In practice, this includes working with a local partner to design the research; producing dissemination products which LASER calls translation products, such as two￾pagers, policy briefs, and videos; and sharing those products with a wide variety of stakeholders. 26 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV from their research, but did not note any particular support from LASER, including ERT training. Two other activities, both buy-ins, reported being asked repeatedly by the LASER team what their translation needs were, but never receiving any translation support after relaying those needs. Other key informants never discussed ERT with the LASER team, as the following quote illustrates. “The first time I became familiar with that term [ERT] was with the survey that was just disseminating [sic] and I’ll be honest… I was disappointed actually, when I clicked on the link I went to the LASER PULSE website I felt that the buy-in is really missing out. The evaluation is absolutely at a critical point right now when that type of translation is really needed and I’ve actually decided to go through another contract within the education center to help with some of that translation work.” – USAID buy-in participant “[We] were asked repeatedly for responses to an ERT needs assessment. [We] responded and asked for very specific support, but did not receive any.” – USAID buy-in participant The LASER implementing partner team did recognize that it was difficult to bring awareness to its ERT work. As one team member put it, “We are failing to draw attention to what it is exactly that ERT is and its impact. ERT is kind of silent work. If we don’t say this is ERT, this is its result, this is how it’s applied, it is often not evident. Researchers might not be aware that their better outcome is because of ERT.” This dynamic may help explain why researchers and USAID had a different perception of the prevalence and effectiveness of ERT than the LASER team did. The LASER team believes that ERT is an “innovation” of LASER and that starting with the stakeholders and collaborating to decide how the research findings will be used is a unique approach. However, interviews with the leadership of each of the comparison mechanisms also mentioned a similar approach to dissemination. 5.2 CONCLUSIONS THE CSFA/SLA PROCESS HAVE NOT LED TO ANY MEANINGFUL SHIFTS TOWARD LOCALLY DERIVED RESEARCH. Some researchers enjoyed the process, but USAID respondents found it confusing and overly academic. There is no evidence that the process influenced the research questions that researchers proposed in their RFA responses in Colombia or East Africa. Moving away from CSFA and toward the less technical, easier to implement SLA was a good decision by the LASER implementing partner team. THERE IS APPETITE FOR RESEARCH TRANSLATION FROM USAID AND RESEARCHERS, AND LASER IS WELL POSITIONED TO CONTRIBUTE TO BEST PRACTICES THROUGHOUT THE AGENCY. The resources LASER has developed around ERT are high quality and have potential to be shared and adapted across the Agency. The inclusion of a dedicated research translation partner is the most unique element of ERT and sets it apart from the dissemination and utilization tools most researchers and USAID awards currently employ. THE ONGOING EVOLUTION AND UNEVEN APPLICATION OF ERT HAS BEEN A MISSED OPPORTUNITY TO INCREASE THE IMPACT OF LASER-FUNDED RESEARCH. ERT as a concept is very popular. The evaluation team found a significant desire for translation among USAID stakeholders and researchers that was not always met by the LASER team, particularly in buy-in research. Across stakeholder groups, there was confusion over what ERT was and if a research activity had benefited from it, particularly in earlier RFAs and buy-in research before ERT was formally developed. However, LASER has made significant improvements over time to refine ERT and develop resources to share and support ERT use, and the evaluation team expects the positive effects of ERT to continue to increase for newer research. Unfortunately, since the ERT process was developed so 27 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV recently, it is not possible to show any evidence that ERT has been successful at increasing locally relevant research or setting the stage for policy uptake at this point. 5.3 RECOMMENDATIONS CONTINUE TO REFINE ERT AND DEVELOP TRAINING SESSIONS AND PROTOCOLS FOR IMPLEMENTATION, INCLUDING DEVELOPING A “MENU OF SERVICES” FOR BUY-INS. The evaluation team found tremendous potential for ERT if it is well-managed, planned for from the beginning, and consistently implemented. The protocols and resources LASER has developed so far will be helpful for researchers, but more tools are needed to help incorporate ERT into the scope creation process with USAID. One option is to create a “menu of services” to share with buy-in clients that describes the type of translation products that can be produced, determines when in the research cycle they are used, and predicts the approximate cost. IF LASER HOLDS FUTURE R4D CONVENINGS, IT SHOULD FOLLOW THE ETHIOPIA MODEL AND HOLD THE EVENT AFTER THE AWARDS HAVE HAPPENED AND RESEARCH HAS BEGUN, AND FOCUS ON ERT. The adaptations to the R4D convening in Ethiopia that occurred in response to the COVID-19 pandemic resulted in a more impactful conference with opportunities to share the ERT process with researchers. If future R4D convenings take place, the LASER program should follow this model. DISCONTINUE WORK ON CSFA/SLA AS THE LASER MODEL MOVES TO EXCLUSIVELY BUY-IN RESEARCH. Though the newer CSFA/SLA model applied in the Global Round and Ethiopia RFAs was better received by researchers than previous iterations, the shift toward exclusively using buy-ins in the remaining years of the LASER program makes further investment in the CSFA/SLA process irrelevant. IF THE CSFA/SLA PROCESS IS TO CONTINUE, LASER SHOULD LOOK FOR OPPORTUNITIES TO CONDUCT THE EXERCISE DURING A MISSION’S PROCESS OF CDCS DEVELOPMENT. Since CSFA/SLA aims to identify priorities and gaps, the most appropriate way to engage USAID M/B/IOs with this tool is during the CDCS development process. The CSFA/SLA tool would be particularly welcome at this time as M/B/IOs strive to incorporate more local knowledge and priorities into their CDCS priorities. Offering CSFA/CLA consultations as a buy-in opportunity for M/B/IOs could be valuable. 6. EQ4: WHAT ARE THE MOST SIGNIFICANT EARLY RESULTS OF LASER? 4.1. What are the most significant early results of LASER? How have research results differed depending on whether activities were core-funded or buy-in investments? 4.2. How does LASER compare to other international research programs in terms of the co￾creation/scoping process, researcher application process, training/capacity-strengthening, and research relevance? 4.3. Where/in what ways did LASER most stimulate engagement and growth for international research for development in terms of institutions, networks, and researchers, including engaging women and LMIC researchers? 28 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 6.1 FINDINGS EARLY RESULTS Four years into the award, the most significant results are the nine program and policy changes identified by the LASER team. These changes include dissemination to various policy and academic audiences and the adoption and promotion of the ERT approach by the USAID Higher Education Network and the Supporting Holistic and Actionable Research in Education (SHARE) program. There was one government-level policy change. The Tusome buy-in research findings were “most likely” integrated into the Ministry of Education and the Government of Kenya’s education programming decisions. With the exception of the Tusome project, however, what the LASER team identifies as program or policy changes are more accurately categorized as program outputs, rather than the types of changes envisioned in the LASER program objective. Key informant interviews with USAID and researchers could not identify any program or policy changes stemming from the research. Much of the RFA research is still in progress, and none of the RFA case study interviews pointed to any research results at this point. Of the buy-ins, only the Tusome buy-in mentioned any research results. The majority of early LASER results are focused on building the infrastructure to manage core funding and buy-in activities, developing and refining the CSFA/SLA and ERT processes, and making and managing awards. Some of the delays in results can be attributed to COVID-19 and its subsequent interruptions to the RFA research in Colombia and East Africa, as well as the cancelation of R4D events in Ethiopia and Vietnam. Other delays, particularly in awarding the East Africa and Colombia RFAs, were the result of management and start-up challenges. In particular, the Colombia RFA awards were delayed significantly as researchers sought clearances for their research, and staff turnover at USAID in one case study country left the RFA initiative without a champion to push the awards forward or look out for the results. Exhibit 9 shows these early results by IR. The evaluation team attempted to examine activities by core and buy-in funding, but the desk review sources do not always disaggregate between buy-in research and research produced through core funding. Many of these early results are drawn from the program MEL data. LASER has been productive at producing research and translation products, engaging researchers and development actors, and training researchers.12 Some highlights from the annual reporting include training on topics including gender, ERT, and CSFA, creating an evidence gap map on private sector engagement, and developing a self-reliance learning agenda for USAID’s Office of Learning, Evaluation, and Research and the Bureau for Policy, Planning, and Learning. The LASER FOCUSED webinar series is another good example of LASER’s training and translation work. Developed in Year 4, LASER FOCUSED included eight webinars on topics determined by the LASER network, including maximizing the LASER-PULSE network experience, tools and training for research translation, and how to apply for and manage USAID grants. Cumulatively, 351 network members participated. Exhibit 9. Early results by IR 12 According to the Performance Indicator Reference Sheets, research products are defined as “intellectual works and data that advance the state of knowledge. This includes journal articles, conference proceedings, book or book chapters, white papers, evaluations and assessments, datasets, and patents.” According to the Performance Indicator Reference Sheets, “Translated for use means a research product or set of research results has been tailored for non-technical audiences with the intent of facilitating the application of the research. Research products translated for use include, but are not limited to: policy briefs, policy recommendations, editorials, media, infographics and blogs. Incorporation of research into a systematic review can also be considered translation for use. Workshops and workshop presentations designed for decision-makers and other non￾technical audiences can also be considered a research product tailored for use.” 29 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV IR 1. INCREASED DELIVERY OF COLLABORATIVE AND EFFECTIVE DEVELOPMENT￾FOCUSED RESEARCH IR 2. INCREASED SYNTHESIS, EXCHANGE, AND TRANSLATION OF RESEARCH RESULTS INTO USEABLE DEVELOPMENT PRODUCTS AND PRACTICES IR 3. INCREASED DISSEMINATION OF TRANSLATED RESEARCH RESULTS FOR EVIDENCE￾BASED SOLUTIONS IR 4. ENHANCED SYSTEMS AND STRUCTURES FOR GENDER AND MINORITY CONSIDERATIONS IN THE HEI NETWORK THAT ENABLE WOMEN AND MINORITIES TO CONDUCT RESEARCH Nine program or policy changes, produced 101 research products Engaged 604 development actors Engaged in 36 collaborative research initiatives Held community of practice events (five in Year 4) Translated 92 research products Trained 185 development actors on translation Developed 119 research translation materials for researchers Built and refined the LASER-PULSE website Released quarterly newsletters Held 96 convenings with decision makers to disseminate research Translated 137 research products shared with networks, policy makers, donors, and/or the private sector Sent five RFA applications from MSI to USAID for final approval Funded 28 female or minority researchers Conducted promotional and dissemination activities, including webinars, blog posts, and newsletters The evaluation team also conducted a fidelity analysis to examine the number and type of activities planned in annual work plans to those activities reported as complete in the corresponding annual report. Details of the analysis can be found in Annex V. The fidelity analysis confirmed the qualitative findings. LASER was slow in meeting program activities in Year 1, improved in Year 2, was then slowed by COVID-19, and was more successful in meeting Year 3 activities. COMPARISONS TO OTHER MECHANISMS Exhibit 10 shows how each of the four examined contracts and cooperative agreements compare to LASER.13 LASER is unique in its combination of core funding with buy-in funding and strong emphasis on researcher capacity-strengthening. Both RTAC and EPIC CORs interviewed spoke to having very clear guidelines and documentation to support researchers and Missions alike, and RTAC mentioned having a form for Missions to develop scopes. Each COR spoke about offering Missions a lot of support to develop scopes that were practical and feasible. While most mechanisms had an interest in ensuring that research products were accessible and useful to a policy or programming audience, there were a variety of approaches for doing so. LASER’s use of ERT to embed translation as an objective from the beginning of the research process was unusual among the mechanisms examined. Another solution was to engage a research translation partner. Only RTAC offered a similar suite of training opportunities to researchers, although other mechanisms were also able to incorporate capacity-strengthening organically through the research process. 13 For more information about each mechanism and how they were selected, see Annex IV. 30 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Exhibit 10. Comparison of research mechanisms CO-CREATION/ SCOPING RESEARCHER APPLICATION PROCESS TRAINING AND CAPACITY￾STRENGTHENING RESEARCH RELEVANCE LASER Has no sector or geographical limits. Uses both core funding and buy-ins. For core funding, RFA is released to the network and researchers apply. Researchers with specific expertise may be approached by LASER. For buy-ins, often a team with a specific expertise or relationship is approached either by USAID or LASER. Researchers are offered a variety of training through website and organically through research work. ERT is to provide pertinent and actionable information to a program and policy audience. RTAC Has scopes for buy-ins developed by M/B/IOs with strong support from RTAC team. Uses both core funding and buy-ins. Has an average buy-in value that is smaller than LASER. RFP is released to network of researchers. Researchers are offered a variety of training through website and organically through research work. Has research translation partner (Population Reference Bureau), which has helped researchers to disseminate products. HESN Uses both core funding and buy-ins. In core work, university researchers generate proposal. For buy-ins, the M/B/IO comes with a specific question or topic. Works through research centers established at universities. Is accomplished through fellowships with research centers. Is dependent on project—have had success in moving away from academic papers and presentations. EPIC Is limited to education sector, no geographic limits. Has limited research, pertinent to internal USAID operations. Uses buy-ins mainly for workshop facilitation services. Has a template for scopes. Contractor recruits research teams. Capacity￾strengthening component focuses on USAID staff. Is highly relevant to internal USAID audience. Has established EducationLinks to promote and distribute research done under other mechanisms. Has designed innovative tools to track EducationLinks use. LER II Has scopes generated by commissioning M/B/IO through buy-ins, with some co-creation in developing research methodologies. Depends on M/B/IO to gather local input for scope. Is limited to DRG sector, no geographic limits. Contractor recruits research team, including development professionals and academic researchers. Not a part of the LER II objectives, but occurs organically through taskings. Is highly relevant to commissioning M/B/IO, broader dissemination dependent on client wishes. 31 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV Exhibit 11 shows how survey respondents compare LASER to other mechanisms on two dimensions: incorporation of research priorities and increasing connection between LMIC and HIC researchers. Among USAID respondents, 65% reported that they had experience with other USAID mechanisms, but only 33% of network respondents had. The overall assessment of LASER among the 119 USAID staff, academic researchers, and development practitioners who had experience with other mechanisms is positive when compared to other mechanisms on both dimensions. Development practitioners viewed connections between LMIC and HIC researchers (40 percent) most favorably. Notably, two USAID respondents said LASER was significantly worse than other mechanisms at incorporating the research priorities of USAID Missions, policy makers, and other in-country partners, the overriding objective of LASER. Exhibit 11. Comparison of LASER to other mechanisms by respondent group (n=119)14 ENGAGEMENT AND GROWTH The LASER program stimulated engagement and growth for international research for development through thoughtful and intentional engagement of researchers, particularly women and LMIC researchers, the development of the LASER-PULSE network, and building relationships with universities in LMICs. According to the Year 4 annual MEL reporting, LASER is currently funding 28 female or U.S. minority researchers (LP-18), and roughly half of the principal investigators on all awards are women. Furthermore, of the 105 partnerships identified in the Year 4 Annual Report, half (55) are with LMIC universities or other institutions.15 The primary means of engaging with women and LMIC researchers were R4D convenings. Interviews with the LASER team emphasized that engaging women and LMIC researchers was built into the RFA and R4D approach from the beginning. In several countries, the LASER team deliberately asked to speak with female researchers at local universities during their site visits. The LASER implementing partner stressed that “meeting with women sends a strong message about what the U.S. values.” Despite this 14 A total of 17 USAID staff, 88 academic professionals, and 14 development professionals responded to this set of questions. 15 The 105 counts individual partnerships. The number of partners is fewer, as many institutions hold multiple awards. 32 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV concerted effort, interviews with RFA winners indicated only hearing about the funding opportunity through existing USAID contacts or U.S.-based HEIs, which limits the diversity of the researcher pool. One RFA respondent described the need to give preference to under-represented researchers when making awards “in order to build capacity and engage a new set of researchers.” In one R4D case study, participants described that the R4D convening led to stronger relationships between the USAID Mission in Uganda and researchers, particularly Resilient Africa Network (RAN). In another case study, respondents felt that the R4D convening resulted in the development of some relationships—specifically, between academics, funding entities, and CSOs—but that these were slow to develop and of limited use to the Mission. The LASER-PULSE network was another important tool for engaging researchers. Researchers who engaged with LASER through both RFAs and buy-ins were required to join the network, and through that network, have access to 18 courses including training on ERT, CSFA, and gender analysis. Interviews with researchers who received RFA awards found that researchers appreciated the idea of the LASER-PULSE network. Among researchers who participated in the survey, 23 percent (n=83) had participated in LASER training, and of those, a majority (60.5 percent, n=48) found it useful. However, none of the researchers the evaluation team interviewed had taken any of the training beyond what was required to receive funding, or used the network to identify new partners. “I really like the network, it has enormous potential, but I don’t know if I am just too busy or too focused on my own work so I haven’t reached out to others to collaborate on anything yet. I would love for LASER to be like, ok you three have all successfully used LASER funding you could do this RFP together or something like that. Like a thematic matchmaking that might occur if there was a clear mechanism that allowed us to apply for something together. It’s clear there is so much happening, but not enough scaffolding to help push us to collaboration.” – RFA winner 6.2 CONCLUSIONS BECAUSE MUCH OF THE CORE- AND BUY-IN-FUNDED RESEARCH IS STILL ONGOING, IT IS TOO EARLY TO SEE MANY LONG-TERM PROGRAM OR POLICY RESULTS. Much of the work of the first three years was building the program infrastructure and adaptively managing RFAs, R4D convenings, and buy-ins until the team developed systems that worked. Research commissioned in the Colombia and East Africa RFAs experienced delays in award and then further delays because of COVID-19. However, LASER can point to nine program and policy changes, and the project MEL data do a good job of capturing a mix of outcomes and outputs that capture key activities across results. LASER’S FOCUS ON RESEARCHER CAPACITY-STRENGTHENING IS UNIQUE AMONG USAID MECHANISMS, PARTICULARLY ITS ENGAGEMENT OF WOMEN AND LMIC RESEARCHERS. No other comparison mechanism the evaluation team could identify focuses as extensively on building connections between LMIC and HIC researchers. Research translation, which is heavily promoted by the LASER implementing partner as a unique competence of the mechanism, is conducted and prioritized in all mechanisms, though with different processes, and only RTAC included a translation partner. LASER HAS BEEN SUCCESSFUL AT ENGAGING AND GROWING INSTITUTIONS, NETWORKS, AND RESEARCHERS. Institutional and individual capacity-strengthening of funding recipients has been successful at helping researchers and universities partner with USAID for the first time. Lessons learned in the process would be very useful for USAID’s New Partnership Initiative to hear about and incorporate into its work. The large LASER network has potential for creating 33 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV connections between researchers, but there is no evidence that any researchers have used it to form new research partnerships. Similarly, the capacity-strengthening resources available through the network are high quality and have potential to be valuable resources, but no researchers the evaluation team spoke with had engaged with the materials beyond the required training. Eighteen researchers (8 percent) who completed the quantitative survey had used the LASER website for skill building. With the current NCE, the LASER team will likely struggle to maintain, let alone grow, the LASER network. 6.3 RECOMMENDATIONS FOCUS ON PRODUCING FEWER OUTPUTS, BUT INCREASING THE POLICY AND PROGRAM IMPACTS OF WHAT IS PRODUCED. The high volume of research output is straining LASER’s ability to focus on disseminating the research and translation products to an audience that could use the findings to influence decision-making. Focusing, for example, on eliminating the CSFA component, or on releasing a single RFA round per year would free up resources for deeper collaboration with translation partners in the design stage, more customized capacity-strengthening and quality control during the research phase and translation phase, and more time for customized dissemination over a longer time period to ensure the findings can truly influence policy and programs. This recommendation will be less applicable as core funding ends at the end of Year 5 and moves toward an exclusively buy-in model. DEVELOP WIDER DISSEMINATION STRATEGIES FOR LASER OPPORTUNITIES TO GROW THE LASER NETWORK. The LASER network has been a powerful tool for sharing funding opportunities with researchers. But since receiving information about the RFA requires first joining the network, more work can be done to develop wider RFA dissemination strategies such as through professional networks that target junior and/or female researchers in LMICs such as African Women in Agricultural Research and Development. Doing so would expand the pool of researchers who learn about LASER and apply for funding and would also increase the size of the LASER network, helping the LASER team meet its MEL targets (Indicator LP-8, number of institutions or individuals associated with CDR research network). This recommendation has the further advantage of promoting the growth of the LASER network through buy-in funding. 7. EQ5: TO WHAT EXTENT HAVE ADAPTATIONS ADDRESSED CHALLENGES? 5.1 To what extent have the adaptations made by the LASER program to date, particularly those made in response to COVID-19, addressed challenges and increased the likelihood of meeting program goals and objectives? 5.2 How did the LASER program adapt to COVID-19? How did these changes affect early results? 5.3. What is the likelihood of LASER meeting final program goals? What changes can USAID and the implementing partner make in the remaining time to ensure the project meets its goals and objectives? 7.1 FINDINGS Research teams and USAID representatives of buy-in projects spoke favorably of the collaborative and adaptive management approach employed by LASER. The LASER program’s reporting documents numerous adaptations made by the program over the course of the award. 34 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Notable changes the program made that increased the likelihood of meeting program goals and objectives include:  Hiring a full-time program director with extensive program management experience  Standardizing systems and improving buy-in management processes  Modifying the CSFA, and CSFA/SLA methodology in each subsequent R4D event to make it more responsive to Mission and researcher needs  Expanding the country-specific RFA approach to include both global and Minority Serving Institutions (MSI) rounds  Expanding support to researchers to navigate USAID clearance processes  Standardizing the ERT model in Year 3 The best evidence for the effectiveness of these changes comes from comparing the experience of participants of the early RFA and R4D convenings in East Africa and Colombia to the more recent Ethiopia RFA and post-award convening and Global Round RFA. USAID/Ethiopia representatives who attended the R4D convening were overwhelmingly positive about the event, praising the emphasis on research translation and the ability to connect with a diverse group of stakeholders. Respondents believed, but were not certain, that new research ideas arose from the event. One respondent said the greatest benefit of the R4D convening was getting an introduction to the research community in Ethiopia. He told the evaluation team, “There’s an entire ecosystem of researchers and there are opportunities for engaging research opportunities in design and implementation,” and he plans to reach out to this community for future Mission CLA needs. Researchers also praised the Ethiopia R4D event for helping them connect to other researchers, and in at least one case, meet their translation partner in person for the first time. This is a clear contrast to the experience of the R4D convening in Colombia, in which USAID Mission attendees failed to identify benefits of attending; were confused by the CSFA process; and though they appreciated the ability to meet members of the research community in Colombia, did not ultimately make any new partnerships as a result. The evolution of the R4D experience illustrates how well LASER identified program weaknesses and successfully addressed them. COVID-19 LASER’s bi-annual and annual reporting describes significant challenges to programming as a result of COVID-19. LASER was unable to conduct R4D convenings in person or in a hybrid manner. LASER reports that this has hampered progress on some indicators. In addition, travel restrictions prevented some LASER-funded research projects from conducting fieldwork. For example, travel restrictions in Kenya and surrounding countries delayed visits for stakeholder meetings and training for East Africa research projects. Additional restrictions also affected buy-in projects, creating barriers to reaching research respondents, slow responses by government agencies, and slow responses by USAID and other partners. As noted in EQ1, COVID-19 also resulted in postponing some private sector engagement events. LASER developed a recalibration plan in Year 2, in part to outline its strategies to adapt programming for COVID-19. The project moved forward with award rounds for Vietnam and Ethiopia, conducted remotely; plans for post-award in-person workshops in Vietnam and Ethiopia once possible; and a 35 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV follow-up sustainability workshop in Colombia when possible. They developed numerous online training programs, including those for CSFA and ERT. LASER also held virtual pre-conference stakeholder meetings in Indonesia to promote ERT. Buy-in researchers noted the adaptive approach used by LASER in response to challenges raised by COVID-19. Training local academic partners in research techniques and conducting interviews online was inherently more difficult, but overall did not negatively affect the quality of the research. In one buy￾in, researchers described how the team shifted from employing an external team of data collectors to local enumerators instead, which may in fact have benefited the research process. Throughout these various challenges, LASER management was credited for its flexibility and support to help keep projects moving forward. “I think [LASER] did a good job of adapting to carrying out work that was possible during the pandemic as much as they could. They were wanting to get out to the field right when the pandemic hit so they had to really adapt and they did and I was satisfied with how they reacted and I think they were doing a good job under the circumstances.” – USAID buy-in respondent The COVID-19 pandemic’s most significant impact on early results was the delay of research findings. Researchers and USAID reported significant delays in research as a result of travel restrictions and the adjustments to remote work. Other impacts included challenges forming relationships and communicating effectively. However, the pandemic did lead to an improved research design in the Northern Iraq buy-in research, as limits on cross-district travel required a modification to research design that engaged enumerators from the study area, which in turn led to improved responsiveness of the research to local conditions. By Year 4, the Global and Ethiopia RFA award winners indicated that the pandemic had little impact on their research, except a need to follow prescribed safety guidelines while conducting research. Both ERT and CSFA activities were negatively affected by COVID-19, but LASER leadership reported that CSFA was more negatively affected than ERT. Because in-person meetings were not possible, the online infrastructure for ERT was strengthened, which interviewees felt actually improved training and tools. In contrast, the use of CSFA was restricted and was not as easily transferred online. LOOKING FORWARD LASER’s MEL plan contains 18 custom indicators. At the end of Year 4, nine indicators (LP-2, LP-3, LP-6, LP-7, LP-10, LP-15, and LP-17) have already met or exceeded the life of activity targets, and LP-14, # of convenings with decision makers to disseminate research, is likely to do the same in Year 5. Four indicators (LP-1, LP-8, LP-12, and LP-16) are below the life of activity trajectory, and with the exception of LP-16, the Year 4 Annual Report states that with the NCE, all indicators are likely to meet their targets. Exhibit 12 shows MEL achievements by year and by IR. The MEL data support the qualitative and survey findings on LASER’s progress showing that in the first two years, the program struggled to produce research outputs and outcomes, but has been more successful in Years 3 and 4. Year 3 was an especially productive year. The downturn in indicators meeting their targets between Years 3 and 4 gives some pause to the idea that LASER is on a constant upward trajectory, particularly among IR 2 and IR 3 indicators, which are focused on capacity-strengthening and research translation. Exhibit 12. Cumulative MEL indicators through Year 4 by year and IR 36 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV USAID LASER management and the LASER implementing partner team both expressed confidence that with the five-year NCE, LASER will be able to meet its final program goals. “With the extra time they will have a greater likelihood of finishing their work well, in a professional manner. I don’t think they’ll need the full five years, probably two or three, but at least the work won’t be left half done and no one will be left hanging.” – USAID LASER management Moving toward an exclusive buy-in model could make it more difficult for LASER to meet its objectives, as all research areas will be directed by Missions and there will be limited to no funding for core activities like developing the LASER-PULSE network, creating training materials, or further developing the ERT or CSFA process except through buy-ins. 7.2 CONCLUSIONS LASER ADAPTATIONS HAVE BEEN ABLE TO ADDRESS CHALLENGES AND INCREASE THE LIKELIHOOD OF MEETING PROGRAM GOALS. Adaptations related to RFA development and selection criteria, moving the R4D approach away from CSF/CSFA and toward ERT, and refining the definition of ERT have been particularly impactful. LASER’S COVID-19 ADAPTATIONS WERE SUCCESSFUL, AND WHILE RESEARCH RESULTS WERE DELAYED, OTHER NEGATIVE IMPACTS WERE LIMITED. In some cases, COVID-19 led to increased localization by limiting travel, and the changes to a post-award R4D convening focused on ERT led to more positive outcomes than the pre-award model. COVID-19 will not impede LASER meeting its program goals, though it does explain low MEL achievements in Year 2. THE NCE INCREASES THE LIKELIHOOD OF MEETING MEL TARGETS BUT NOT THE OVERARCHING OBJECTIVE. The NCE gives the LASER team five more years to refine its research translation and the capacity-strengthening processes that are essential for enhancing the application of its solutions to development challenges; namely, research translation and researcher capacity￾strengthening. If the LASER team can successfully integrate these leading principles of its core-funded 37 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV research into buy-ins, the shift to an exclusively buy-in model will not affect LASER’s ability to accomplish the program MEL targets and IRs. These include bringing local perspectives into buy-in scope creation, planning for research translation during buy-in scope development, including translation partners in all buy-ins, and actively recruiting a diverse pool of researchers to conduct buy-in work. However, for the reasons related to the theory of change stated under EQ1, achieving the IRs and MEL targets is still unlikely to lead to the collaboration needed to produce policy and programmatic changes. 7.3 RECOMMENDATIONS USE BUY-IN RECRUITMENT EFFORTS TO PROMOTE AND EXPAND THE LASER NETWORK. The primary value of the LASER network has been in its cross-sectoral and international database of researchers, particularly those from LMICs. While searching for the best researchers to staff a buy-in, LASER should go beyond circulation to the LASER network to include LMIC platforms like AWARD and others specific to the country of the buy-in and wider U.S. HEI networks. There is also potential for cross-pollination with RTAC’s network of researchers. LASER should still require registration with the LASER-PULSE network to apply for funding. DEVELOP A PROCESS FOR BRINGING LOCAL PERSPECTIVES INTO THE SCOPE DEVELOPMENT AND RESEARCH TRANSLATION PROCESSES FOR BUY-INS. The greatest threat to LASER achieving its program results during the shift to exclusively buy-in research is the top￾down nature of Mission-created scopes of work. One option to address this is for LASER to work closely with the AOR and with buy-in clients to develop and pilot a process where LMIC researcher voices and priorities are brought into the development of the scope before the RFA is announced. This will require resources in the co-creation process and may make LASER more expensive for Missions to access compared to other research buy-in mechanisms. However, since it will help Missions meet their localization requirements, they may be willing to incur the added expense. LASER should also think carefully about how to adapt the translation partner model to a variety of types of buy-ins. Implementing partners and M/B/IOs may be natural translation partners for some buy-ins, but other types of research may require more creativity to identify a strong translation partner. USAID M/B/IOs may also need some socialization about why a research translation partner is important and a worthwhile component of the scope, which LASER is well positioned to provide. FIND WAYS TO SHARE LESSONS LEARNED ON HEI CAPACITY-STRENGTHENING, ESPECIALLY LMIC HEI CAPACITY-STRENGTHENING, ACROSS THE AGENCY. LASER’s experience with strengthening the capacity of LMIC institutes and researchers offers important insights for USAID as the Agency implements the new Local Capacity-Building Policy. Work with USAID and the New Partnership Initiative, the primary tools for engaging new local partners, are geared toward civil society and private sector businesses. As the evaluation notes, academic institutions have unique barriers that prevent them from engaging successfully with USAID, and LASER’s experience supporting numerous LMIC universities and researchers uniquely positions it to offer lessons learned to other USAID initiatives, including Local Works, Work with USAID, and the New Partnership Initiative. The LASER team and the AOR should explore opportunities for sharing lessons with these teams. 38 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX 1. LIST OF EVALUATION QUESTIONS In consultation with USAID, the evaluation team has identified the following five core EQs to guide the evaluation: 1. What are the strengths and weaknesses of the LASER design? 1.1. What evidence exists that the development hypothesis underlying LASER’s ToC is valid in practice? How might the ToC be adapted to account for early evidence on these development hypotheses? Hypotheses of interest include: 1.1.1. Involving diverse partners in research question identification leads to contextually relevant studies; 1.1.2. Building the capacity of researchers to identify research needs, find appropriate partners; and 1.1.3. Providing researchers with a strong and active network of peers to collaborate with will give them a body of knowledge and resources to leverage for submitting strong research plans/proposals and generating development relevant research. 1.2. What elements of the design of LASER have contributed to vs. hindered implementation? For example, to what extent is combining buy-ins for USAID M/B/IOs and more open-ended core funding under one cooperative agreement hindering or helping implementation? How did integrating attention to gender sensitivities and/or local stakeholders hinder or support implementation? 2. What are the challenges and successes of LASER M/B/IO buy-ins, particularly in terms of scope development, mission engagement, management, and collaboration with academic partners? 2.1. To what extent has the buy-in process and research product quality changed and/or improved over time? 3. To what extent have CSFA and ERT been effective in helping stakeholders to build relationships with USAID and each other, generate locally relevant research, and set the foundation for policy uptake? 3.1. What lessons can be drawn from LASER's experience with CSFA and ERT to improve the design of future research activities? 3.2. What can be learned from differences in the success of projects that received or did not receive deep ERT assistance and integration? 4. What are the most significant early results of LASER? How have research results differed depending on whether activities were core-funded or buy-in investments? 4.1. Early results of interest include: 4.1.1. IR 1: Increased delivery of collaborative and effective development-focused research 4.1.2. IR 2: Increased synthesis, exchange, and translation of research results into useable development products and practices 4.1.3. IR 3: Increased dissemination of translated research results for evidence-based solutions 39 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV 4.1.4. IR 4: Enhanced systems and structures for gender and minority considerations in the HEI network that allow women and minorities to conduct research 4.1.5. Unexpected, unforeseen benefits or challenges, particularly as they relate to COVID-19 adaptations 4.2. How does LASER compare to other international research programs in terms of the co￾creation/scoping process, researcher application process, training/capacity-strengthening, and research relevance from the perspective of: 4.2.1. Researchers that are carrying out LASER research grants and/or buy-ins; Researchers that are part of the LASER network but not carrying out a specific activity; USAID M/B/IOs; and 4.2.2. USAID staff, academic researchers, development practitioners/IPs 4.3. Where/in what ways did LASER most stimulate engagement and growth for international research for development in terms of institutions, networks, and researchers, including engaging women and LMIC researchers? 5. To what extent have the adaptations made by the LASER program to-date, particularly those made in response to COVID-19, addressed challenges and increased the likelihood of meeting program goals and objectives? 5.1. How did the LASER program adapt to COVID-19? How did these changes affect early results? 5.2. What is the likelihood of LASER meeting final program goals? What changes can USAID and the implementing partner make in the remaining time to ensure the project meets its goals and objectives? 40 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX I1. KII SUMMARY TABLE STAKEHOLDER GROUP TOTAL MALE FEMALE IP Leadership 11 4 7 LASER USAID Leadership 3 2 1 Buy-ins / USAID 17 7 10 Buy-ins / Researchers 6 4 2 Buy-ins / LASER Project Manager 1 0 1 Comparative Mechanisms 5 0 5 USAID RFA/R4D 10 7 3 RFA Winners 8 6 2 R4D Participants 6 4 2 Total 67 34 33 41 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV ANNEX III. LASER-PULSE NETWORK SURVEY DEMOGRAPHICS A total of 235 individuals responded to the LASER-PULSE Researcher Network survey. Both men (54.9 percent) and women (43 percent) were well-represented in the survey. LMIC researchers (75.1 percent) were more likely to respond to the survey than were HIC researchers (18.3 percent). Respondents reported participating in a variety of LASER activities, including: RFAs (26.8 percent), R4Ds (9.5 percent), buy-ins (7.8 percent), trainings (20.2 percent), the LASER-PULSE website (16.8 percent), and the researcher network (7.8 percent). GENDER Male 129 (54.9 percent) Female 101 (43.0 percent) RACE/ETHNICITY Black 104 (44.4 percent) White 40 (17.1 percent) Asian/Pacific Islander 38 (16.2 percent) Latinx 19 (8.0 percent) Multiple race/ethnicity 5 (2.1 percent) Other 11 (4.7 percent) ORGANIZATION TYPE Donor organization 9 (3.8 percent) Government 20 (8.5 percent) HEI 156 (66.4 percent) Non-governmental organization 24 (10.2 percent) Private company 15 (6.4 percent) Other 9 (3.8 percent) ROLE Researcher 160 (69.3 percent) Development professional 45 (19.5 percent) Donor 8 (3.5 percent) Other 14 (6.1 percent) INSTITUTION TYPE HIC 42 (18.3 percent) LMIC 172 (75.1 percent) LASER ACTIVITIES RFA 110 (26.8 percent) 42 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV R4D Convening 39 (9.5 percent) Buy-in 32 (7.8 percent) Training 83 (20.2 percent) Website Resources 69 (16.8 percent) LASER Network 32 (7.8 percent) Other 13 (3.2 percent) None 24 (5.8 percent) 43 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV ANNEX IV. LASER-PULSE USAID M/B/IO SURVEY DEMOGRAPHICS The LASER-PULSE USAID M/B/IO survey had 24 total responses. Respondents were nearly equally distributed between males (45.8 percent) and females (50.0 percent). On average, respondents had worked at USAID for 8.9 years. USAID staff who had participated in a buy-in activity had the greatest representation (75.0 percent), followed by RFAs (25.0 percent), and R4D convenings (16.7 percent). GENDER Male 11 (45.8 percent) Female 12 (50.0 percent) No reply 1 (4.2 percent) YEARS AT USAID Average 8.9 Maximum 20 Minimum 0.5 HIRING MECHANISM Foreign Service Officer (FSO) 5 (20.8 percent) Foreign Service National (FSN) 6 (25.0 percent) General Services 3 (12.5 percent) Personal Services Contractor 5 (20.8 percent) Other 5 (20.8 percent) LASER PROGRAM RFA 6 (25.0 percent) R4D Convening 4 (16.7 percent) Buy-in 18 (75.0 percent) Other 4 (16.7 percent) 44 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX V. COMPARISON MECHANISM SELECTION CRITERIA KIIs with the CORs of four research mechanisms explored how each mechanism compares to LASER through the co-creation/scoping process, researcher application process, training/capacity￾strengthening, research quality, and research relevance. The Comparison KIIs were conducted with USAID CORs from each of the four selected mechanisms. Each mechanism was chosen to speak to a specific aspect of the LASER mechanism. ● Higher Education Solutions Network (HESN) spoke to building networks and relationships between HEIs, development professionals, and local actors. ● Research Technical Assistance Center (RTAC) is a global network of academic researchers similar to LASER that provides timely and on-demand research expertise for USAID. RTAC was selected because of its task of establishing, expanding, and maintaining a global research network of university-based scholars, creating and maintaining a website, and developing training. ● The Learning, Education, and Research (LER) II Task Order is a five-year award designed to advance learning activities on DRG activities to assist with decision making. This global activity consists of individual buy-in taskings from Missions and the DRG Office and will speak to the challenges and successes of Mission buy-ins and the benefits and disadvantages of a stand-alone buy-in mechanism, as well as the benefits and challenges of a sector-specific mechanism. ● The Education Performance Improvement, Communications, and Knowledge (EPIC) was a five-year program of support services in the E3 Office of Education to improve education sector impact, programming, measurement, management and performance by strengthening 1) organizational effectiveness; 2) professional development and training; 3) internal and external communications and outreach; and 4) knowledge management uptake and brokering. EPIC was selected due to its highly collaborative approach to co-designing activities and extensive work with Missions. In addition, EPIC had an innovative approach to dissemination through the EducationLinks website. Of the comparison mechanisms selected, RTAC and HESN are most comparable to LASER. Both partner with academic researchers and/or research institutions and do not have a geographic or sectoral focus. In addition, both RTAC and HESN place an emphasis on translating research and ensuring its relevance to a policy or programming audience. EPIC and LER II are sectoral focused mechanisms offering insight into components of LASER: EPIC utilized a strong translation model in the EducationLinks website and LER II is an example of a standalone buy-in mechanism. All interviews were conducted virtually using Google Meet. Transcripts of all interviews were collected for analysis. Interviews lasted 30-60 minutes. Content analysis was used to analyze the Comparison Mechanism KIIs with coding in Excel. Analysis involved reading and re-reading the transcripts and carefully coding and analyzing data according to queries that are designed to correspond directly to the EQs for this evaluation. 45 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV ANNEX VI. FIDELITY ANALYSIS Figure 6 compares the activities planned in LASER’s annual workplans to those that LASER reported as completed in the Annual Reports in Years 1-3.16 Met activities are those that were planned in the workplan and reported as completed in the annual report, not met are those activities that were planned in the workplan but reported as not completed in the annual report. Finally, unmatched activities are those that were either planned and not reported in the annual report, or reported as completed in the annual report, but not planned in the workplan. The total number of planned activities increased in Years 2 and 3. The graphic shows that over the three years of LASER’s implementation, there was significant increase in met activities from Year 1 to Year 2. Many of the unmet activities in Year 1 were delayed due to the startup challenges referenced elsewhere, which were then completed in Year 2. This includes the Colombia “fast track” R4D, first round R4D standard grants, and first round fast track grants. In Year 2, LASER began to see postponed and canceled or adapted activities due to COVID, which account for a large portion of unmet activities. Examples are the canceled R4D in Vietnam and postponed R4D in Ethiopia, in addition to a planned joint conference in Indonesia. There were also a number of activities listed as in development or in nascent stages, but not achieved, such as optimizing the LASER-PULSE networking website and developing two online courses responding to needs in gap analysis. In Year 3, there was a decrease in unmet activities. The few unmet activities in Year 3 were delayed, but have since been awarded, such as award of the Ethiopia and Vietnam RFA grants. 16 At the time of this analysis, the final report for Year 4 was not available. 46 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX VII. EVALUATION SCOPE OF WORK SECTION C – DESCRIPTION/SPECIFICATIONS/ STATEMENT OF WORK C.1 PURPOSE OF THE EVALUATION The goal of the Mid-Term Performance Evaluation of Long-Term Assistance and Services for Research is to improve USAID’s Bureau for Development, Democracy, and Innovation (DDI)/Innovation, Technology, and Research Hub/Research Division’s (ITR/R) understanding of the strengths and weaknesses of LASER in terms of design, partnership choices, implementing mechanism, benefits to stakeholders, and mission engagement. USAID’s ITR/R seeks a deeper understanding of the strengths and weaknesses of the LASER research network model in order to improve current and future ITR/R programming implemented by higher education institutions. The focus of the evaluation is on the design and viability of the LASER approach. The primary audience for the evaluation is DDI/ITR/R leadership and staff. The LASER Prime Awardee, Purdue University, is an additional audience for key findings. Secondary audiences, targeted for certain deliverables under this task order, are:  LASER consortium members and sub-award members  University-based organizations interested in engaging in international development programs  USAID M/B/IOs who may be interested in leveraging, creating, or managing networks similar to LASER  USAID counterparts, such as NGOs and host country government stakeholders C.1.1. DESCRIPTION OF PROJECT TO BE EVALUATED TABLE 1: SUMMARY INFORMATION 47 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV Strategy/Project/Activity Name Mid-Term Performance Evaluation of Long-Term Assistance and Services for Research (LASER) USAID Office Bureau for Democracy, Development and Innovation (DDI)/Innovation, Technology, and Research Hub (ITR) Implementer(s) Purdue University (Prime) Partners: Catholic Relief Services (CRS) Makerere University University of Notre Dame Indiana University Cooperative Agreement/Contract # Cooperative Agreement # 7200AA18CA00009 Total Estimated Ceiling of the Evaluated Project/Activity (TEC) $70,000,000 Life of Strategy/Project/Activity August 2018 - July 2023 Active Geographic Regions USA, Kenya, South Africa, Ethiopia, Malawi, Uganda, Somalia, South Sudan; Nepal, Vietnam, Cambodia, Iraq, Laos, Colombia, Tanzania Development Objective(s) (DOs) Use of research for improved development outcomes Required evaluation? Yes External or internal evaluation? External C.2. BACKGROUND C.2.1 DESCRIPTION OF THE PROBLEM AND CONTEXT USAID has aimed to increase its engagement with Higher Education Institutions (HEIs) and their partners around the world, in order to tap into the unique capabilities of HEIs. For example, world class HEIs are respected for their ability to: test and pilot new ideas; build capacity and foster an enabling environment for innovation and research; provide trusted information; act as a convener by bringing multiple stakeholders together to address common problems; serve a critical role by training the future workforce; and utilize their ever-growing alumni networks as graduates move on to pursue roles in every sector of society. For these reasons, the USAID Bureau for Development, Democracy, and Innovation (DDI)/Innovation, Technology, and Research Hub (ITR) partners extensively with U.S. and international HEIs, aiming to leverage their unique aptitudes to develop innovative and cross- sectoral solutions to development challenges across the globe. USAID/ITR’s longstanding partnerships with numerous U.S. and developing country HEIs have progressively brought increased scientific rigor, new technological solutions, and methodological innovation to bear on solving complex development challenges. By supporting partnerships between U.S. HEIs and HEIs in USAID Partner Countries, ITR/R takes a demand-driven approach to development, while at the same time fostering mutual idea 48 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV sharing and building individual and institutional capacity for development-relevant research and innovation around the world. LASER is a next generation activity based on the Research Division’s foundational programs of the Higher Education Solutions Network (HESN) implemented from 2012-2022 and the Partnership for Enhanced Engagement in Research (PEER) implemented from 2011-2024. HESN is a partnership between USAID and seven top universities to channel the ingenuity of university students, researchers, and faculty towards global development challenges. PEER is an international grants program that funds scientists and engineers in developing countries who partner with U.S. government-funded researchers to address global development challenges. Consultations with USAID staff and the midterm evaluation of HESN revealed that HESN provided value to USAID missions and operating units. PEER has been noted to provide a similar value to USAID missions and operating units with its ability to fund in- country research in particular topic areas. In particular, USAID missions and operating units emphasized the value of easy to access evidence and rapid, on demand research and consultative services. As the original HESN and PEER programs were winding down, Long Term Assistance and Services for Research (LASER) was launched, along with its sister program, Research Technical Assistance Center (RTAC), to support long term research efforts that address specific development needs identified by the Agency. These two programs complement ITR/R’s other programs: The Science, Technology, Innovation and Partnerships Annual Program Statement (STIP APS) and the Science Policy programs (including USAID Research Policy and Coordination, American Association for the Advancement of Sciences Science & Technology Policy Fellows (AAAs), Jefferson Science Fellows, and Research and Innovation Fellows). Together these programs were designed to leverage the unique capabilities of the higher education community to contribute to: 1. The ITR/R’s Science Objective: increased use of scientific research for improved development outcomes, and 2. The objectives of various Missions, Bureaus, and Independent Offices (M/B/IOs) across the Agency. In 2016-2018, USAID DDI/ITR/R conducted a series of external and internal consultations to help inform the best types of mechanisms and program designs that would address the identified needs for rigorous evidence-based programs. Specifically, ITR/R conducted a broad literature review, consulted with 200+ partners and stakeholders internally (within USAID) and externally, and held numerous planning retreats and team discussions to inform the future program designs. In addition, mid-term evaluations of HESN and PEER were completed in Spring 2016 and Spring 2017, respectively. These evaluations, as well as other relevant inputs, also provided valuable input into the design process. These findings have helped inform ITR/R’s proposed objectives, strategic approach, and organizational structure, and helped to design a suite of activities centered around:  increasing the translation of evidence to impact;  generating targeted evidence through innovative research approaches; and  improving USAID’s use of research. Advancing and building local capacity, through ensuring that experts from developing countries are an integral part of the generation and translation of evidence, was a cross- cutting theme throughout the Research Division’s higher education implementation plan and activities. 49 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV Amongst the suite of new programs launched in 2018, LASER was specifically designed to address the following lessons that ITR/R synthesized during these consultations:  Researchers and university partners would like long-term commitments from USAID, in respect to both funds and also the length of collaborations and projects. There was agreement that short term opportunities are appropriate for some subset of the higher education sector, but many established researchers wanted intellectually interesting projects that have the potential to be published in high-impact journals. Further, academic partners discussed the desire to be involved early in the creation of projects (co-creation), and to use these discussions to get a clearer understanding of the local ecosystem (actors, challenges, opportunities) in-country. Stakeholders expressed interest in accessing more of USAID’s core-programming in-country, and to act as translators of evidence, data, innovations, etc. to local actors.  Throughout consultations, development practitioners noted that they lack information on how or where to access research that is relevant to their work. Even if the access issue were to be addressed, development research is often not presented in a manner that is usable or actionable by development professionals. There is a need for synthesis and translation between academic researchers, practitioners, policymakers, and innovators.  Based on consultations with partners, a need was identified for possible long-term research activities that would require original data collection or complex research design(s). Projects under this type of collaboration would require multiple consultations over a period of months, and /or specialized equipment, skill, or resources. This type of work would allow USAID and our partners to be leaders in the field of research for development.  In discussing needs with USAID staff, ITR identified a need for simpler access to researchers and scientists who can co-create long-term research projects based on USAID’s evolving priorities. There was a felt barrier and burden in terms of procuring and managing research awards on the part of USAID staff. USAID staff also noted that research results were often inaccessible in form, untimely in delivery, and irrelevant to the original problem that USAID staff were seeking to solve.  Academic partners (researchers and scientists) look for ways to make their research more impactful and are in need of a program that will allow for engagement with the academic community to answer cross-cutting, risky, or novel research questions. C2.2 DESCRIPTION OF THE INTERVENTION TO BE EVALUATED AND THEORY OF CHANGE In 2018, USAID’s Innovation, Technology, and Research (ITR) Hub (formerly the Global Development Lab) launched LASER to improve development outcomes through research engagement. The five-year, $70 million cooperative agreement allocates: $20 million in funding for core work to researchers to explore development questions that span regions and sectors, the answers to which can benefit the broader development community; and (up to) $50 million of additional funding, contingent on the availability of funds and interest, to support universities and USAID Missions, Bureaus, and Independent Offices (M/B/IOs) in partnering through buy-in agreements to identify evidence gaps, conduct research, and deploy evidence-based solutions specific to M/B/IO needs. After a competitive procurement process, Purdue University was selected to lead the LASER program. The LASER Partners for University-Led Solutions Engine (PULSE) consortium leads a network composed of dozens of universities across 24 low- and middle-income countries (LMICs) and 50 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV eight universities in the US. This network boasts more than 2,300 researchers and practitioners across 56 countries. The LASER award for PULSE lasts until July 2023. The purpose of LASER is to support international and U.S based Higher Education Institutions (HEI)17 and networks, to improve development research opportunities and evidence generation and uptake by development actors and policy makers. Through USAID’s partnership, LASER aims to achieve its purpose by identifying new research questions, fund research activities, translate research results into development impact, and build capacity of local Higher Education Institutions and researchers. Additionally, LASER’s goal is to create, engage, and grow a large, international network of university researchers, centers, and institutes across disciplines that can 1) independently identify and address new and impactful research questions with high relevance to the international development community AND 2) partner with USAID Missions, Bureaus, and Independent Offices (M/B/IOs) to address unique research needs identified by USAID. C.2.3. CATEGORIES OF LASER’S SERVICES:  LASER research grants (Request for Applications) - Grants of up to $250,000 for researcher– practitioner teams to generate research‐driven solutions to field‐ sourced challenges that translate into policy and practice. These are competitively awarded through a request for application (RFA) process. They are generally funded by DDI/ITR/R funds to support the core activities of the award. Currently five requests for applications (RFAs) are underway: East Africa, Colombia, Ethiopia, Vietnam, and a Global Round.  R4Ds - Research for Development convenings: The Research for Development (R4D) Conference is planned as a semi-annual three-day convening of regional research, national and local government officials, NGO, and private sector partners, as well as USAID representatives. In this forum LASER applies Comprehensive Success Factor Analysis (CSFA), a Purdue-developed methodology, to first screen and mine a large volume of documentation to discover development priorities across a large number of stakeholders and iteratively refine that into priority research topics and eventually research questions.  The first two R4D conferences were held in person in Uganda and Colombia.  These included workshops on research translation and other capacity building topics.  The next three RFAs (Ethiopia, Vietnam, and the Global Round) were designed and released without an R4D conference due to COVID-19’s restriction on travel/gathering. Once restrictions are lifted, LASER PULSE plans to hold post-award conferences to provide similar capacity building workshops and networking opportunities. It is unclear how this might look for their newest RFA release, a Global Round research call. 17 A higher education institution is an organization that provides educational opportunities that build on secondary education, providing learning activities in specialized fields. It aims at learning at a high level of complexity and specialization. Higher/tertiary education includes what is commonly understood as academic education but also includes advanced vocational or professional education. This may include public or private universities, colleges, community colleges, academically affiliated research institutes, and training institutes, including teacher training institutes. 51 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV  All researchers are required to take LASER PULSE-hosted online modules designed to build capacity around topics like research translation and gender considerations and the R4D workshops supplement but are not meant to replace these online trainings.  CSFA - Comprehensive Success Factor Analysis/System Level Analysis (SLA): CSFA is an innovation systems-level approach for gathering information from diverse stakeholders for defining priority research areas. LASER sometimes uses the term Comprehensive Issue Analysis.  Embedded Research Translation - Embedded Research Translation is an iterative co-design process among academics, practitioners, and other stakeholders in which research is intentionally applied to a development challenge. Core to this approach are four pillars- partnership, process, product, and dissemination.  Research Capacity Building - LASER PULSE works directly with the RFA awardee and Buy-in teams and its wider network of researchers and development actors to strengthen and sustain their capacity for development research and research translation through training, webinars, and mentorship.  Research Translation/ Translation of Research to Impact - see Embedded Research Translation  Sustainable Network - The LASER PULSE network convenes 2,300+ researchers, practitioners, and policymakers in 56 countries, including all USAID partner countries  USAID Buy-ins: LASER provides specific collaborations with USAID Missions, Bureaus, and Independent Offices (M/B/IOs) through “buy-ins”, which are modifications to the LASER cooperative agreement with DDI/ITR/R that enable expanded M/B/IO engagement and funding. These buy-ins must be within the scope of the LASER cooperative agreement and are co-created between LASER and the interested USAID entity, as approved by the LASER AOR to provide specific research and other activity support. C.2.4. LASER’S THEORY OF CHANGE STATEMENT: Closer collaboration between academic researchers, development practitioners, policymakers, and donors result in new research that is readily translated into useful policies, products, and practices as evidence-based solutions to development challenges. LASER’s theory of change is built on a series of development hypotheses:  Local partnership leads to research translation;  Local partnership leads to contextually relevant research studies;  Involving diverse partners in research question identification leads to research translation;  Involving diverse partners in research question identification leads to contextually relevant studies;  Building the capacity of researchers to identify research needs, appropriate partners, and appropriate communication approaches will improve the translation of their research;  Providing researchers with a strong and active network of peers to collaborate with will give them a body of knowledge and resources to leverage 52 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Figure 2. LASER Results Framework C.3 PROJECT OR ACTIVITY MONITORING, EVALUATION, AND LEARNING (MEL) PLAN C.3.1. EVALUATION QUESTIONS Through the Mid-Term Performance Evaluation of Long-Term Assistance and Services for Research ITR/R seeks to understand the strengths and weaknesses of LASER in terms of design, implementation and collaboration, benefits to stakeholders, and mission engagement. To do this, the evaluation must answer the following questions below. Additional questions might be developed, or initial questions changed or narrowed, in consultation with the evaluation team. The contractor will review and finalize questions in collaboration with USAID prior to finalizing the evaluation design. DESIGN: 53 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV 1. What are the strengths and weaknesses of the LASER design? a. What evidence exists that the development hypothesis underlying LASER’s 54 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV theory of change (TOC) are valid in practice? How might the theory of change be adapted to account for early evidence on these development hypotheses? Hypotheses of interest: ■ Involving diverse partners in research question identification leads to contextually relevant studies. ■ Building the capacity of researchers to identify research needs, find appropriate partners, and use communication approaches will improve the translation of their research throughout the research process. ■ Providing researchers with a strong and active network of peers to collaborate with will give them a body of knowledge and resources to leverage for submitting strong research plans/proposals and generating development relevant research. b. What elements of the design of LASER have contributed vs. hindered implementation? For example, to what extent is combining buy-ins for USAID M/B/IOs and more open-ended core funding under one cooperative agreement hindering or helping implementation? How did integrating attention to gender sensitivities and/or local stakeholders hinder or support implementation? IMPLEMENTATION of priority program elements: 2. Buy-ins: What are the challenges and successes of LASER M/B/IO buy-ins, particularly in terms of scope development, mission engagement, management, and collaboration with academic partners? To what extent has the buy-in process and research product quality changed and/or improved over time? 3. To what extent have Comprehensive Success Factors Analysis (CSFA) and Embedded Research Translation (ERT) been effective in helping stakeholders (NGOs, academic researchers, policy makers, donors, local government) to build relationships with USAID and each other, generate locally relevant research, and set the foundation for policy uptake? What lessons can be drawn from LASER's experience with CSFA and ERT to improve the design of future research activities? EARLY RESULTS 55 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV 4. What have been the most significant early results of LASER? To what extent have early results differed depending on funding model: M/B/IO buy-in vs. core-funded request for applications? a. Early results of interest include: i. Ability to engage local researchers, policy, and practitioner audiences throughout the research process; ii. Getting research started/ off the ground; iii. Engagement of USAID M/B/IOs (alignment of research with country priorities and research needs, introduction of new research directions to M/B/IOs). iv. Stakeholder engagement and satisfaction with process; v. Unexpected, unforeseen benefits or challenges. b. How does LASER compare to other international research programs in terms of the co-creation/scoping process, researcher application process, training/capacity building, and research relevance from the perspective of: i. Researchers that are carrying out LASER research grants and/or buy-ins; ii. Researchers that are part of the LASER network but not carrying out a specific research activity; iii. USAID M/B/IOs; iv. Stakeholders such as development practitioners/NGO implementers, academic researchers, policy makers, local governments, and donors (called Boundary partners by LASER). c. Where/ in what ways did LASER most stimulate engagement and growth for international research for development in terms of institutions, networks, and researchers, including engaging women and minorities? ADAPTIVE MANAGEMENT 5. To what extent have the adaptations made by the LASER program to date addressed challenges and increased likelihood of meeting program goals and objectives? For example: updating processes and timeline to obtain mission concurrence, adding an additional layer of quality control for reviewing final reports, providing additional training and support to local researchers, translating technical research into digestible formats. 56 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV USAID expects the evaluation team to generate actionable and utilizable recommendations based on evaluation findings. USAID is particularly interested in recommendations related to what USAID should have done differently in the design of LASER as well as suggestions for changes USAID and the implementing partner can make in the remaining time to ensure the project meets its goals and objectives. For more information about recommendations, please preview the deliverable section, recommendation workshop. C.4 EVALUATION DESIGN AND METHODOLOGY This evaluation must use rigorous qualitative methods and/or mixed methods. The offeror must propose a method that is best fit to provide answers to the identified evaluation questions and ensure the evaluation team has the technical expertise and capacity to implement the chosen method according to existing standards. This means, for example, achieving acceptable minimums for sampling and triangulation before putting forward findings and recommendations. Further, the evaluation must be cognizant of the complexities and nuance in such work and appropriately capture important contextual information. Offerors should propose methods and provide justification in their proposal. The selected offeror, in collaboration with USAID, will finalize the evaluation methods in the Evaluation Design, deliverable 2, and before virtual site visits begin. Methods to consider include: An implementation study such as rapid cycle fidelity analysis: A method that uses fidelity criteria (e.g., acceptability, appropriateness, adoption, cost)18 to understand LASER's progress towards its goals and objectives. It will allow USAID to understand what elements of the project are working, support for activities, level of uptake of research activities, and how changes to the work plan affect implementation. Qualitative comparative analysis (QCA) to analyze patterns across project elements and across comparator programs (see 2b. below for comparator programs). The design identifies sets of conditions that are most likely to lead to a given output or outcome. The data analysis process embedded in a QCA can reduce the qualitative complexity of different items, such as distinct funding programs, and compare each one's essential characteristics related to specific outcomes.19 Most significant change: a technique to collect and analyze data from diverse project stakeholders on their stories of change and determine the most impactful elements of the project from their perspective. This could capture early LASER results as well as outcomes that were difficult to predict and/or not foreseen during the original project design.20 It is critical that the proposed method entail purposeful, structured analysis and triangulation of findings. 18 For example, see the article Implementation research: What it is and how to do it; fidelity criteria are on page 19 Qualitative Comparative Analysis can be defined as “a means of analyzing the causal contribution of different conditions (e.g. aspects of an intervention and the wider context) to an outcome of interest” (BetterEvaluation.org, 2021). An example of qualitative comparative analysis used in development can be found here: https://journals.sagepub.com/doi/full/10.1177/1098214017710502 20 For more information on Most Significant Change see: https://www.researchgate.net/publication/275409002_The_'Most_Significant_Change'_MSC_Technique_A_Gu ide_to_Its_Use and https://www.intrac.org/wpcms/wp-content/uploads/2017/01/Most-significant-change.pdf 57 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV C.4.1. DATA COLLECTION The offeror should propose data collection methods that fit the evaluation questions and proposed evaluation approach. Key stakeholder sources include:  LASER-PULSE implementing partner leadership;  USAID ITR/R leadership, AORs, and activity managers;  LASER-PULSE Consortium partner staff;  USAID M/B/IO staff supporting LASER buy-ins;  USAID M/B/IO staff participating in Research for Development Convenings and/or processes;  USAID M/B/IO staff who manage LASER-like programs that provide research services (see 2b. below);  University-researchers providing LASER products and services;  Researchers from the PULSE network across US and LMIC universities;  Policymakers, NGO implementers, and policy experts in LMICs where LASER has been active (what LASER calls boundary partners). The evaluation design must include a sampling plan that defines the sampling strategy and data sources. The plan must also describe an inclusive approach to capturing data that ensures underrepresented groups are included in data collection, particularly key informant interviews and focus groups. Potential data collection activities include: 1. Document review of LASER project materials: a. USAID’s LASER RFA, USAID-LASER-PULSE Cooperative Agreement b. LASER annual work plans, bi-annual performance reports, LASER-PULSE MEL Plan, LASER-PULSE Learning Agenda c. LASER buy-in scopes, LASER RFAs, gender analyses, RFD convening reports, LASER buy-in close out reports d. Performance data stored in the M&E platform DevResults e. LASER generated products to date: buy-in deliverables, publications produced using LASER funding, thematic reports, presentations; f. LASER website, LASER training materials and tools 58 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV USAID will share this documentation with the evaluation team prior to field work and data collection. 2. Review of other USAID programs that provide research services such as HESN, PEER, RTAC, Feed the Future, and USAID higher education programming. Review includes at a minimum: a. HESN, PEER, RTAC mid-term evaluations; HESN and PEER long-term evaluations; b. Review of evaluations and/or reports of at least 2 other USAID programs that provide research services. We request that Offerors explain how they will identify and review these programs in their proposal. Offerors should also suggest specific comparator programs. 3. Key informant interviews with stakeholders. If key informant interviews are conducted, the contractor should plan for a minimum of 6 interviews per homogeneous group. (See How Many Interviews are Enough for a rapid review of the research on sample size in qualitative studies). 4. Virtual Site Visits to conduct interviews and/or focus groups with USAID M/B/IO staff, local researchers, and boundary partners such as local policymakers and local NGO implementers. East Africa is suggested for a possible virtual site visit. The contractor must identify and explain the selection of virtual site visits’ “locations” in the evaluation design if this method is chosen for data collection. 5. Online survey/s 6. Focus groups 7. Cost analysis that considers the value for money of program elements such as program management, researcher time, travel, and other direct costs. C.4.2. EVALUATION DESIGN MATRIX: The design matrix below is illustrative. We request that Offerors propose a similar, but more detailed, matrix in their proposal. A final design matrix must be included in the evaluation design for review by USAID. The evaluation Contract Officer’s Representative (COR), with support from the ITR Activity Manager assigned to the evaluation, will approve the finalized evaluation design three weeks or more prior to the launch of fieldwork/interviews. Illustrative Evaluation Design Matrix Table 3: 59 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV Question s Suggested Data Sources Suggested Data Collection Methods Possible Data Analysis Methods 1.DESIGN: What are the strengths and weaknesses of the LASER design? LASER program staff Survey, KIIs, focus groups, document review Fidelity Analysis, Comparative Qualitative Analysis 2. Buy-ins: What are the challenges and successes of LASER M/B/IO buy-ins, particularly in terms of scope development, mission engagement, management, and collaboration with academic partners? To what extent has the buy-in process and research product quality changed and/or improved over time? LASER stakeholders and clients, LASER program staff Survey, KIIs, focus groups Fidelity Analysis, Comparative Qualitative Analysis Most Significant Change 60 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV C.5 EVALUATION TEAM COMPOSITION The contractor must provide information about evaluation team members, including their curricula vitae, and explain how they meet the requirements in the evaluation SOW. The contractor must also provide one writing sample or a link to a past evaluation report composed by (at a minimum) the Evaluation Advisor (Team Lead) proposed for the project team. Per ADS 201.3.5.14, all team members must provide to USAID a signed statement attesting to a lack of conflict 3. To what extent have LASER Fidelity Analysis, Comprehensive Success Factors stakeholders and Survey, KIIs, Comparative Analysis (CSFA) and Embedded clients, LASER focus groups Qualitative Analysis Research Translation (ERT) program staff Most Significant been effective in helping Change stakeholders (NGOs, academic researchers, policy makers, donors, local government) to build relationships with USAID and each other, generate locally relevant research, and set the foundation for policy uptake? 4. What are the most significant M/B/IO staff, Survey, KIIs, early results of LASER? How LASER program focus groups, Comparative have research results differed staff (USAID and power analysis, Qualitative Analysis, depending on whether activities IP) case studies Most Significant were core funded or buy-in Change investments? 5. To what extent have the M/B/IO staff, Survey, KIIs, adaptations made by the LASER LASER program focus groups, Fidelity Analysis, program to date addressed staff (USAID and case studies, Comparative challenges and increased IP) document review Qualitative Analysis likelihood of meeting program goals and objectives? 6. What changes can USAID M/B/IO staff, Survey, KIIs, Fidelity Analysis and the implementing partner LASER program focus groups, Comparative make in the remaining time to staff (USAID and document review Qualitative Analysis ensure the project meets its goals IP) and objectives? 61 | LASER MID-TERM EVALUATION PERFORMANCE REPORT USAID.GOV of interest or describing an existing conflict of interest relative to the project or activity being evaluated (i.e., a conflict of interest form). Evaluation teams with members from USAID partner countries is desirable. C.5.1. CONTRACTOR STAFF The Contractor shall provide at a minimum, the following key personnel for the performance of this task order: Evaluation Advisor (Team lead) -- 1 position Roles and Responsibilities: Serve as team lead for designing, conducting, and managing the evaluation throughout the life of the task order; provide quality control oversight for evaluation; and ensure the integrity of the methodology, process, and evaluation findings. Responsibilities include design of evaluation methodologies, data collection methods and protocols, and facilitation of interviews and focus groups, as well as data quality verification. As team lead, the evaluation advisor is responsible for guiding a multidisciplinary cross- cultural team and leading efforts to draft the final evaluation report. Minimum Qualifications:  Holds a Master’s degree, (PhD preferred), in international development or a related field such as economics, public administration/policy, sociology, or program evaluation;  Has at least 10 years of professional experience in international development including at least 8 years of work designing and executing evaluations of development programs with a strong preference for experience with research technology, and innovation programs;  Strong written and verbal communications skills; and  More than 8 years of experience designing and conducting rigorous qualitative evaluations; extensive experience conducting qualitative data collection such as key informant interviews and focus groups; extensive experience with qualitative data analysis, coding, and qualitative analysis software. Evaluation Methods Specialist -- at least 1 position Roles and Responsibilities: Develop quantitative and qualitative data collection tools to support evaluation design and implementation. Support the collection, coding, processing, and analysis of quantitative and qualitative data and ensure inclusive measures (tools and approaches) are taken to seek and collect data from a diversity of groups. Minimum Qualifications:  Holds a bachelor’s or master’s degree in economics, statistics, public administration/policy, sociology, anthropology, or a related field; and  5+ years designing and implementing evaluations of international development programming. Data Analyst-- at least 1 position 62 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV The contractor may choose to propose additional team members, such as technical advisor or additional data analysts, in order to meet the objectives of this task order. The contractor must provide information about evaluation team members, including their curricula vitae, and explain how they meet the requirements in the evaluation SOW. A writing sample is required from the Evaluation Advisor (team lead). Additional submissions of writing samples and/or links to past evaluation reports and related deliverables composed by proposed team members are highly desirable. Proposed key personnel are expected to be the people who execute the work of this contract. Any substitutes to the proposed key personnel must be vetted and approved by the Evaluation COR (including the review of a writing sample) before they begin work. USAID may request an interview with any of the proposed evaluation team members via conference call, Skype, or other means (either pre￾award as part of the selection process or post-award, if there are any proposed changes to key personnel). In addition to the Offeror’s evaluation team, the Offeror must work in close collaboration with a monitoring and evaluation specialist in USAID/DDI/ITR/R throughout the implementation of this task order. Table 4: Task LOE Evaluation Advisor (Team lead) LOE for Evaluation Methods Specialist LOE for Data Analyst Administrativ e Assistant Total LOE in days Document review/work planning and evaluation design drafting Team planning meetings in Washington DC (including travel time) (travel if applicable, otherwise virtual); Finalize Workplan 63 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Inception report and finalize evaluation design and develop data collection instruments (may require some additional meetings with USAID staff) Preparations for travel and data collection (scheduling calls and visits, visas, arranging vehicles, lodging etc.) (if applicable) Data collection, includes interviews with key informants, DC-based focus groups, and telephone interviews (DC￾based and phone/teleconference) Data collection site 1, plus in and out brief, and travel time Data collection site 2, plus in and out brief, and travel time 3 7 7 0 17 Data analysis 3 4 5 1 13 Recommendation’s workshop (preparation and workshop) 1 0.5 1 1 3.5 Draft report and presentation 64 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV The evaluation COR may observe some of the data collection efforts. USAID may also delegate one or more staff members to participate in selected evaluation activities. The evaluation COR will inform the contractor in writing about any desired USAID participation. C.5.2. ILLUSTRATIVE ESTIMATED LOE IN DAYS BY ACTIVITY FOR A TEAM OF 4 C.5.3. LIST OF ANNEXES [END OF SECTION C] Revise report and slides Submission of final report to DEC, data to DDL 0.5 0 2.5 1 4 TOTALS: LASER Monitoring Evaluation and Learning Plan￾REDACTED 65 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX VIII. EVALUATION DESIGN REPORT Mid-Term Performance Evaluation of Long￾Term Assistance and Services for Research (LASER) Evaluation Design Report February 18, 2022 LINC LLC 1100 15th St NW, 4th Floor, Washington, DC 20005 Email: info@linclocal.org Website: www.linclocal.org This document was produced for review by the United States Agency for International Development. It was prepared by LINC under the USAID EVAL-ME II IDIQ, Contract #7200AA20D00017, Task Order #7200AA21F00004. 66 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV CONTENTS 1. Introduction and Background 4 1.1 LASER Background 4 2. Evaluation Design 6 2.1 Evaluation questions 6 2.2 Overview of Evaluation Design 8 2.3 Indicators and Outcomes 10 3. Data Sources, Sampling, and Analysis 13 3.1. USAID M/B/IO Quantitative Survey (N=52) 14 3.2. LASER-PULSE Network Quantitative Survey (N=437) 16 3.3. LASER-PULSE Leadership KII (N=7) 17 3.4. RFA KII (N=4) and FGDs (N=2) 17 3.5. R4D Case Study KIIs (N=28) 19 3.6. Buy-In KIIs (N=24) 20 3.7. Comparison Mechanism KII (N=4) 21 3.8 Risks and Limitations 22 4. Administrative 23 4.1 Roles and Responsibilities 23 4.2 Communications Outreach Plan 24 4.3 Dissemination Plan 25 4.4 Diversity, Equity, and Inclusion 27 4.5 Ethical Considerations 27 Annex 1. Evaluation Matrix 28 Annex 2. KII List 34 Annex 3. USAID M/B/IO Quantitative Survey 39 Annex 4. LASER-PULSE Network Survey 51 Annex 5. LASER Core Team KII Guide 66 Annex 6. RFA KII Guide 67 LASER-PULSE Staff (Global/colombia) 67 USAID/Mission Staff (Global/Colombia) 68 RFA Subgrant Awardees (Global/colombia) 69 Annex 7. KII Guide- R4D Convenings KIIs 72 LASER Activity Manager 72 USAID Mission participants, research participants, and Boundary Partner participants 73 Annex 8. KII Guide- Buy-Ins KIIs 76 Buy-In LASER Activity Manager 76 Buy-In USAID Mission POCs 78 Buy-In Principal Investigators 80 Buy-In Boundary Partners 82 Annex 9. KII Guide - Comparison Mechanism Guide 84 ANNEX 10. FINAL REPORT OUTLINE 86 67 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ACRONYMS AOR Agreement Officer’s Representative COR Contracting Officer’s Representative CSFA Comprehensive Success Factor Analysis DEC Development Experience Clearinghouse DDI Bureau for Development, Democracy, and Innovation ERT Embedded Research Translation ET Evaluation Team FGD Focus Group Discussion FSN Foreign Service National HEI Higher Education Institutions HIC High Income Country IP Implementing Partner ITR/R Innovation, Technology, and Research Hub/Research Division KII Key Informant Interview LASER Long-Term Assistance and Services for Research LMICs Low-and middle-income countries M/B/IOs Missions, Bureaus, and Independent Offices MSC Most Significant Change PULSE Partners for University-Led Solutions Engine R4Ds Research for Development convening SLA Systems Level Analysis TOC Theory of Change USAID United States Agency for International Development 68 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 1. INTRODUCTION AND BACKGROUND USAID’s Innovation, Technology, and Research Hub (ITR) launched the Long-Term Assistance and Services for Research (LASER) program in 2018 to improve development outcomes through research engagement. Based on a gap identified by researchers and practitioners alike, the purpose of LASER is to support international and U.S based Higher Education Institutions (HEI) and networks to improve development research opportunities, evidence generation, and uptake by development actors and policy makers. Through USAID’s partnership, LASER aims to achieve its purpose by identifying new research questions, fund research activities, translate research results into development impact, and build capacity of local HEIs and researchers. This document presents the research design and instruments for the mid-term performance evaluation of LASER, which is being conducted by LINC, together with its partners The Cloudburst Group and DevLab@Duke University. The evaluation objectives are: 1. To improve USAID’s Bureau for Development, Democracy, and Innovation (DDI)/Innovation, Technology, and Research Hub/Research Division’s (ITR/R) understanding of the strengths and weaknesses of LASER in terms of design, partnership choices, implementing mechanism, benefits to stakeholders, and mission engagement. 2. To gain a deeper understanding of the strengths and weaknesses of the LASER research network model to improve current and future programming implemented by higher education institutions. The evaluation will analyze how LASER has been used in practice, identify its strengths and weaknesses, and address what USAID should have done differently in the design of LASER. The evaluation team (ET) will answer research questions about LASER’s outputs and outcomes to-date, and the evaluation findings will provide actionable recommendations for changes that USAID and the implementing partner can make in the remaining two years of implementation to ensure the project meets its goals and objectives, as well as for future iterations of the mechanism. This evaluation design report describes the research approach and methodology in detail, including an evaluation design matrix, sampling methodology and methods, analysis plan, communication outreach and dissemination plans, list of potential interviewees, and all qualitative and quantitative instruments. 1.1 LASER BACKGROUND LASER funds researchers in exploring development questions that span regions, sectors, and universities. It supports USAID Missions, Bureaus, and Independent Offices (M/B/IOs) through core research grants and Research 4 Development (R4D) Convenings, as well as through buy-in agreements21 to conduct research and deploy evidence-based solutions that benefit M/B/IO needs. Launched in 2018, LASER is intended as a five-year cooperative agreement. LASER is implemented by Purdue University, in consortium with four additional partners: Catholic Relief Services (CRS), Indiana University (IU), Makerere University in Uganda (Makerere), and University of Notre Dame (UND). Per LASER’s theory of change (TOC) (See Figure 1 below), the evaluation will explore if/how LASER activities result in new research that is readily translated into useful policies, products, and practices as evidence-based solutions to development challenges. 21 A buy-in is when a USAID Mission, Bureau, or Independent Office uses its own funding to “buy” the services of a project or activity originally procured/contracted by another USAID office. 69 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Figure 1. LASER TOC The primary categories of LASER’s services are: 1. LASER research grants (Request for Applications): Grants of up to $250,000 for researcher– practitioner teams to generate research‐driven solutions to field‐sourced challenges that translate into policy and practice. These are competitively awarded through an RFA process. They are generally funded by DDI/ITR/R funds to support the core activities of the award. Six RFAs are currently underway: East Africa, Colombia, Ethiopia, Vietnam, a Global Round, and a Minority Serving Institutions Round. 2. Research for Development convenings (R4Ds): The R4D Conferences were originally planned as a semi-annual three-day convening of regional research, national and local government officials, NGO, and private sector partners, as well as USAID representatives, but have taken on alternative forms in response to COVID-19. In this forum, LASER applies Comprehensive Success Factor Analysis (CSFA), a Purdue-developed methodology, to first screen and mine a large volume of documentation to discover development priorities across a large number of stakeholders and iteratively refine that into priority research topics and eventually research questions. R4D convenings have been held in East Africa, Colombia, Vietnam, Thailand, and Ethiopia. 3. Comprehensive Success Factor Analysis (CSFA)/Systems Level Analysis (SLA): CSFA is an innovation systems-level approach for gathering information from diverse stakeholders for defining priority research areas. LASER sometimes uses the term Comprehensive Issue Analysis. 4. Embedded Research Translation: Embedded Research Translation is an iterative co-design process among academics, practitioners, and other stakeholders in which research is intentionally 70 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV applied to a development challenge. Core to this approach are four pillars: partnership, process, product, and dissemination. 5. Research Capacity Building: LASER PULSE works directly with the RFA awardee, buy-in teams, and its wider network of researchers and development actors to strengthen and sustain their capacity for development research and research translation through training, webinars, and mentorship. 6. Sustainable Network: The LASER PULSE network convenes 2,500+ researchers, practitioners, and policymakers in 61 countries22, including all USAID partner countries 7. USAID Buy-ins: LASER provides specific collaborations with USAID M/B/IOs through “buy￾ins”, which are modifications to the LASER cooperative agreement with DDI/ITR/R that enable expanded M/B/IO engagement and funding.23 This evaluation will review the strengths and weaknesses of LASER’s services and approach, including an analysis of LASER research grants, R4Ds, research capacity building, and USAID buy-ins. The ET will pay special attention to differences in the performance of core-funded or buy-in investments, as well as how the LASER program adapted to the COVID environment. 2. EVALUATION DESIGN 2.1 EVALUATION QUESTIONS In consultation with USAID, the ET has identified the following five core evaluation questions (EQs) to guide the evaluation: 1. What are the strengths and weaknesses of the LASER design? 1.1. What evidence exists that the development hypothesis underlying LASER’s theory of change (TOC) is valid in practice? How might the TOC be adapted to account for early evidence on these development hypotheses? Hypotheses of interest include: 1.1.1. Involving diverse partners in research question identification leads to contextually relevant studies; 1.1.2. Building the capacity of researchers to identify research needs, find appropriate partners; and 1.1.3. Providing researchers with a strong and active network of peers to collaborate with will give them a body of knowledge and resources to leverage for submitting strong research plans/proposals and generating development relevant research. 1.2. What elements of the design of LASER have contributed to vs. hindered implementation? For example, to what extent is combining buy-ins for USAID M/B/IOs and more open-ended core funding under one cooperative agreement hindering or helping implementation? How did integrating attention to gender sensitivities and/or local stakeholders hinder or support implementation? 2. What are the challenges and successes of LASER M/B/IO buy-ins, particularly in terms of scope development, mission engagement, management, and collaboration with academic partners? 22 LASER PULSE network numbers as of December 13, 2021. As this number is constantly growing, the most recent numbers can be found on the network website. 23 These buy-ins must be within the scope of the LASER cooperative agreement and are co-created between LASER and the interested USAID entity, as approved by the LASER AOR to provide specific research and other activity support. 71 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 2.1. To what extent has the buy-in process and research product quality changed and/or improved over time? 3. To what extent have Comprehensive Success Factors Analysis (CSFA) and Embedded Research Translation (ERT) been effective in helping stakeholders to build relationships with USAID and each other, generate locally relevant research, and set the foundation for policy uptake? 3.1. What lessons can be drawn from LASER's experience with CSFA and ERT to improve the design of future research activities? 3.2. What can be learned from differences in the success of projects that received or did not receive deep ERT assistance and integration? 4. What are the most significant early results of LASER? How have research results differed depending on whether activities were core-funded or buy-in investments? 4.1. Early results of interest include: 4.1.1. IR 1: Increased delivery of collaborative and effective development-focused research 4.1.2. IR 2: Increased synthesis, exchange, and translation of research results into useable development products and practices 4.1.3. IR 3: Increased dissemination of translated research results for evidence based solutions 4.1.4. IR 4: Enhanced systems and structures for gender and minority considerations in the HEI network that allow women and minorities to conduct research 4.1.5. Unexpected, unforeseen benefits or challenges, particularly as they relate to COVID-19 adaptations 4.2. How does LASER compare to other international research programs in terms of the co￾creation/scoping process, researcher application process, training/capacity building, and research relevance from the perspective of: 4.2.1. Researchers that are carrying out LASER research grants and/or buy-ins; Researchers that are part of the LASER network but not carrying out a specific research activity; USAID M/B/IOs; and 4.2.2. USAID staff, academic researchers, development practitioners/implementing partners 4.3. Where/in what ways did LASER most stimulate engagement and growth for international research for development in terms of institutions, networks, and researchers, including engaging women and Low-and Middle-Income Country (LMIC) researchers? 5. To what extent have the adaptations made by the LASER program to-date, particularly those made in response to COVID-19, addressed challenges and increased the likelihood of meeting program goals and objectives? 5.1. How did the LASER program adapt to COVID-19? How did these changes affect early results? 5.2. What is the likelihood of LASER meeting final program goals? What changes can USAID and the implementing partner make in the remaining time to ensure the project meets its goals and objectives? 2.2 OVERVIEW OF EVALUATION DESIGN The evaluation will comprehensively address each of the research questions posed above and validate the program logic underlying each activity. To achieve these objectives, the ET will assess performance indicators and outcomes through a mixed-methods approach that triangulates findings from multiple quantitative and qualitative sources. As described in more detail below, the ET will analyze performance indicators and outcomes across each data source. In addition, our research methodology employs a case study approach to examine four LASER buy-ins, to be selected after quantitative data collection, and two R4D convenings, one in Colombia and one in East Africa. For each case study, the ET will conduct a detailed document review of all project documents related to the activity and conduct six interviews with various stakeholders. Using coded 72 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV interview data, the ET will analyze patterns across project elements to answer questions about each activity’s implementation, design, early results, and adaptive management. Case studies will be purposefully selected in collaboration with USAID/DDI and based on the results of the online surveys. Our methodology also utilizes fidelity analysis24 to understand how the LASER mechanism performed against several implementation outcomes, including acceptability, appropriateness, and fidelity. Through careful document review - including agreements, workplans, annual reporting, and monitoring and evaluation data supplemented by key informant interviews (KIIs) and quantitative surveys - the ET will be able to answer questions related to LASER’s successes and challenges, as well as trade-offs between fidelity to the workplan and the ability to manage adaptively. Fidelity analysis will be most useful in answering research questions related to the program design, implementation, and adaptive management. A mixed methods approach was selected to allow for triangulation of findings using data from multiple sources and to ensure rich, comprehensive data that thoroughly explores the research questions. We note that this is not a causal analysis of outcomes and impacts, and we cannot attribute any changes specifically to LASER. Overall, the evaluation design utilizes the following five qualitative data sources: ● LASER-PULSE Core Team KIIs ● RFA KIIs ● Buy-In Case Study KIIs ● R4D Case Study KIIs ● Comparison Mechanism KIIs The evaluation design utilizes the following two quantitative data sources: ● USAID/M/B/IO Quantitative Survey ● LASER-PULSE Research Network Survey The evaluation design utilizes the following secondary data sources: ● Annual reports, workplans, MEL plans ● Buy-in research products and program descriptions ● CSFA and ERT research products ● RFA solicitations and grant proposals Our quantitative data will be analyzed with descriptive and inferential statistics (where possible) to answer the research questions. The ET will generate descriptive statistics from the survey results to show the distribution of responses across questions. Response tabulations will summarize responses in an easily understood manner; these findings will be compared and contrasted with the results from the qualitative data collection. Where possible, responses will be further disaggregated by gender and LMIC/HIC respondents. The ET will test the significance of those differences and use this to suggest possible variation between these groups. but the comparisons can help to better understand possible diversity of experience and outcomes. Our qualitative data sources will be explored through Most Significant Change (MSC) analysis and content analysis. First, the evaluation will employ content analysis to add depth and triangulate the other research methods. By conducting a content analysis of documents, KIIs, and focus group 24 Fidelity analysis refers to the extent that the LASER agreement was implemented as it was designed. 73 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV discussions (FGDs), the ET will conduct thematic analysis organized by research question. The ET will identify instances where the evidence coincides and where there are divergences, making sure to indicate whether any divergences are noted by type of respondent, or by type of LASER activity. Second, the evaluation will collect and analyze personal accounts of change related to LASER program activities from USAID/DDI and Mission staff, academic researchers, and boundary partners to understand how and in what context LASER activities achieve their objectives. In deciding what account is most significant, the ET will pay particular attention to variations in what different stakeholder groups value. MSC will be most useful in answering research questions related to the early results of LASER, the success of buy-ins, CSFA, and ERT. R4D AND RFA SITE SELECTION LASER core activities frequently included an RFA and R4D Convening in the same country, both of which involve CSFA and ERT. The ET chose to select two R4D sites and two RFA processes to focus the key informant interviews and in-depth document review. The ET considered four possible sites where LASER conducted both an RFA and an R4D Convening: East Africa, Colombia, Vietnam, and Ethiopia. Two sites (East Africa and Colombia) had activities conducted in Year 1 and early in Year 2. Activities in Vietnam and Ethiopia both took place late in Year 2 and early in Year 3. At the urging of the LASER leadership and IP team, the ET chose to focus on the two in-person R4D Convenings, one in East Africa and one in Colombia. Choosing the East Africa and Colombia sites will allow the team to identify lessons learned from East Africa that influenced the Colombia implementation. Both R4D Convenings took place in person and have had ample time to see results and impacts from the convening. To complement the Colombia R4D, the team has also selected Colombia as a site for the in-depth RFA assessment. This will allow the ET to examine linkages and complementarities between LASER’s activities in a region. However, as the RFA process has evolved from a single country to other structures, including global and minority researcher calls for applications, the ET has chosen the Global RFA round as the second RFA “site.” Including a later RFA process will allow the team to capture the evolution of the RFA process over time, as well as how the RFA process was adapted to COVID-19. 2.3 INDICATORS AND OUTCOMES The primary indicators and outcomes that will be measured under each evaluation question are elaborated in Table 2 below. In Annex 1, we present the detailed Evaluation Matrix which shows the relationship between evaluation questions, indicators and outcomes, data sources, methods, and analysis plan. Table 2. Outcomes and Indicators Evaluation Question Indicators and Outcomes Evaluation Question 1 EQ1.0 What are the strengths and weaknesses of the LASER design? ● Research development ● Capacity building ● Demand-driven research ● LMIC and female researcher integration ● Research dissemination 74 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ● Research utilization ● Stakeholder engagement ● Effectiveness of LASER contract management EQ1.1 What evidence exists that the development hypothesis underlying LASER’s theory of change (TOC) is valid in practice? How might the TOC be adapted to account for early evidence on these development hypotheses? ● Development of locally-relevant research ● Quality and uptake of capacity building ● Extent and quality of research translation ● Extent and strength of research networks ● Quality of research products EQ1.2 What elements of the design of LASER have contributed vs. hindered implementation? ● % of deliverables met on time ● Percent of MEL targets achieved, surpassed, or underperformed ● Implementation successes ● Implementation challenges Evaluation Question 2 EQ2.0 What are the challenges and successes of LASER M/B/IO buy-ins, particularly in terms of scope development, mission engagement, management, and collaboration with academic partners? ● Efficiency of management ● Client satisfaction ● Research product utilization ● Research product dissemination ● Timeliness of research (results available when needed for utilization) ● Access to high-quality researchers ● Access to diverse researchers EQ2.1 To what extent has the buy-in process and research product quality changed and/or improved over time? ● Change in efficiency of management ● Change in client satisfaction ● Change in research product utilization ● Change in research product dissemination ● Change in timeliness of research (results available when needed for utilization) ● Change in access to high-quality researchers ● Change in access to diverse researchers Evaluation Question 3 EQ3.0 To what extent have Comprehensive Success Factors Analysis (CSFA) and Embedded Research Translation (ERT) been effective in helping stakeholders to build relationships with USAID and each other, generate locally relevant research, and set the foundation for policy uptake? ● New relationships established through CSFA ● New relationships established through ERT ● New relationships sustained after CSFA ● New relationships sustained after ERT ● New research generated through CSFA ● New research generated through ERT ● Policy or programming changes related to CSFA ● Policy or programming changes related to ERT EQ3.1 What lessons can be drawn from LASER's experience with CSFA and ERT to improve the design of future research activities? ● New relationships established through CSFA ● New relationships established through ERT ● New relationships sustained after CSFA ● New relationships sustained after ERT ● New research generated through CSFA ● New research generated through ERT 75 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ● Policy or programming changes related to CSFA ● Policy or programming changes related to ERT EQ3.2 What can be learned from differences in the success of projects that received or did not receive deep ERT assistance and integration? ● New relationships established through CSFA ● New relationships established through ERT ● New relationships established on projects that did not include ERT ● New relationships sustained after CSFA ● New relationships sustained after ERT ● New relationships sustained on projects that did not include ERT ● New research generated through CSFA ● New research generated through ERT ● Policy or programming changes related to CSFA ● Policy or programming changes related to ERT ● Policy or programming changes related to research that did not include ERT Evaluation Question 4 EQ4.0 What are the most significant early results of LASER? How have research results differed depending on whether activities were core-funded or buy-in investments? ● Research products developed ● Capacity of researchers increased (subgroups of interest: Women, LMIC) ● Research findings disseminated ● Research findings influence programming/policy ● HEI network size ● HEI network engagement ● Relationships built between HEIs and policymakers (subgroup of interest: women, LMIC) ● Comparison of early results of core research vs. buy-ins EQ4.1 How does LASER compare to other international research programs in terms of the co￾creation/scoping process, researcher application process, training/capacity building, and research relevance? ● Efficiency of management ● Client satisfaction ● Research product utilization ● Research product dissemination ● Timeliness of research (results available when needed for utilization) ● Access to high-quality researchers ● Access to diverse researchers ● Ease of use for clients ● Ease of use for researchers ● Researcher capacity building 4.2 Where/in what ways did LASER most stimulate engagement and growth for international research for development in terms of institutions, networks, and researchers, including engaging women and Low-and Middle-Income Country (LMIC) researchers? ● LASER-PULSE network size (by women, LMIC) ● LASER-PULSE network engagement (by women, LMIC) through trainings, responses to RFAs, utilizing networking features of websites, etc. ● # of RFA applications (by women, LMIC) ● # of R4D convening participants (by women, LMIC) ● # of participants in capacity building trainings (by women, LMIC) ● Reasons for engagement ● Barriers to engagement Evaluation Question 5 76 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV EQ5.0 To what extent have the adaptations made by the LASER program to-date, particularly those made in response to COVID-19, addressed challenges and increased the likelihood of meeting program goals and objectives? ● Adaptations to RFA process ● Adaptations to Mission buy-ins ● Adaptations to R4D Convenings ● Adaptations to capacity building trainings and resources ● Adaptations to CSFA/ERT process ● Buy in timeline, pre-and post COVID ● Buy-in outputs, pre-and post COVID ● Buy-in budget fidelity, pre-and post COVID ● RFA timeline, pre-and-post COVID ● RFA outputs, pre-and-post COVID EQ5.1 How did the LASER program adapt to COVID-19? How did these changes affect early results? ● Adaptations to RFA process due to COVID-19 ● Adaptations to Mission buy-ins due to COVID-19 ● Adaptations to R4D Convenings due to COVID-19 ● Adaptations to capacity building trainings and resources due to COVID-19 ● Adaptations to CSFA/ERT process due to COVID-19 ● IP and AOR perception of adaptation impacts on early results ● USAID M/B/IO perception of adaptation impacts on early results EQ5.2 What is the likelihood of LASER meeting final program goals? What changes can USAID and the implementing partner make in the remaining time to ensure the project meets its goals and objectives? ● % of MEL indicators at or exceeding targets ● Recommendations from LASER leadership ● Recommendations from Missions ● Recommendations from researchers ● Recommendations from boundary partners 3. DATA SOURCES, SAMPLING, AND ANALYSIS To answer the research questions, the evaluation will use eight sources of qualitative and quantitative data. Our evaluation data sources include (1) USAID/M/B/IO Quantitative Survey, (2) LASER-PULSE Research Network Survey, (3) LASER-PULSE Core Team Interviews, (4) RFA KIIs, (5) Buy-In KIIs, (6) R4D KIIs, (7) Comparison Mechanism KIIs, and (8) a comprehensive desk review of reports, project documents, and research products. This section provides a detailed description of the data sources, sampling plan, and data collection methods. Table 3 summarizes the data sources used to address each of the evaluation questions. Table 3. Data Source Crosswalk 77 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 3.1. USAID M/B/IO QUANTITATIVE SURVEY (N=52) The USAID M/B/IO Survey will capture the experiences of USAID M/B/IO staff who participated in an RFA development process, an R4D Convening, a buy-in, or a combination of the three. The survey includes separate modules to measure key indicators for each of the three components: RFA process, R4D Convenings, and Buy-Ins. Each respondent will respond to the module(s) that are relevant to their LASER experience. All respondents will also answer a module on their experience soliciting research from other USAID mechanisms. In the RFA module, we will collect information on whether the RFA process translated into new relationships, locally-relevant research, or research utilization, as well as Mission satisfaction with the quality, timeliness, and usefulness of the research produced. The ET will be able to compare responses from different RFA processes to see if changes to the RFA development process, such as involving the Mission in the co-creation of the SOW, have made the RFAs more likely to achieve their desired approach. 78 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV In the Buy-In module, respondents will be asked about their satisfaction with the quality, timeliness, and usability of the research results, as well as if any new relationships or research ideas have come from the buy-in. In the R4D module, respondents will be asked about benefits to the Mission from participating in R4D convenings, including finding new partners and developing new research questions. The R4D module also asks about the usefulness of the CSFA and ERT processes in developing locally-led research priorities. All respondents will be asked about other USAID mechanisms they have procured research from, and how LASER compares on quality and diversity of researchers, ease of procuring research services, timeliness of research, and usability of research. The survey instrument can be found in Annex 3. Sample: The M/B/IO Survey will be sent to USAID/M/B/IO staff who participated in an RFA development process, attended an R4D Convening, or commissioned a buy-in. Names and contact information for USAID/M/B/IO staff will be collected with support from the LASER-PULSE consortium. Response rates are based on past experiences surveying USAID staff. A total sample of 52 is the target for the survey of USAID/M/B/IO staff; the breakdown in the sample is shown in the table below. The survey will be sent to a total of 21 individuals, or three Mission staff for each of the seven RFAs that have taken place or are in progress. A response rate of 67% is assumed reasonable, giving a sample size of 14. For USAID staff involved in any R4D convening, the survey will be sent to 125 individuals (25 from each of the five convenings) and a lower response rate of 20% is assumed for a total sample of 25. Finally, the survey will be sent to 26 USAID contacts involved in a LASER buy-in, two for each of the 13 buy-ins. For this group, a 50% response rate is assumed for a sample of 13. The difference in assumed response rates between groups is based on previous experience with surveys of USAID staff members and depends on the level of engagement of the group. Sample Assumptions N USAID/M/B/IO staff who were involved in any RFA process Assumes three Mission staff for each of the seven RFAs that have taken place or are in progress (21 total), 67% response rate. 14 USAID/M/B/IO staff who were involved in any R4D Convening Assumes 25 USAID/M/B/IO staff for each of the five convenings (125 total). 20% response rate 25 USAID/M/B/IO staff who were involved in a LASER buy-in Assumes two USAID/M/B/IO contacts for each of the 13 buy-ins, 50% response rate 13 Total 52 Data collection: Per USAID regulations (ADS 508, 505, and 551), the USAID M/B/IO Survey will require clearance from IRD, Privacy, and Section 508 teams before we can send the survey to USAID 79 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV staff. This process is expected to take two to four weeks and will require the COR to submit the required documentation on behalf of the team. Once approved, the ET will utilize a three-step online survey distribution method (email invitation, first email reminder, and a second final invitation to non-respondents). Before the start of survey distribution, the ET will also seek help from USAID/ITR/R and the IP to send an introductory email to all potential respondents to alert them about the upcoming survey. The goal is to maximize responses by sending two reminders to each survey recipient. Each survey will be designed to be short (10–15 minutes). The surveys will be administered via the Qualtrics online surveying platform. If USAID/M/B/IO staff have left the Mission but remain at USAID, they will still be sent the survey. If they have left USAID, the ET will not be able to survey them. This has been taken into account in the estimated sample sizes above and is not anticipated to affect the total sample size. Data analysis: Descriptive statistics will be used to analyze the survey results. 3.2. LASER-PULSE NETWORK QUANTITATIVE SURVEY (N=437) The LASER-PULSE Network survey will collect the perspectives of researchers who have had a variety of types of engagement with the research network. Like the survey of USAID M/B/IO staff, the survey of researchers will include modules on the RFA process, R4D convenings, and buy-Ins. In addition, there will be questions on engagement with the LASER network and comparative mechanisms that will go to all network researchers, including those who do not engage in RFAs, R4Ds, or buy-Ins. Each respondent will answer the modules that are relevant to their LASER experience. The RFA module includes questions on whether the process translated into new relationships, locally relevant research and utilization, as well as researcher satisfaction with their experience. The ET will be able to compare responses from different RFA processes to see if changes in the process altered researcher experience and perception of outcomes. Researchers who were involved in an R4D convening will respond to a module that asks whether the convening helped them to find new research partners, learn new skills, and develop new research questions. Researchers working on buy-ins will reply to a module that addresses what they gained from the buy-in, including changes in research quality, research translation, and usability. All modules include questions addressing the usefulness of the CSFA and ERT processes in developing locally-led research priorities. Demographic questions will allow the team to compare outcomes between female and male researchers and between LMIC and HIC researchers. The survey instrument can be found in Annex 4. Sample: The LASER-PULSE network survey will be sent to all ~2,500 members of the network. The target sample size for the entire network is calculated using the below equation, in which z is the z￾score, p is the population, and e is the margin of error. With a population of more than 2,400 researchers (as stated by the LASER website), a confidence level of 95%, and margin of error of 5%, the target sample will be 332 respondents. This is a 14% response rate and is feasible. Respondents above this level will yield a higher confidence level and lower margin of error. 80 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV The ET is aware that the number of researchers who have participated in LASER R4D, RFA, or buy-Ins is much smaller than the number of researchers in the network. For that reason, researchers who participated in R4D convenings, applied for subgrants through an RFA process, or were part of a buy-in will receive personalized email invitations to take the survey. We expect a response rate of 50% for respondents who are personally contacted. Sample Assumptions N All LASER-PULSE members 2,500 members, 14% response rate 332 LASER-PULSE members who were involved in any R4D Convening Assumes 25 researchers for each of the 5 convenings (125 total), 50% response rate 62 LASER-PULSE members who were involved in an RFA Assumes 4 researchers for each of the 15 sub￾grants (60 total), 50% response rate 30 LASER-PULSE members who were involved in a LASER buy-in Assumes 2 respondents for each of the 13 buy-ins (30 total), 50% response rate 13 Total 437 Data collection: Names and contact information for LASER-PULSE Network researchers will be collected with support from the LASER-PULSE consortium through their website. The survey will be distributed through Qualtrics and include up to three reminders. Responses will be monitored on a rolling basis and, if necessary, the team will make additional efforts through personalized email outreach to respondents to ensure a sample that reflects the views of LMIC and female researchers. Researchers who were involved in an R4D Convening, an RFA, or a buy-in will receive a personalized invitation sent through their primary LASER POC to encourage their participation in the survey. Data analysis: Descriptive statistics will be used to analyze the survey results. We will disaggregate and compare the data across the following subgroups: female and male researchers and LMIC and HIC researchers. 3.3. LASER-PULSE LEADERSHIP KII (N=7) KIIs with USAID LASER AORs and LASER leadership will collect information about the strengths and weaknesses of the LASER design, the theory of change, and elements that have contributed or hindered implementation. They will also address ways that LASER stimulated engagement and growth for international research, ways that LASER adapted to COVID-19, and the likelihood of LASER achieving its program goals. The KII Guide can be found in Annex 5. Sample: The ET will conduct KIIs with LASER’s past and current AORs at USAID/ITR/R, as well as LASER’s leadership team, which consists of the Principal Investigator, Program Director, Academic Director, and Technical Director. 81 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Sample Assumptions N LASER AORs Past and present AORs 3 LASER Leadership Principal Investigator, Program Director, Academic Director, and Technical Director 4 Data Collection: These KIIs will be conducted as two group KIIs, one with USAID/ITR/R and one with LASER leadership. The interviews will be conducted virtually using a videoconferencing platform (e.g., Zoom, Google Meet, or similar); there will be no travel or on-site visits. Interviews will be recorded and transcribed for analysis. Interviews are anticipated to last roughly an hour. Data Analysis: Content analysis and Fidelity Analysis will be used to analyze the Leadership KIIs. Coding will be conducted using Nvivo to conduct analysis of patterns, code co-occurrence matrices, extract quotes, and run other custom queries. Analysis will involve reading and re-reading the transcripts and carefully coding and analyzing data according to queries that are designed to correspond directly to the evaluation questions for this evaluation, as well as subgroup analyses. 3.4. RFA KII (N=4) AND FGDS (N=2) To understand how the RFA process and the grants awarded through RFAs have contributed to LASER’s objectives and early results, KIIs and FGDs discussions will be held with LASER Activity Managers and USAID Mission staff who lead the development of the RFA solicitation and the review of applications, as well as FGDs with the winners of subgrants from the Colombia and Global RFAs. The LASER Activity Manager and USAID Mission staff KIIs will answer questions about the successes and challenges of the RFA process, particularly as it related to the application of CSFA and ERT, how the co-creation and scoping process contributed to achieving LASER results, and how the RFA subgrants contributed to achieving LASER results. The KIIs will also examine how the RFA process engages women and LMIC researchers. Researchers from the subgrants will be asked questions about how they learned about the RFA and why they applied, how ERT and CSFA were applied in their proposed approaches, outcomes of their research, including new research questions, partnerships, and policies. They will also discuss how they adapted their research to COVID-19, and how the LASER RFA process compares to other USAID research mechanisms they have received funding from if applicable. Particular attention will be paid to the responses of women researchers and researchers from LMIC. The KII Guide can be found in Annex 6. Sample: For each site, the ET will conduct KIIs with the LASER Activity Manager and USAID Mission staff who were the primary organizer of the RFA solicitation and selection process in Colombia and the Global RFA. To capture the perspective of the RFA process from the perspective of the researchers, the ET will conduct two FGDs, one with subgrant winners from the Ethiopia RFA and one with subgrant winners from the Colombia RFA. The ET will ask the LASER team for assistance in identifying and contacting the correct personnel. Sample Assumptions N, N, Global 82 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Colombia LASER Activity Manager Assumes a different activity manager for each site 1 1 USAID Mission staff involved in RFAs 1-2 POCs per Mission will participate in single KII per site 1 1 FGD 3-5 winners of subgrants in Global Round and Colombia will take part in one FGD per site 1 1 Total 3 3 Data Collection: The interviews will be conducted virtually using a videoconferencing platform (e.g., Zoom, Google Meet, or similar); there will be no travel or on-site visits. Interviews will be recorded and transcribed for analysis. Interviews are anticipated to last roughly an hour. Data Analysis: Content analysis and coding will be conducted using Nvivo to conduct analysis of patterns, code co-occurrence matrices, extract quotes, and run other custom queries. In addition to content analysis, we will use Most Significant Change to identify what components of the RFA process are most impactful for meeting LASER’s objectives, based on the perspectives of all stakeholders interviewed. Analysis will involve reading and re-reading the transcripts and carefully coding and analyzing data according to queries that are designed to correspond directly to the evaluation questions for this evaluation, as well as subgroup analyses. We will disaggregate findings by the Colombia and Global RFA processes, as well as by women and LMIC researchers. 3.5. R4D CASE STUDY KIIS (N=28) R4D convening KIIs will focus on whether and how the convening supported the development of new or existing research. Specifically, interviewees will be asked about the benefits of attending the convening, as well as whether and how the convening facilitated any new relationships, new research ideas, or policy. In addition, the guide also asks interviewees to reflect on the impact of the COVID-19 pandemic on the experience of the convening and whether and how the CSFA/ERT processes at the convening supported their work and initial results. The KII Guide can be found in Annex 7. Sample: The R4D KIIs will be conducted across two case studies, each centering on an R4D convening. One case study will be the Colombia convening, which was held in-person and earlier in the contract. The second case study will be the East Africa R4D convening, which was held virtually (due to the COVID-19 pandemic) and later in the contract, with sub-awards at earlier stages. Each case study will include interviews with the USAID LASER Activity Manager, IP LASER Activity Manager, two participants from the host mission (i.e., USAID/Colombia or USAID/KEA three researcher participants, and three boundary partner participants. 83 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Sample Assumptions N, Colombia N, East Africa USAID LASER Activity Manager Assumes the USAID LASER Activity Manager differed per convening 1 1 IP LASER Activity Manager Assumes the LASER Activity Manager differed per convening 1 1 USAID Mission participants Assumes one interview with 2-3 Mission participants 2-3 2-3 Researcher participants Assumes 2 interviews with 3-4 researchers each 6-8 6-8 Boundary partner participants Assumes 2 interviews with 3-4 boundary partners each 6-8 6-8 Total 16-21 16-21 Data Collection: The interviews will be conducted virtually using a videoconferencing platform (e.g., Zoom, Google Meet, or similar); there will be no travel or on-site visits. Interviews will be recorded and transcribed for analysis. Interviews are anticipated to last roughly one hour. Data analysis: An evaluative case study approach will be used to analyze the R4D KIIs. In addition to evaluative case studies, we will use Most Significant Change to identify what components of the R4D Convenings are most impactful for meeting LASER’s objectives, based on the perspectives of all stakeholders interviewed, including differences between in-person and virtual formats. Coding will be conducted using Nvivo to conduct analysis of patterns, code co-occurrence matrices, extract quotes, and run other custom queries. Analysis will involve reading and re-reading the transcripts and carefully coding and analyzing data according to queries that are designed to correspond directly to the evaluation questions for this evaluation, as well as subgroup analyses. 3.6. BUY-IN KIIS (N=24) KIIs with those involved in projects established through the LASER Buy-In mechanism will highlight the experience of these individuals through four case studies. Specifically, the interviews will explore the challenges and successes of the LASER buy-in mechanism, CSFA/ERT effectiveness, early results, and changes/adaptations due to the COVID-19 pandemic, including LASER support of these adaptations. The interviews will also ask about the participant’s experience with the buy-in process as a whole, benefits and drawbacks, suggestions for improvement, including the engagement of LMIC/women researchers and engagement with the LASER network. The KII Guide can be found in Annex 8. Sample: The Buy-In KIIs will be conducted across four case studies. The case studies will be selected based on the results of both the USAID M/B/IO survey and the LASER-PULSE Network Quantitative Survey. Two of the case studies will be chosen based on greater/lesser ERT assistance, a third case 84 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV study will be chosen based on strong women/LMIC engagement, and fourth based on deep engagement with local stakeholders and policy. Each case study will include interviews with the Mission POC, LASER Activity Manager, two members of the research team, and two boundary partners, if relevant. Sample Assumptions N per case study N for all four case studies Mission POC Assumes one Mission POC per buy-in case study 1 4 LASER Activity Manager Assumes one LASER activity manager per buy-in case study 1 4 Research Team Members Assumes at least two research team members per buy-in 2 8 Boundary Partners (if relevant) Assumes at least two boundary partners per buy-in 2 8 Total 6 24 Data collection: The interviews will be conducted virtually using a videoconferencing platform (e.g., Zoom, Google Meet, or similar); there will be no travel or on-site visits. Interviews will be recorded and transcribed for analysis. Interviews are anticipated to last roughly one hour. Data analysis: An evaluative case study approach will be used to analyze the Buy-In KIIs. Coding will be conducted using Nvivo to conduct analysis of patterns, code co-occurrence matrices, extract quotes, and run other custom queries. Analysis will involve reading and re-reading the transcripts and carefully coding and analyzing data according to queries that are designed to correspond directly to the evaluation questions for this evaluation. 3.7. COMPARISON MECHANISM KII (N=4) KIIs with the CORs of four research mechanisms will explore how each mechanism compares to LASER through the co-creation/scoping process, researcher application process, training/capacity building, research quality, and research relevance. The KII Guide can be found in Annex 9. Sample: The Comparison KIIs will be conducted with USAID CORs from each of the four selected mechanisms. Each mechanism was chosen to speak to a specific aspect of the LASER mechanism. ● The Partnership for Enhanced Engagement in Research (PEER) has been selected due to its similar program objectives to LASER, including supporting global research collaborations, particularly with women, and strengthening research translation into policy. 85 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ● The Higher Education Solutions Network (HESN), will speak to building networks and relationships between HEIs, development professionals, and local actors. ● The Research Technical Assistance Center (RTAC) is a global network of academic researchers similar to LASER that provides timely and on-demand research expertise for USAID. RTAC was selected because of its task of establishing, expanding, and maintaining a global Research Network of university-based scholars, creating and maintaining a website, and developing training. ● The Learning, Education, and Research (LER) II Task Order, a five-year Activity designed to advance learning activities on DRG activities to assist with decision making. This global activity consists of individual buy-in taskings from Missions and the DRG Office and will speak to the challenges and successes of Mission buy-ins, and the benefits and disadvantages of a stand-alone buy-in mechanism, as well as the benefits and challenges of a sector-specific mechanism. Sample Assumptions N USAID AOR - PEER Current AOR 1 USAID AOR - HESN Current AOR 1 USAID COR - RTAC Current COR 1 USAID COR - LER II Current COR 1 Total Current COR or AOR 4 Data Collection: The interviews will be conducted virtually using a videoconferencing platform (e.g., Zoom, Google Meet, or similar); there will be no travel or on-site visits. Interviews will be recorded and transcribed for analysis. Interviews are anticipated to last roughly 30 minutes. Data analysis: Content analysis will be used to analyze the Comparison Mechanism KIIs. Coding will be conducted using Nvivo to conduct analysis of patterns, code co-occurrence matrices, extract quotes, and run other custom queries. Analysis will involve reading and re-reading the transcripts and carefully coding and analyzing data according to queries that are designed to correspond directly to the evaluation questions for this evaluation, as well as subgroup analyses. 3.8 RISKS AND LIMITATIONS In addition to general risks such as limited time and the availability of proposed respondents, critical to the quality of the analysis is the willingness of respondents to openly and honestly discuss issues and share information and insights. The ET will make every effort to build a strong rapport with respondents by explaining the purpose of the study and choosing a private and comfortable environment to complete the interviews. If respondents are concerned about how results will be used, despite the full disclosures and guarantees given in the consent statement before a survey, the risk exists that respondents may limit their answers. In addition, there are risks and limitations specific to conducting interviews remotely. Due to the nature of the sample, it is unlikely that there will be systematic bias in access to technology that allows for remote interviews. This is particularly true after two years of the COVID-19 pandemic, as many 86 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV institutions and organizations have been working remotely for all or part of that time. The ET is highly experienced in conducting remote interviews and establishing the rapport necessary to put the interviewee at ease and allow for a successful interview. There is also a risk of positive bias in reports of the strengthening of LASER design and utilization. Interviewees know that positive results from the LASER mechanism is desired, and in some cases required, and may be more likely to overstate success and understate challenges. To combat this, the interviewer will stress the need for accuracy and encourage honest feedback wherever possible. In addition, triangulation with available documentation will be used to verify findings. The ET recognizes that with a relatively limited pool of respondents, the potential for outliers to skew aggregate results is greater. This is exacerbated if participation rates are low, particularly among LMIC and female researchers. To bolster participation, the ET will engage USAID to introduce the survey and also send a follow-up email encouraging participation. A final limitation is the short time for outcomes to accrue since the start of the LASER activity, both due to the midterm nature of the evaluation and the implementation delays necessitated by COVID-19. 4. ADMINISTRATIVE 4.1 ROLES AND RESPONSIBILITIES Roles and responsibilities for each ET member are described below. Roles and responsibilities are designed to build on each team member’s specific expertise, and roles are clearly differentiated to achieve efficiency and avoid duplication of efforts. Table 3. Evaluation Team Roles and Responsibilities Name Role Responsibility Key Personnel Heather Huntington, PhD Team Lead Dr. Huntington will lead the design of the evaluation methodology, author the evaluation design report, contribute to the workplan, and lead the design of the survey instruments. She will oversee the Evaluation Methods Specialist, Qualitative Methods Specialists, and Data Analyst during qualitative data collection and analysis, be the lead author on the findings report, and lead each presentation. Aleta Starosta Evaluation Methods Specialist Ms. Starosta will author the work plan, support the evaluation design report, lead the document review and qualitative interviewing and coding, and assist the Team Lead in analyzing and triangulating data across sources, drafting 87 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV reports, and disseminating findings in presentations, workshops, and briefs. Ryan Hatano Data Analyst Mr. Hatano will support qualitative and quantitative data collection and analysis. He will have primary responsibility for implementing the survey, including programming the online survey software, developing and cleaning the sampling frame, and promoting strong response rates. Mr. Hatano will work under the guidance of Dr. Huntington and prepare descriptive statistics and data visualizations for the final report and presentations. Other Team Members Jessica Sperling, PhD Qualitative Methods Specialist Dr. Sperling will provide guidance on qualitative and evaluative methods to qualitative specialist Noelle Wyman Roth and the broader team. Noelle Wyman Roth Qualitative Methods Specialist Ms. Wyman Roth will lead the design of the qualitative data collection, support qualitative data collection and analysis, including document review, conduct qualitative interviewing and coding in Nvivo, and contribute to written deliverables. Miriam Counterman, PhD Quantitative Methods Specialist Dr. Counterman will contribute to the design of quantitative data collection and lead KIIs and FGDs. She will lead the quantitative survey sampling and execution, analyze findings in R, and contribute to written deliverables. Patrick Lohmeyer Task Order Manager Mr. Lohmeyer will be the primary point-of-contact for USAID on award management, oversee the LINC consortium, and manage the schedule, financial management, and contractual compliance. 4.2 COMMUNICATIONS OUTREACH PLAN The table below shows how the ET intends to reach out to each of the key stakeholder groups who will be involved in the evaluation to inform them about the evaluation and encourage their participation in the research. The team will require a letter of introduction from USAID/ITR/R, which the ET will draft. The team will also require LASER staff assistance procuring contact information and, in some cases, introductions to stakeholders. We will do our best to minimize the burden to the IP and 88 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV USAID/ITR/R by drafting all introduction correspondence, assuming communication responsibilities once introductions are made, and attempting to find all contact information ourselves when possible. Table 4. Communications Outreach Plan Stakeholder Method Who When LASER Staff Introduction by USAID/ITR/R staff, followed by a presentation of the design report USAID/ITR/R (introduction) ET (presentation) January 2022 USAID/M/B/IO Survey Participants Email introduction of the survey and its purpose USAID/ITR/R March 2022 LASER-PULSE Survey Participants Email introduction of the survey and its purpose LASER Network Activity Manager March 2022 USAID/Colombia Email introduction of the evaluation and its purpose, as well as requested contribution Mission Briefing USAID/ITR/R ET February 2022 March 2022 USAID/KEA Email introduction of the evaluation and its purpose, as well as requested contribution Mission Briefing USAID/ITR/R ET February 2022 March 2022 RFA subgrant award recipients, Colombia Email introduction of the evaluation and its purpose, as well as requested contribution LASER RFA Activity Manager March 2022 89 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV RFA subgrant award recipients, Global Email introduction of the evaluation and its purpose, as well as requested contribution LASER RFA Activity Manager March 2022 Buy-In Case Study participants Email introduction of the evaluation and its purpose, as well as requested contribution Letter of introduction from USAID for boundary partners LASER Buy-In Activity Manager USAID/ITR/R May 2022 R4D Case Study participants Email introduction of the evaluation and its purpose, as well as requested contribution Letter of introduction from USAID for boundary partners LASER R4D Activity Manager USAID/ITR/R May 2022 4.3 DISSEMINATION PLAN The main audience for this evaluation is USAID/ITR/R and the LASER consortium. Due to potential procurement sensitivities, dissemination with wider audiences may not be appropriate, but will be discussed with the ITR/R team once the report is final. The primary dissemination products are described below. Inception Presentation: The ET will present document review findings to USAID DDI/ITR/R staff after the completion of the desk review, as well as present any other key updates from the evaluation to-date. This will take place in March 2022. M/B/IO Briefings: The ET will meet virtually with site visit partners to introduce the evaluation and discuss the team’s assignment, initial assumptions, evaluation questions, methodology, and work plan. These are planned to take place in March 2022. Midterm Briefing: The ET will hold a mid-term briefing with USAID DDI/ITR/R staff in approximately July 2022 on the status of the evaluation, data collection, and potential challenges. This will include a discussion of findings to-date and how they will inform follow-up interviews. It will also be a platform to validate initial findings and solicit input on potential recommendations. Evaluation Report. After all research phases are complete, the ET will draft a 35–50 page evaluation report that is consistent with the guidance provided in Annex 10, Final Report Format. The report will address each of the questions identified in the SOW and any other issues the team considers having a 90 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV bearing on the objectives of the evaluation.25 Once the initial draft evaluation report is submitted and the recommendations workshop (detailed below) is complete, USAID/LASER will have 15 business days in which to review and comment on the initial draft, after which point the COR will submit the consolidated comments to the ET. A final report will be submitted within ten business days of receiving comments. Once the final report is approved, it will be made 508 compliant and posted to the DEC. Recommendations Workshop. Between the first and final draft of the evaluation report, the ET will hold a recommendations workshop for USAID DDI/ITR/R and IP staff. During the workshop, the ET will share a summary of findings and conclusions with the USAID team and discuss preliminary evaluation recommendations. The ultimate objective of the workshop is to validate the study findings and solicit feedback on the initial recommendations. Suggestions from the recommendations workshop will be incorporated into the second draft of the report, along with comments from the USAID/LASER team. Evaluation Brief and Fast Facts Presentation. After the ET drafts the evaluation report, they will also draft an evaluation brief that summarizes the evaluation findings for non-technical audiences. The brief will be 1-2 pages long and employ 1-2 graphics. The ET will also create a ten-slide Fast Facts presentation to highlight summary findings for secondary external audiences. For both the Evaluation Brief and Fast Facts presentation, the ET will respond to one round of comments from the USAID team. The final approved versions of each deliverable will be made 508 compliant and posted to the Development Experience Clearinghouse (DEC). Final Presentations. The ET will hold two final presentations after the report is final through Google Meet to discuss findings, conclusions, and recommendations. Potential audiences for each presentation include USAID DDI/ITR/R, LASER staff, participants in case study interviews, and/or members of the LASER PULSE network, as agreed upon with USAID. Each presentation will be tailored to the audience, with customized slide decks and handouts. Post-Evaluation Action Plan. After the report is finalized, the ET will develop a draft Post￾Evaluation Action Plan, as required by the ADS 201.3.5.18 (A), with the intent that USAID DDI/ITR/R team will complete the Plan and develop initial next steps for implementing the accepted recommendations. The table below lists major dissemination activities, their platform and target audience, when the activity will take place, and who will lead each activity. Table 5. Dissemination Plan Dissemination Activity Platform/Venue Target Audience When Inception Presentation 1-hour virtual presentation USAID/ITR/R March 2022 USAID M/B/IO Briefings 1-hour virtual presentation USAID/KEA USAID/Colombia March 2022 25 Any such issues will only be included in the report after consultation with USAID 91 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Mid-term briefing 1-hour virtual presentation USAID/ITR/R July 2022 Recommendations workshop 2-hour virtual workshop USAID/ITR/R LASER staff January / February 2023 Evaluation report 35-50 page report posted to the DEC USAID/ITR/R LASER staff March 2023 Evaluation Brief 2-page infographic posted to the DEC Secondary external audiences, including USAID Missions, boundary partners, and HEIs March 2023 Fast Facts Presentation PowerPoint Deck Secondary external audiences, including USAID Missions, boundary partners, and HEIs March 2023 Final Presentations (2) 1-hour virtual presentation USAID/ITR/R LASER staff April 2023 Post-Evaluation Action Plan (PEAP) Draft PEAP template shared for internal use USAID/ITR/R April 2023 4.4 DIVERSITY, EQUITY, AND INCLUSION Throughout the evaluation, the LINC Consortium will ensure researchers from LMICs as well as USAID Foreign Service Nationals (FSNs) are included in the sampling plan. Data will be disaggregated by LMIC/High Income Country (HIC) status whenever possible. In particular, the evaluation design, methodology, data collection, analysis, and reports will capture the situations and experiences of LMIC and HIC researchers engaging in LASER activities. The ET will use qualitative and quantitative questions that can identify both positive and negative unintended consequences for all researchers. If disproportionate participation in data collection arises between LMIC and HIC researchers, the ET will explore potential explanations through the qualitative research The team will also pay close attention to the representation of women and minority researchers engaged in LASER activities and disaggregate data by gender and race, when applicable. Where data is available, the ET will examine the proportion of requests for proposals received from and awarded to women and minority researchers, and the gender breakdown of research teams on research produced through buy-ins. The team will also be sensitive to the various dynamics impacting how women and minority researchers engage with the LASER mechanism, such as inequalities in access through their home institutions and cultural factors, and the unique obstacles or opportunities they may face. 92 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 4.5 ETHICAL CONSIDERATIONS The evaluation will follow all ethical practices for human subjects research, protecting respondent confidentiality and sensitive information. Survey respondents will be guaranteed confidentiality and will be able to refuse to answer any of the questions. No findings will be attributed to respondents by name in the report. The ET will make all efforts to ensure that the process of conducting qualitative interviews/discussions will be as independent as possible. In particular, when conducting interviews/discussions, the representatives of the activity or project management will be invited to leave the room. In cases where this will not be possible, the circumstances of such situations will be noted in the final report as potential limitations and/or bias of the evaluation. ANNEX 1. EVALUATION MATRIX Research Question Indicators and Outcomes Data Sources Methods Analysis Plan Research Question 1 RQ1.0 What are the strengths and weaknesses of the LASER design? ● Research development ● Capacity building ● Demand-driven research ● LMIC and female researcher integration ● Research dissemination ● Research utilization ● Stakeholder engagement ● LASER leadership KIIs ● Annual reports and MEL plans ● Interviews ● Desk review ● Coding/content analysis RQ1.1 What evidence exists that the development hypothesis underlying LASER’s theory of change (TOC) is valid in practice? How might the TOC be adapted to account for early evidence on these development hypotheses? ● Development of locally-relevant research ● Quality and uptake of capacity building ● Extent and quality of research translation ● Extent and strength of research networks ● Quality of research products ● KIIs with LASER Core team ● Annual reports and MEL plans ● Interviews ● Desk review ● Coding/content analysis RQ1.2 What elements of the design of LASER have contributed vs. hindered implementation? ● % of deliverables met on time ● Percent of MEL targets achieved ● Implementation successes ● Implementation challenges ● LASER leadership KIIs ● RFA KIIs ● Buy-In KIIs ● R4D KIIs ● Annual reports and MEL plans ● Interviews ● Desk review ● Case Study ● Coding/content analysis ● Fidelity analysis 93 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Research Question 2 RQ2.0 What are the challenges and successes of LASER M/B/IO buy-ins, particularly in terms of scope development, mission engagement, management, and collaboration with academic partners? ● Efficiency of management ● Client satisfaction ● Research product utilization ● Research product dissemination ● Timeliness of research (results available when needed for utilization) ● Access to high-quality researchers ● Access to diverse researchers ● USAID M/B/IO Survey ● Buy-In KIIs ● Comparison mechanism KIIs ● Survey ● Case Study ● Interviews ● Descriptive statistics ● Coding/context analysis RQ2.1 To what extent has the buy-in process and research product quality changed and/or improved over time? ● Change in efficiency of management ● Change in client satisfaction ● Change in research product utilization ● Change in research product dissemination ● Change in timeliness of research (results available when needed for utilization) ● Change in access to high-quality researchers ● Change in access to diverse researchers ● USAID M/B/IO Survey ● LASER￾PULSE network survey ● Buy-In KIIs ● Desk review of buy-in research products ● Survey ● Case Study ● Interviews ● Descriptive statistics ● Coding/context analysis Research Question 3 94 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV RQ3.0 To what extent have Comprehensive Success Factors Analysis (CSFA) and Embedded Research Translation (ERT) been effective in helping stakeholders to build relationships with USAID and each other, generate locally relevant research, and set the foundation for policy uptake? ● New relationships established through CSFA ● New relationships established through ERT ● New relationships sustained after CSFA ● New relationships sustained after ERT ● New research generated through CSFA ● New research generated through ERT ● Policy or programming changes attributed to CSFA ● Policy or programming changes attributed to ERT ● USAID M/B/IO Survey ● LASER￾PULSE network survey ● RFA KIIs ● R4D KIIs ● Buy-In KIIs ● Desk review of CSFA and ERT materials and Annual Reports ● Survey ● Case Study ● Interviews ● Desk Review ● Descriptive statistics ● Coding/context analysis RQ3.1 What lessons can be drawn from LASER's experience with CSFA and ERT to improve the design of future research activities? ● New relationships established through CSFA ● New relationships established through ERT ● New relationships sustained after CSFA ● New relationships sustained after ERT ● New research generated through CSFA ● New research generated through ERT ● Policy or programming changes attributed to CSFA ● Policy or programming changes attributed to ERT ● USAID M/B/IO Survey ● LASER￾PULSE network survey ● RFA KIIIs ● R4D KIIs ● Buy-In KIIs ● Desk review of CSFA and ERT materials and Annual Reports ● Survey ● Case Study ● Interviews ● Desk Review ● Descriptive statistics ● Coding/context analysis RQ3.2 What can be learned from differences in the success of projects that received or did not receive deep ERT assistance and integration? ● New relationships established through CSFA ● New relationships established through ERT ● New relationships sustained after CSFA ● New relationships sustained after ERT ● New research generated through CSFA ● RFA KIIs ● R4D KIIs ● Buy-In KIIs ● Case Study ● Interviews ● Coding/context analysis 95 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ● New research generated through ERT ● Policy or programming changes attributed to CSFA ● Policy or programming changes attributed to ERT Research Question 4 RQ4.0 What are the most significant early results of LASER? How have research results differed depending on whether activities were core￾funded or buy-in investments? ● Research products developed ● Capacity of researchers increased (subgroups of interest: Women, LMIC) ● Research findings disseminated ● Research findings influence programming/policy ● HEI network size ● HEI network engagement ● Relationships built between HEIs and policymakers (subgroup of interest: women, LMIC) ● LASER Leadership KIIs ● RFA KIIs ● Buy-In KIIs ● R4D KIIs ● Annual reports and MEL documents ● Interviews ● Case Study ● Desk review ● Most significant change ● Coding/context analysis RQ4.1 How does LASER compare to other international research programs in terms of the co- creation/scoping process, researcher application process, training/capacity building, and research relevance? ● Efficiency of management ● Client satisfaction ● Research product utilization ● Research product dissemination ● Timeliness of research (results available when needed for utilization) ● Access to high-quality researchers ● Access to diverse researchers ● Ease of use for clients ● Ease of use for researchers ● Researcher capacity building ● Comparison Mechanism KIIs ● USAID M/B/IO Survey ● LASER-PULSE Network Survey ● Evaluations and reports of comparison mechanisms ● Interviews ● Survey ● Desk review ● Coding/context analysis ● Descriptive statistics 4.2 Where/in what ways did LASER most stimulate engagement and growth for international research for development in terms of institutions, networks, and researchers, including engaging ● LASER-PULSE network size (by women, LMIC) ● LASER-PULSE network engagement (by women, LMIC) ● # of RFA applications (by women, LMIC) ● Annual reports, MEL reports ● LASER-PULSE Network Survey ● RFA KIIs ● R4D KIIs ● Desk review ● Interviews ● Survey ● Case Study ● Most significant change ● Coding/context analysis ● Descriptive statistics 96 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV women and Low- and Middle￾Income Country (LMIC) researchers? ● # of R4D convening participants (by women, LMIC) ● # of participants in capacity building trainings (by women, LMIC) ● Reasons for engagement ● Barriers to engagement ● Buy-in KIIs Research Question 5 RQ5.0 To what extent have the adaptations made by the LASER program to-date, particularly those made in response to COVID-19, addressed challenges and increased the likelihood of meeting program goals and objectives? ● Adaptations to RFA process ● Adaptations to Mission buy-ins ● Adaptations to R4D Convenings ● Adaptations to capacity building trainings and resources ● Adaptations to CSFA/ERT process ● LASER Leadership KIIs ● R4D KIIIs ● Buy-in KIIs ● RFA KIIs ● Annual reports and MEL documents ● Interviews ● Case Study ● Desk review ● Coding/context analysis ● Descriptive statistics ● Fidelity analysis RQ5.1 How did the LASER program adapt to COVID-19? How did these changes affect early results? ● Adaptations to RFA process due to COVID-19 ● Adaptations to Mission buy-ins due to COVID-19 ● Adaptations to R4D Convenings due to COVID-19 ● Adaptations to capacity building trainings and resources due to COVID-19 ● Adaptations to CSFA/ERT process due to COVID-19 ● Comparison Mechanism KIIs ● USAID M/B/IO Survey ● LASER-PULSE Network Survey ● Evaluations and reports of comparison mechanisms ● Interviews ● Survey ● Desk review ● Coding/context analysis ● Descriptive statistics ● Fidelity analysis RQ5.2 What is the likelihood of LASER meeting final program goals? What changes can USAID and the implementing partner make in the remaining time to ensure the project meets its goals and objectives? ● % of MEL targets on￾track ● Recommendations from LASER leadership ● Recommendations from Missions ● Recommendations from researchers ● Recommendations from boundary partners ● LASER core leadership KIIs ● RFA KIIs ● Buy-In KIIs ● R4D Case studies ● Annual reports, MEL plans ● Interviews ● Desk review ● Coding/context analysis ● Descriptive statistics 97 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 98 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX 2. KII LIST The KII list below shows the organization, position, name, and contact information for each intended qualitative respondent. Names and contact information will be updated on a rolling basis as the evaluation team works with the LASER-PULSE team to identify respondents. Data Sources Organization Position Name Contact information 1 LASER-PULSE Core Team KII USAID/ITR/R LASER AOR Brian Bingham bbingham@usaid.g ov 2 LASER-PULSE Core Team KII USAID/ITR/R LASER AOR Siena Fleischer sfleischer@usaid.g ov 3 LASER-PULSE Core Team KII USAID/ITR/R LASER AOR Kevin Roberts kroberts@usaid.go v 4 LASER-PULSE Core Team KII LASER-PULSE/Purdue Program Director Pallavi Gupta gupta604@purdue. edu 5 LASER-PULSE Core Team KII LASER-PULSE/Purdue Technical Director Betty Bugusu bbugusu@@purdu e.edu 6 LASER-PULSE Core Team KII LASER-PULSE/Purdue Business Manager Suzi Cyr skinman@purdue. edu 7 LASER-PULSE Core Team KII LASER-PULSE/Purdue Communication Specialist Jeff Goecker jgoecker@purdue. edu 8 LASER-PULSE Core Team KII LASER-PULSE/Purdue Research Project Manager Pamela McClure awardsmanager@l aserpulse.org 9 LASER-PULSE Core Team KII LASER￾PULSE/Makerere University Gender Specialist Ruth Nsibirano rnsibirano2015@g mail.com 10 LASER-PULSE Core Team KII LASER-PULSE/CRS Research Translation Advisor Laura Riddering laura.riddering@cr s.org 11 LASER-PULSE Core Team KII LASER-PULSE/Notre Dame M&E Specialist Frederick Rossi frossi2@nd.edu 99 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Data Sources Organization Position Name Contact information 12 LASER-PULSE Core Team KII LASER-PULSE/CRS Research Translation Strategy Lead Alexandra Towns alexandra.towns@ crs.org 13 Comparison Mechanisms USAID/ITR/R USAID AOR - PEER Aaron Burr Andrew Gerard aburr@usaid.gov agerad@usaid.gov 14 Comparison Mechanisms USAID/ITR/R USAID AOR - HESN Margaret Linak mlinak@usaid.gov 15 Comparison Mechanisms USAID/ITR/R USAID COR/AltCOR - RTAC Tara Hill thill@usaid.gov 16 Comparison Mechanisms USAID/ITR/R USAID COR/AltCOR - LER II Brandy Witthoft bwitthof@maxwell .syr.edu 17 RFA KII - Global LASER-PULSE RFA Activity Manager - Ethiopia TBD 18 RFA KII - Global USAID/Ethiopia RFA participants 19 RFA FGD - Global 23 RFA KII - Colombia LASER-PULSE RFA Activity Manager - Colombia TBD 24 RFA KII - Colombia USAID/Colombia RFA participants TBD 25 RFA FGD - Colombia New York University School of Medicine Associate Professor, Department of Psychology Brown brownad@newsch ool.edu 26 Universidad EAFIT, Medellín, Colombia Professor Hillon yhillon@eafit.edu.c o 27 Universidad del Norte Educator Vieira cvieira@uninorte.e du.co 28 Texas A&M AgriLife Research Professor Wingenbac h wingenbach@tamu .edu 29 Universidad ICESI Director of the School of Economics Zuluaga bzuluaga@icesi.ed u.co 100 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Data Sources Organization Position Name Contact information 30 R4D Case #1 Colombia LASER-PULSE LASER project manager TBD 31 R4D Case #1 Colombia USAID Mission attendees (Group KII) TBD 32 R4D Case #1 Colombia HEI Researcher participants (Group KII) TBD 33 R4D Case #1 Colombia HEI Researcher participants (Group KII) TBD 34 R4D Case #1 Colombia Boundary Partner Boundary partner participants (Group KII) TBD 35 R4D Case #1 Colombia Boundary Partner Boundary partner participants (Group KII) TBD 36 R4D Case #2 East Africa LASER-PULSE LASER project manager TBD 37 R4D Case #2 East Africa USAID Mission attendees (Group KII) TBD 38 R4D Case #2 East Africa HEI Researcher participants (Group KII) TBD 39 R4D Case #2 East Africa HEI Researcher participants (Group KII) TBD 40 R4D Case #2 East Africa Boundary Partner Boundary partner participants (Group KII) TBD 41 R4D Case #2 East Africa Boundary Partner Boundary partner participants (Group KII) TBD 42 Mission buy-in Case Study #1 - Deep ERT Support LASER-PULSE LASER Activity Manager TBD 101 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Data Sources Organization Position Name Contact information 43 Mission buy-in Case Study #1 - Deep ERT Support USAID Mission POC TBD 44 Mission buy-in Case Study #1 - Deep ERT Support HEI PI TBD 45 Mission buy-in Case Study #1 - Deep ERT Support HEI PI TBD 46 Mission buy-in Case Study #1 - Deep ERT Support Boundary Partner TBD Boundary Partner TBD 47 Mission buy-in Case Study #1 - Deep ERT Support Boundary Partner TBD Boundary Partner TBD 48 Mission buy-in Case Study #2 - Less ERT Support LASER-PULSE LASER Activity Manager TBD 49 Mission buy-in Case Study #2 - Less ERT Support USAID Mission POC TBD 50 Mission buy-in Case Study #2 - Less ERT Support HEI Research team TBD 51 Mission buy-in Case Study #2 - Less ERT Support HEI Research team TBD 52 Mission buy-in Case Study #2 - Less ERT Support Boundary Partner Boundary Partners TBD 53 Mission buy-in Case Study #2 - Less ERT Support Boundary Partner Boundary Partners TBD 102 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Data Sources Organization Position Name Contact information 54 Mission buy-in Case Study #3 - Women/LMIC researcher involvement LASER-PULSE LASER Activity Manager TBD 55 Mission buy-in Case Study #3 - Women/LMIC researcher involvement USAID Mission POC TBD 56 Mission buy-in Case Study #3 - Women/LMIC researcher involvement HEI Research team TBD 57 Mission buy-in Case Study #3 - Women/LMIC researcher involvement HEI Research team TBD 58 Mission buy-in Case Study #3 - Women/LMIC researcher involvement Boundary Partner Boundary Partners TBD 59 Mission buy-in Case Study #3 - Women/LMIC researcher involvement Boundary Partner Boundary Partners TBD 60 Mission buy-in Case Study #4 - Engagement with boundary partners LASER-PULSE LASER Activity Manager TBD 61 Mission buy-in Case Study #4 - USAID Mission POC TBD 103 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Data Sources Organization Position Name Contact information Engagement with boundary partners 62 Mission buy-in Case Study #4 - Engagement with boundary partners HEI Research team TBD 63 Mission buy-in Case Study #4 - Engagement with boundary partners HEI Research team TBD 64 Mission buy-in Case Study #4 - Engagement with boundary partners Boundary Partner Boundary Partners TBD 65 Mission buy-in Case Study #4 - Engagement with boundary partners Boundary Partner Boundary Partners TBD 104 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX 3. USAID M/B/IO QUANTITATIVE SURVEY Thank you for your participation in this brief survey about your experience with the DDI/ITR/R LASER mechanism. The survey is a component of a mid-term evaluation that aims to inform decisions about ongoing and future implementation of LASER. All data and identifying information will be anonymized; it will be impossible to reconstruct your answers. No one will be identified by name, and it will be impossible to attribute any survey responses or findings to you. This questionnaire will take about 20 minutes to complete. Module A. Demographics A1 Job Title Text A2. Hiring Mechanism 1= Foreign Service Officers (FSO) 2= Foreign Service National (FSN) 3=General Schedule (GS) 4=Civil Service (CS) 5=ISC 97=Other (specify) A3 Years with USAID Numerical A4 Gender 1=Female 2=Male 97=Other 999=Prefer not to answer A4b. Other, specify text A5 USAID Missions engage with LASER in a number of ways. Please select all LASER Activities you have participated in. If 1, ask Module B If 2, ask Module C 1=Request for Applications (RFA) 2=Research 4 Development (R4D) Convening 3=Buy-In 105 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV If 3, ask Module D 97=Other (specify) Module B. RFA Asked if A5=1 B1 Which Request for Applications (RFAs) did you participate in? 1=East Africa 2=Colombia 3=Vietnam 4=Ethiopia 5=Global 6=Minority Serving Institutions B2 Which of the following Request for Applications (RFA) activities were you involved in? 1=Co-created the Request for Applications (RFA) 2=Participated in a Comprehensive Success Factors Analysis (CSFA) process to identify research questions 3=Participated in Embedded Research Translation (ERT) 4=Reviewed RFA solicitation 5=Reviewed applicants 97=Other, specify B3 How useful did you find the Request for Applications (RFA) process for establishing locally relevant research priorities? 1=Very useful 2=Useful 3=Could be useful in other situations, but was not useful in this case 4=Was not useful 888=Don’t know 106 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 999=Prefer not to answer/ Not Applicable B4 How useful did you find the Request for Applications (RFA) process for making connections with researchers from low- and middle income countries? 1=Very useful 2=Useful 3=Could be useful in other situations, but was not useful in this case 4=Was not useful 888=Don’t know 999=Prefer not to answer/ Not Applicable B5 As a result of the Request for Applications (RFA) process, did you establish any new working relationships with researchers or policymakers? Select all that apply If 1, 2, or 3, ask B5b 1=Yes, with researchers from high-income countries (HIC) 2=Yes, with researchers from low or middle income (LMIC) countries 3=Yes, with host country government or policymakers 4=No 888=Don’t know 999=Prefer not to answer/ Not Applicable B5b Have any of these relationships led to new collaborations outside of the RFA? 0=No 1=Yes 888=Don’t know 999=Prefer not to answer/ Not Applicable B6 If B2=2 How useful did you find the Comprehensive Success Factors Analysis (CSFA) and/or Embedded Research Translation(ERT) in the RFA co-creation process for developing locally-led research priorities? 4=Very useful 3=Useful 4=Could be useful in other situations, but was not useful at the R4D convening 1=Was not useful 888=Don’t know 999=Prefer not to answer/ Not Applicable 107 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV B7 If B2=2 How useful did you find the Comprehensive Success Factors Analysis (CSFA) and/or Embedded Research Translation(ERT) in the RFA co-creation process for developing new research questions? 4=Very useful 3=Useful 2=Could be useful in other situations, but was not useful in the RFA co-creation process 1=Was not useful 888=Don’t know 999=Prefer not to answer/ Not Applicable B8 Overall, how satisfied are you with how responsive the Request for Applications (RFA) research has been to the needs in your particular context? 5=Very satisfied 4=Satisfied 3=Neither satisfied nor dissatisfied 2=Dissatisfied 1=Very Dissatisfied 888=Don’t know 999=Prefer not to answer/ Not Applicable B9 Overall, how satisfied are you with the quality of the research products produced through the Request for Applications (RFA) ? 5=Very satisfied 4=Satisfied 3=Neither satisfied nor dissatisfied 2=Dissatisfied 1=Very Dissatisfied 888=Don’t know 999=Prefer not to answer/ Not Applicable B10 Overall, how satisfied are you with the usefulness of the research findings and recommendations for your work? 5=Very satisfied 4=Satisfied 3=Neither satisfied nor dissatisfied 2=Dissatisfied 1=Very Dissatisfied 108 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 888=Don't Know 999=Prefer not to answer/ Not Applicable B11 How impactful has the research conducted under the Request for Applications (RFA) been for informing changes in USAID programming? 1=Large impact 2=Moderately impact 3=Slightly impact 4=No impact at all 888=Do not Know 999=Prefer not to answer/ Not Applicable B12 How impactful has the research conducted under the Request for Applications (RFA) been for informing changes in host government policies? 1=Large impact 2=Moderately impact 3=Slightly impact 4=No impact at all 888=Do not Know 999=Prefer not to answer/ Not Applicable Module C. R4D Convening Asked if A5=2 C1 Which Research 4 Development (R4D) convening did you attend? 1=East Africa 2=Colombia 3=Vietnam 4=Ethiopia 5=Thailand C2 What benefits did you experience from participating in the Research 4 Development (R4D)convening? Select all that apply 1=Developed a strong and active network of peers to collaborate with 2=Found appropriate research partners 3=Found appropriate policy audience 109 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 4=Learned new research skills 5=Learned how to better translate research to a development or policy audience 6=Learned new strategies for developing research based on local demands 7=Other (specify) 888=Don’t know 999=Prefer not to answer/ Not Applicable C3 How useful did you find the Research 4 Development (R4D)convening for establishing locally relevant research priorities? 1=Very useful 2=Useful 3=Could be useful in other situations, but was not useful in this case 4=Was not useful 888=Don’t know 999=Prefer not to answer/ Not Applicable C4 How useful did you find the Research 4 Development (R4D)convening for making connections with researchers from Low or Middle Income Countries (LMIC)? 1=Very useful 2=Useful 3=Could be useful in other situations, but was not useful in this case 4=Was not useful 888=Don’t know 999=Prefer not to answer/ Not Applicable C5 As a result of the Research 4 Development convening, did you establish any new working relationships with researchers or policymakers? Select all that apply If 1, 2, or 3, ask C5b 1=Yes, with researchers from high￾income countries 2=Yes, with researchers from low or middle income (LMIC) countries 3=Yes, with host country government or policymakers 110 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 4=No 888=Don’t know 999=Prefer not to answer/ Not Applicable C5b Have any of these relationships led to new collaborations outside of the R4D convening? 0=No 1=Yes 888=Don’t know 999=Prefer not to answer/ Not Applicable C6 How useful did you find the Comprehensive Success Factors Analysis (CSFA) and/or Embedded Research Translation(ERT) in the Request for Applications (RFA) for developing locally-led research priorities? 4=Very useful 3=Useful 4=Could be useful in other situations, but was not useful at the R4D convening 1=Was not useful 888=Don’t know 999=Prefer not to answer/ Not Applicable C7 How useful did you find the CSFA and/or ERT in the R4D Convening for developing new research questions? 4=Very useful 3=Useful 2=Could be useful in other situations, but was not useful at the R4D Convening 1=Was not useful 888=Don’t know 999=Prefer not to answer/ Not Applicable C8 How impactful was the R4D Convening for informing changes in USAID programming? 1=Large impact 2=Moderately impact 3=Slightly impact 111 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 4=No impact at all 888=Don’t know 999=Prefer not to answer/ Not Applicable C9 How impactful was the R4D Convening for informing changes in host government policies? 1=Large impact 2=Moderately impact 3=Slightly impact 4=No impact at all 888=Don’t know 999=Prefer not to answer/ Not Applicable D. Buy-In Module Asked if A5=3 D1 Please confirm you participated in [preload] buy￾in activity 0=No 1=Yes D1b. If no, which buy-in did you participate in? text D1 Why did you choose the LASER mechanism to conduct this research? Select all that apply 1=Access to high quality researchers 2=Access to a diverse cohort of researchers 3=Speed of research start-up 4=Familiar with mechanism 5=LASER’s emphasis on demand-driven research 97=Other, specify 888=Don’t know 999=Prefer not to answer/ Not Applicable 112 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV D2 As a result of the buy-in, did you establish any new working relationships with researchers or policymakers? select all that apply If not 4, ask D2b 1=Yes, with researchers from high￾income countries (HIC) 2=Yes, with researchers from low or middle income (LMIC) countries 3=Yes, with host country government or policymakers 4=No 888=Don’t know 999=Prefer not to answer/ Not Applicable D2b Have any of these relationships led to new collaborations outside of the buy-in? 0=No 1=Yes 888=Don’t know 999=Prefer not to answer/ Not Applicable D3 Overall, how satisfied are you with the timeliness of research produced through the buy-in? 1=Very satisfied 2=Satisfied 3=Neither satisfied nor dissatisfied 4=Dissatisfied 5=Very Dissatisfied 888=Don’t know 999=Prefer not to answer/ Not Applicable D4 Overall, how satisfied are you with how responsive the buy-in research has been to the needs in your particular context? 5=Very satisfied 4=Satisfied 3=Neither satisfied nor dissatisfied 2=Dissatisfied 1=Very Dissatisfied 888=Don’t know 999=Prefer not to answer/ Not Applicable 113 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV D5 Overall, how satisfied are you with the quality of the research products produced through the buy-in? 5=Very satisfied 4=Satisfied 3=Neither satisfied nor dissatisfied 2=Dissatisfied 1=Very Dissatisfied 888=Don’t know 999=Prefer not to answer/ Not Applicable D6 Overall, how satisfied are you with the usefulness of the research findings and recommendations for your work? 5=Very satisfied 4=Satisfied 3=Neither satisfied nor dissatisfied 2=Dissatisfied 1=Very Dissatisfied 888=Don’t know 999=Prefer not to answer/ Not Applicable D7 How impactful was the Research 4 Development (R4D)Convening for informing changes in USAID programming? 1=Large impact 2=Moderately impact 3=Slightly impact 4=No impact at all 888=Don’t know 999=Prefer not to answer/ Not Applicable D8 How impactful was the R4D Convening for informing changes in host government policies? 1=Large impact 2=Moderately impact 3=Slightly impact 4=No impact at all 888=Don’t know 999=Prefer not to answer/ Not Applicable 114 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Module E: Other Mechanisms Asked to all participants E1 Have you ever solicited research for USAID under a non-LASER funding mechanism? If not 1, skip to conclusion 0=No 1=Yes 888=Don’t know 999=Prefer not to answer/ Not Applicable E2 [If yes] Which mechanisms do you have experience with? Select all that apply 1=RTAC 2=HESN 3=PEER 4=DRG Learning, Evaluation, and Research (LER) 5=HARPnet 6=HELIX 7=LAC Education Support Contract 8=Global Health Center for Innovation and Impact (CII) 9=MEASURE evaluation (Data for Impact) 10=Data and Evidence for Education Programs (DEEP) 11=Education Performance Improvement, Communications, and Knowledge (EPIC) 12=Education Support Initiative 13=Global Health Program Cycle Improvement Project (GH Pro) 14=USAID Kenya and East Africa Evaluation, Assessment, and Analysis (EAA) 97=Other, specify E2b Other, specify text 115 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV E3 How does the ease of applying for LASER funding compare to applying for funding from other USAID research mechanisms with which you are familiar? 5=LASER is significantly better 4=LASER is somewhat better 3=The two are comparable 2=LASER is somewhat worse 1=LASER is significantly worse 888=Don’t know 999=Prefer not to answer/ Not Applicable E4 How does the quality of the research products produced by the LASER mechanism compare to the products from other USAID research mechanisms that you are familiar with? 5=LASER is significantly better 4=LASER is somewhat better 3=The two are comparable 2=LASER is somewhat worse 1=LASER is significantly worse 888=Don’t know 999=Prefer not to answer/ Not Applicable E5 How does the timeliness of the research products produced by the LASER mechanism compare to the products from other USAID research mechanisms that you are familiar with? 5=LASER is significantly better 4=LASER is somewhat better 3=The two are comparable 2=LASER is somewhat worse 1=LASER is significantly worse 888=Don’t know 999=Prefer not to answer/ Not Applicable E6 Compared to other USAID mechanisms you are familiar with, how well does LASER incorporate the research priorities of USAID Missions, policymakers, and other local partners? 5=LASER is significantly better 4=LASER is somewhat better 3=The two are comparable 2=LASER is somewhat worse 116 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 1=LASER is significantly worse 888=Don’t know 999=Prefer not to answer/ Not Applicable E7 Compared to other USAID mechanisms you are familiar with, how well does LASER increase connections between LMIC and HIC researchers and USAID? 5=LASER is significantly better 4=LASER is somewhat better 3=The two are comparable 2=LASER is somewhat worse 1=LASER is significantly worse 888=Don’t know 999=Prefer not to answer/ Not Applicable F. Conclusion Thank you for your participation in this survey. F1 Is there anything else you would like us to know about your experience with the LASER Activity? text 117 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX 4. LASER-PULSE NETWORK SURVEY Thank you for your participation in this brief survey about your experience with the USAID LASER mechanism. The survey is a component of a mid-term evaluation that aims to inform decisions about ongoing and future implementation of LASER. All data and identifying information will be anonymized; it will be impossible to reconstruct your answers. No one will be identified by name, and it will be impossible to attribute any survey responses or findings to you. This questionnaire will take about 20 minutes to complete. Module A. Demographics A1 What is your gender? 1=Male 2=Female 3=Other 999=Prefer not to respond A2 What is your ethnicity? select all that apply 1=Caucasian 2=Black 3=Asian/Pacific Islander 4=Latinx 97=Other, specify 999=Prefer not to Respond A2b Other, specify Text A3 What type of institution do you represent? 1=Higher Education Institution 2=Government 3=NGO 4=Donor Organization 5=Private company 97=Other, specify A3b Other, specify Text A4. How would you describe your primary role in the development space? 1=Researcher 2=Development Professional 118 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 3=Donor 4=Industry Collaborator 97=Other (specify) A4b Other, specify Text A5. Is your research institution located in a low or middle income country (LMIC) or a high income country (HIC)? 1=LMIC 2=HIC 888=Don’t know 999=Prefer not to answer/ Not Applicable/NA A6 How did you learn about the LASER-PULSE network? 1=From the LASER Activity 2=From USAID 3=From my Higher Education Institutions (HEI) 4=From another HEI 5=From a from friend or colleague 97=Other, specify A6b Other, specify Text A7. What is the primary reason you joined the LASER network? 1=Required to participate in a LASER activity 2=Recommended by a colleague/friend 3=Seeking opportunity for research funding 4=Opportunity for networking 5=Opportunities for skill-building 888=Don’t know 999=Prefer not to answer/ Not Applicable A8. How often do you engage with the LASER - PULSE website (such as searching for 1=Once a week 119 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV resources, applying for opportunities, contacting other resources, etc)? 2=Once a month 3=Once a quarter 4=1-2 times a year 5=Less than once a year 6=Signed up but have not engaged since 888=Don’t know 999=Prefer not to answer/ Not Applicable A9. What LASER activities have you participated in? Select all that apply If 1, ask Module B and Module E If 2, ask Module C and Module E If 3, ask Modules D Module E If 4, ask A10a and b If 5, ask A11 If 6= ask A12 1=Responded to a Request for Applications (RFA) 2=Research 4 Development (R4D) convening 3=Buy-in with Mission 4=Participated in a training 5=Used resources from the website 6=Used the LASER network to make new research connection (independent from R4D or RFA process) 97=Other (specify) 888=Don’t know 999=Prefer not to answer/ Not Applicable A10 A8a. Which training/s have you participated in? Select all that apply Will request a list of trainings from LASER team A10b A8b. How useful have trainings been for informing or improving your work? 1=Very useful 2=Somewhat useful 3=Not useful at all 888=Don’t know 999=Prefer not to answer/ Not Applicable A11 Which resources have you used? will request a list of resources from LASER team 120 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV A11b Other, specify text A12 Has this network connection resulted in a collaboration with other LASER researchers? 0=No 1=Yes 888=Don't Know 999=Prefer not to answer/ Not Applicable Module B. RFA Asked to all researchers who selected A9=1 B1 What is your engagement with the RFA? Select all that apply 1=I applied for RFA funding, but did not receive it 2=I applied for and received RFA funding 999=Prefer not to answer/ Not Applicable B2 How did you learn about the RFA? 1=From the LASER Activity 2=From USAID 3=From my Higher Education Institution (HEI) 4=From another HEI 5=From a from friend or colleague 97=Other, specify B2b Other, specify B3 Which RFA/s did you apply for? Select all that apply 1=East Africa 2=Colombia 3=Vietnam 4=Ethiopia 5=Global 6=Minority Serving Institutions 121 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV B4 In total, how many applications did you submit? B5 Compared to other funding opportunities (with USAID or with other donors), how much effort was required to complete the application? 1=Much more 2=More 3=About the same 4=Less 5=Much less 888=Don’t know 999=Prefer not to answer/ Not Applicable B6 As a result of the RFA, did you establish any new working relationships with researchers or policymakers? Select all that apply If 1, 2, or 3, ask B6b 1=Yes, with researchers from high￾income countries 2=Yes, with researchers from low or middle income (LMIC) countries 3=Yes, with host country government or policymakers 4=No 888=Don’t know 999=Prefer not to answer/ Not Applicable B6b Have any of these relationships led to new collaborations outside of the RFA? 0=No 1=Yes 888=Don’t know 999=Prefer not to answer/ Not Applicable B7 Were you awarded a subgrant? If 0, 888, or 999, end module. 1=Yes 0=No 888=Don’t know 999=Prefer not to answer/ Not Applicable 122 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV B8 Did you participate in a Comprehensive Success Factors Analysis (CSFA) process with the LASER team as part of this award? If 0, skip to B10 1=Yes 0=No 888=Don’t know 999=Prefer not to answer/ Not Applicable B9 How did the CSFA process impact your research outcomes? Generate a list of options after consultation with LASER activity manager B9b Other, specify text B10 Did you receive Embedded Research Translation (ERT) support from LASER as a part of the award? If 0, skip to B12 1=Yes 0=No 888=Don’t know 999=Prefer not to answer/ Not Applicable B11 How did ERT impact your research outcomes? Select all that apply generate a list of options after consultation with LASER activity manager B11b Other, specify text B12 Has your research under this Request for Applications (RFA) produced results at this time? If 0, end module 1=Yes 0=No 888=Don’t know 999=Prefer not to answer/ Not Applicable B13 Have the results of your research been translated to a policy or development focused audience? 1=Yes 0=No 888=Don’t know 999=Prefer not to answer/ Not Applicable 123 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV B14 Who were you research results shared with? 1=USAID Mission 2=USAID Washington 3=Higher Education Institutions (HEIs) in high income countries (HIC) 4=Higher Education Institutions in low or middle income (LMIC) 5=Host country government 6=CSOs 7=Other donors 97=Other, specify 888=Don’t know 999=Prefer not to answer/ Not Applicable B14b Other, specify Text B15 How were your research results shared? 1=Wrote policy briefing or other summary document 2=Presented research at an event for a policy or development-focused audience in the United States 3=Presented research at an event for a policy or development-focused audience outside the United States 4=Presented research at an academic conference 97=Other (specify) 888=Don’t know 999=Prefer not to answer/ Not Applicable B15b Other, specify Text B16 Did your findings inform USAID programming? 1=Yes 0=No 124 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 888=Don’t know 999=Prefer not to answer/ Not Applicable B16b How? Text B17 Did your findings inform host government policy? 1=Yes 0=No 888=Don’t know 999=Prefer not to answer/ Not Applicable B17b How? Text Module C. R4D Convening Asked to all researchers who selected A9=2 C1 Which Research 4 Development (R4D) convening did you attend? 1=East Africa 2=Colombia 3=Vietnam 4=Ethiopia 5=Thailand C2 What benefits did you experience from participating in the R4D convening? Select all that apply 1=Developed a strong and active network of peers to collaborate with 2=Found appropriate research partners 3=Found appropriate policy audience 4=Learned new research skills 5=Learned how to better translate research to a development or policy audience 6=Learned new strategies for developing research based on local demands 125 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 7=Other (specify) 888=Don't Know 999=Prefer not to answer/ Not Applicable C3 How useful did you find the R4D convening for establishing locally relevant research priorities? 1=Very useful 2=Useful 3=Could be useful in other situations, but was not useful in this case 4=Was not useful 888=Don't Know 999=Prefer not to answer/ Not Applicable C4 How useful did you find the R4D convening for making connections with researchers from Low or Middle Income Countries (LMIC)? 1=Very useful 2=Useful 3=Could be useful in other situations, but was not useful in this case 4=Was not useful 888=Don't Know 999=Prefer not to answer/ Not Applicable C5 As a result of the R4D convening, did you establish any new working relationships with researchers or policymakers? Select all that apply If 1, 2, or 3, ask C5b 1=Yes, with researchers from high￾income countries 2=Yes, with researchers from low or middle income (LMIC) countries 3=Yes, with host country government or policymakers 4=No 888=Don't Know 999=Prefer not to answer/ Not Applicable 126 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV C5b Have any of these relationships led to new collaborations outside of the R4D convening? 0=No 1=Yes 888=Don’t know 999=Prefer not to answer/ Not Applicable C6 How useful did you find the Comprehensive Success Factors Analysis (CSFA) and/or Embedded Research Translation(ERT) in the Request for Applications (RFA) for developing locally-led research priorities? 4=Very useful 3=Useful 4=Could be useful in other situations, but was not useful at the R4D convening 1=Was not useful 888=Don't Know 999=Prefer not to answer/ Not Applicable C7 How useful did you find the CSFA and/or ERT in the R4D Convening for developing new research questions? 4=Very useful 3=Useful 2=Could be useful in other situations, but was not useful at the R4D Convening 1=Was not useful 888=Don't Know 999=Prefer not to answer/ Not Applicable C8 How impactful was the R4D Convening for informing changes in USAID programming? 1=Large impact 2=Moderately impact 3=Slightly impact 4=No impact at all 888=Don’t know 999=Prefer not to answer/Not applicable 127 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV C9 How impactful was the R4D Convening for informing changes in host government policies? 1=Large impact 2=Moderately impact 3=Slightly impact 4=No impact at all 888=Don’t know 999=Prefer not to answer/Not applicable Module D. Buy-In Asked to all researchers who select A7=3, Buy-Ins D1 Please confirm you participated in [preload] buy￾in activity 0=No 1=Yes D1b. If no, which buy-in did you participate in? text D2 What benefits did you experience from participating in a buy-in? Select all that apply 1=Developed a strong and active network of peers to collaborate with 2=Found appropriate research partners 3=Found appropriate policy audience 4=Learned new research skills 5=Learned how to better translate research to a development or policy audience 6=Learned new strategies for developing research based on local demands 7=Other (specify) 888=Don't Know 999=Prefer not to answer/ Not Applicable D3 How useful did you find the buy-in research for establishing locally relevant research priorities? 1=Very useful 2=Useful 128 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 3=Could be useful in other situations, but was not useful in this case 4=Was not useful 888=Don't Know 999=Prefer not to answer/ Not Applicable D4 How useful did you find the buy-in research for making connections with researchers from low￾and middle-income countries (LMICs)? 1=Very useful 2=Useful 3=Could be useful in other situations, but was not useful in this case 4=Was not useful 888=Don't Know 999=Prefer not to answer/ Not Applicable D5 As a result of the buy-in research, did you establish any new working relationships with researchers or policymakers? Select all that apply If 1, 2, or 3, ask D5b 1=Yes, with researchers from high￾income countries 2=Yes, with researchers from low or middle income (LMIC) countries 3=Yes, with host country government or policymakers 4=No 888=Don't Know 999=Prefer not to answer/ Not Applicable D5b Have any of these relationships led to new collaborations outside of the buy-in? 0=No 1=Yes 888=Don’t know 999=Prefer not to answer/ Not Applicable D6 Did you participate in a Comprehensive Success Factors Analysis (CSFA) process with the LASER team as part of the buy-in? If 0, skip to D8 1=Yes 0=No 888=Don’t know 129 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 999=Prefer not to answer/ Not Applicable D7 How did the Comprehensive Success Factors Analysis (CSFA) process impact your research outcomes? generate a list of options after consultation with LASER activity manager D7b Other, specify text D8 Did you receive Embedded Research Translation (ERT) support from LASER as a part of the buy￾in? If 0, skip to D10 1=Yes 0=No 888=Don’t know 999=Prefer not to answer/ Not Applicable D9 How did ERT impact your research outcomes? Select all that apply generate a list of options after consultation with LASER activity manager D9b Other, specify text D10 Has your research under this buy-in produced results at this time? If 0, end module 1=Yes 0=No 888=Don’t know 999=Prefer not to answer/ Not Applicable D11 Have the results of your research been translated to a policy or development focused audience? 1=Yes 0=No 888=Don’t know 999=Prefer not to answer/ Not Applicable D12 Who were you research results shared with? 1=USAID Mission 2=USAID Washington 3=Higher Education Institutions (HEIs) in High Income Countries (HIC) 4=Higher Education Institutions (HEIs) in Low or Middle Income Countries (LMIC) 130 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 5=Host country government 6=CSOs 7=Other donors 97=Other, specify 888=Don’t know 999=Prefer not to answer/ Not Applicable D12b Other, specify Text D13 How were your research results shared? 1=Wrote policy briefing or other summary document 2=Presented research at an event for a policy or development-focused audience in the United States 3=Presented research at an event for a policy or development-focused audience outside the United States 4=Presented research at an academic conference 97=Other (specify) 888=Don’t know 999=Prefer not to answer/ Not Applicable D13b Other, specify Text D14 Did your findings inform USAID programming? 1=Yes 0=No 888=Don’t know 999=Prefer not to answer/ Not Applicable D14b How? Text D15 Did your findings inform host government policy? 1=Yes 0=No 131 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 888=Don’t know 999=Prefer not to answer/ Not Applicable D15b How? Text Module E. Comparison Mechanisms This module is asked to all respondents E1 Have you ever conducted research for USAID under a non-LASER funding mechanism? If 0, 888, or 999, skip to Module F, conclusions 0=No 1=Yes 888=Don't Know 999=Prefer not to answer/ Not Applicable E2 Which mechanisms do you have experience with? 1=Research Technical Assistance Center (RTAC) 2=Higher Education Solutions Network (HESN) 3=Partnerships for Enhanced Engagement in Research (PEER) 4=DRG Learning, Evaluation, and Research (LER) 5=Health Research Program (HARPnet) 6=Higher Education for Leadership, Innovation, and Exchange (HELIX) 7=LAC Education Support Contract 8=Global Health Center for Innovation and Impact (CII) 9=MEASURE evaluation (Data for Impact) 10=Data and Evidence for Education Programs (DEEP) 11=Education Performance Improvement, Communications, and Knowledge (EPIC) 12=Education Support Initiative 132 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 13=Global Health Program Cycle Improvement Project (GH Pro) 14=USAID Kenya and East Africa Evaluation, Assessment, and Analysis (EAA) 97=Other, specify E2b Other, specify text E3 Compared to other USAID mechanisms you are familiar with, how useful are the training and resources available to researchers? ? 5=LASER is significantly better 4=LASER is somewhat better 3=The two are comparable 2=LASER is somewhat worse 1=LASER is significantly worse 997= NA 888=Don't Know 999=Prefer not to answer/ Not Applicable E4 Compared to other USAID mechanisms you are familiar with, how well does LASER incorporate the research priorities of USAID Missions, policymakers, and other local partners? 5=LASER is significantly better 4=LASER is somewhat better 3=The two are comparable 2=LASER is somewhat worse 1=LASER is significantly worse 997= NA 888=Don't Know 999=Prefer not to answer/ Not Applicable E5 Compared to other USAID mechanisms you are familiar with, how well does LASER increase connections between LMIC and HIC researchers? 5=LASER is significantly better 4=LASER is somewhat better 3=The two are comparable 2=LASER is somewhat worse 1=LASER is significantly worse 133 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 888=Don't Know 999=Prefer not to answer/ Not Applicable F. Conclusion Thank you for your participation in this survey. F1 Is there anything else you would like us to know about your experience with the LASER-PULSE network or the LASER Activity? text 134 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX 5. LASER CORE TEAM KII GUIDE LINC, the Cloudburst Group, and the DevLab@Duke have been commissioned by USAID/ITR/R to conduct a midterm performance evaluation of the Long-Term Assistance and Services for Research (LASER) Activity, implemented by Purdue University and its partners. The goal of the evaluation is to improve USAID/ITR/R’s understanding of the strengths and weaknesses of the LASER design to improve current and future ITR/R programming implemented by higher education institutions. Today, we’d like to speak with you about your experience managing the LASER Activity. Our conversation will last approximately one hour. I will take a transcript of our conversation, but please be assured that we will carefully safeguard any information you provide and any information you provide will be used exclusively for the purpose of the assessment. No staff will be identified by name or by position. Do you have any questions before we begin? 1. Please tell me a little about your role in the LASER activity. 2. What do you see as the greatest strengths of the LASER design? Listen for: research development, capacity development, demand-driven research, diversity of researchers, utilization/dissemination, stakeholder engagement 3. In your opinion, what are the greatest weaknesses to the LASER design? What would you change in a future design, if you could? Listen for: research development, capacity development, demand-driven research, diversity of researchers, utilization/dissemination, stakeholder engagement 4. Do you feel like the TOC has been proven? ? Why or why not? 5. In your opinion, what are some of the most significant early results of the LASER Activity? Listen for: research development, capacity development, demand-driven research, diversity of researchers, utilization/dissemination, stakeholder engagement 6. What adaptations has the LASER activity made to adapt to COVID-19? How has COVID-19 impacted early results? 7. What is the likelihood of LASER meeting its final program goals? Why? 8. What changes could be made in the remaining time to help the Activity achieve its goals and objectives? 135 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX 6. RFA KII GUIDE LASER-PULSE STAFF (GLOBAL/COLOMBIA) LINC, the Cloudburst Group, and the DevLab@Duke have been commissioned by USAID/ITR/R to conduct a midterm performance evaluation of the Long-Term Assistance and Services for Research (LASER) Activity, implemented by Purdue University and its partners. The goal of the evaluation is to improve USAID/ITR/R’s understanding of the strengths and weaknesses of the LASER design to improve current and future ITR/R programming implemented by higher education institutions. Today, we’d like to speak with you about your experience leading LASER’s RFA in [country], funded in [year}. Our conversation will last approximately one hour. I will take a transcript of our conversation, but please be assured that we will carefully safeguard any information you provide and any information you provide will be used exclusively for the purpose of the assessment. No staff will be identified by name or by position. Do you have any questions before we begin? 1. I’d like to learn more about your experience during the RFA process. What role did you play? 2. Can you tell me about the solicitation development process for the [country] RFA? a. Who participates, how and when? b. How did this differ from previous RFAs within LASER? i. What led to this change? ii. How do you think this change impacted the outcomes of the RFA process? 3. Can you tell me about the RFA promotion/application process for the [country] RFA? a. How is the RFA shared? What are the requirements for applying? How many people typically apply? How many women/LMIC? b. How did this differ from previous RFAs? i. What led to this change? ii. How do you think this change impacted the outcomes of the RFA process? 4. Can you tell me about the RFA award process for the [country] RFA? a. How were winners chosen? Who decides? b. What type of capacity building or trainings are offered to grant winners? What impacts have you seen as a result of these trainings? c. How did this differ from previous RFAs? i. What led to this change? ii. How do you think this change impacted the outcomes of the RFA process? 5. Can you tell me about the RFA research implementation process for the [country] RFA? 136 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV a. How has COVID-19 impacted implementation? b. How did this differ from previous RFAs? i. What led to this change? ii. How do you think this change impacted the outcomes of the RFA process? 6. Can you tell me about the RFA research dissemination/utilization process for the [country] RFA? a. How and to whom were results shared? What was most effective? b. Did findings lead to any program or policy changes? What? c. How did this differ from previous RFAs? i. What led to this change? ii. How do you think this change impacted the outcomes of the RFA process? 7. Overall, what elements in the RFA process do you think worked well? What elements would you change? 8. Do you have any ideas for the LASER team about how they could make the RFA process more inclusive, particularly for women and LMIC researchers? USAID/MISSION STAFF (GLOBAL/COLOMBIA) LINC, the Cloudburst Group, and the DevLab@Duke have been commissioned by USAID/ITR/R to conduct a midterm performance evaluation of the Long-Term Assistance and Services for Research (LASER) Activity, implemented by Purdue University and its partners. The goal of the evaluation is to improve USAID/ITR/R’s understanding of the strengths and weaknesses of the LASER design to improve current and future ITR/R programming implemented by higher education institutions. Today, we’d like to speak with you about your experience contributing to LASER’s RFA in [country], funded in [year}. Our conversation will last approximately one hour. I will take a transcript of our conversation, but please be assured that we will carefully safeguard any information you provide and any information you provide will be used exclusively for the purpose of the assessment. No staff will be identified by name or by position. Do you have any questions before we begin? 1. I’d like to learn more about your experience during the RFA process. What role did you play? 2. One of the goals of LASER is to develop locally-relevant research, and it employs several tools to do so. This includes co-creation, embedded research translation, and Comprehensive Success Factor Analysis (CSFA). Were any of these tools a part of the RFA process? Which ones? a. If yes, how did these tools impact the RFA design? i. Probe: Strengthen relevance, engage more stakeholders, b. If yes, how did these tools impact RFA research implementation? 137 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV c. If yes, how did these tools impact RFA research dissemination/utilization? 3. Were any government stakeholders, CSOs, or other external stakeholders engaged in the RFA process? a. Which ones? b. How did the involvement of these partners impact the final solicitation? 4. Are any of the research findings from the awarded subgrants complete? a. If yes, were the findings shared with you? b. If yes, have the findings impacted your work? How? probe for specific program or policy changes 5. What elements in the RFA process do you think worked well? What elements would you change? 6. Do you have any ideas for the LASER team about how they could make the RFA process more inclusive, particularly for women and LMIC researchers? RFA SUBGRANT AWARDEES (GLOBAL/COLOMBIA) LINC, the Cloudburst Group, and the DevLab@Duke have been commissioned by USAID/ITR/R to conduct a midterm performance evaluation of the Long-Term Assistance and Services for Research (LASER) Activity, implemented by Purdue University and its partners. The goal of the evaluation is to improve USAID/ITR/R’s understanding of the strengths and weaknesses of the LASER design to improve current and future ITR/R programming implemented by higher education institutions. Today, we’d like to speak with you about your experience conducting research funded through LASER’s RFA in [country], funded in [year}. Our conversation will last approximately one hour. I will take a transcript of our conversation, but please be assured that we will carefully safeguard any information you provide and any information you provide will be used exclusively for the purpose of the assessment. No staff will be identified by name or by position. Do you have any questions before we begin? 1. How did you first learn about the (Colombia/Global) RFA? What made you decide to apply? Pay particular attention to responses from women/LMIC researchers 2. One of the goals of LASER is to develop locally-relevant research, and it employs several tools to do so. This includes co-creation, embedded research translation, and CSFA. a. How did the co-creation process change your initial research design? i. Probe: Strengthen relevance, b. Did your research include an ERT or CSFA component? i. How did your research incorporate ERT or CSFA? ii. What impact did the ERT/CSFA process have on your research design? What about research implementation? Dissemination? Utilization? iii. Would you recommend using an ERT and CSFA component in future research? 138 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV c. If your research is complete, how have you shared the results? Are you aware of any utilization of the findings? If so, how was it used and by whom? 3. What are some of the initial results of your research? a. Probe: New relationships (researchers, policymakers, LMIC), policy recommendations, new research questions 4. How has COVID-19 impacted your research? What challenges have you faced, and how have you adapted? a. How has the LASER team supported adaptation? b. How do you think these adaptations will impact the final result of your research? 5. What elements in the RFA process do you think worked well? What elements would you change? 6. Do you have any ideas for the LASER team about how they could make the RFA process more inclusive, particularly for women and LMIC researchers? 7. How do you engage with the LASER network? a. Did you take any of the trainings LASER developed? Which ones? How did they impact your research? b. Have you found any new research partners through the network? c. What would make you more likely to engage with the network? 8. Do you work on any other USAID-funded research activities? Under what mechanisms? How does working with LASER compare? Probe: ease/speed of application process, solicitation of applications from LMIC researchers, minority researchers, women researchers, co-creation process, emphasis on locally-relevant research 139 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX 7. KII GUIDE- R4D CONVENINGS KIIS LASER ACTIVITY MANAGER LINC, the Cloudburst Group, and the DevLab@Duke have been commissioned by USAID/ITR/R to conduct a midterm performance evaluation of the Long-Term Assistance and Services for Research (LASER) Activity, implemented by Purdue University and its partners. The goal of the evaluation is to improve USAID/ITR/R’s understanding of the strengths and weaknesses of the LASER design to improve current and future ITR/R programming implemented by higher education institutions. Today, we’d like to speak with you about your experience leading LASER’s R4D Convening in [country] in [year}. Our conversation will last approximately one hour. I will take a transcript of our conversation, but please be assured that we will carefully safeguard any information you provide and any information you provide will be used exclusively for the purpose of the assessment. No staff will be identified by name or by position. Do you have any questions before we begin? 1. Tell me about your role with the [LASER convening]. 2. To your mind, what was the greatest success of the R4D Convening in [country]? 3. What was the greatest challenge? How did you meet it? 4. How did the [county] R4D Convening differ from previous convenings? a. What impacts do you think those changes had on the outcome of the RFA? b. Were those changes continued in future R4Ds? c. What would you change about this convening for future convenings, if you could? 5. What benefits do you believe participants received by attending the R4D convening? What, if anything, did you gain from attending? Probes: How did the convening support___? i. finding new research partners and peers for collaboration ii. finding appropriate policy audiences iii. learning how to better translate research to a development/policy audience iv. learning new strategies for developing research based on local demands v. learning new research methods/skills 6. To your knowledge, did attending the R4D convening lead to any research influencing a USAID program or government policy? a. If so, which research? How did that come about? b. If not, why not? 7. What steps did you take to ensure participation from a diverse audience? 140 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV a. LMIC researchers b. women c. Boundary partners/policy makers 8. Do you have any ideas about how future R4D Convenings could be more inclusive, particularly for women and LMIC researchers? USAID MISSION PARTICIPANTS, RESEARCH PARTICIPANTS, AND BOUNDARY PARTNER PARTICIPANTS LINC, the Cloudburst Group, and the DevLab@Duke have been commissioned by USAID/ITR/R to conduct a midterm performance evaluation of the Long-Term Assistance and Services for Research (LASER) Activity, implemented by Purdue University and its partners. The goal of the evaluation is to improve USAID/ITR/R’s understanding of the strengths and weaknesses of the LASER design to improve current and future ITR/R programming implemented by higher education institutions. Today, we’d like to speak with you about your experience with the LASER R4D convening in [country] in [year}. Our conversation will last approximately one hour. I will take a transcript of our conversation, but please be assured that we will carefully safeguard any information you provide and any information you provide will be used exclusively for the purpose of the assessment. No staff will be identified by name or by position. Do you have any questions before we begin? R4D Convening Process/Experience 1. Tell me about your role with the [LASER convening]. 2. How did you first learn about the LASER R4D convening? a. Why did you decide to attend? 3. What benefits, if any, were there to attending the R4D convening? What, if anything, did you gain from attending? Probes: How did the convening support___? ■ finding new research partners and peers for collaboration ■ finding appropriate policy audiences ■ learning how to better translate research to a development/policy audience ■ learning new strategies for developing research based on local demands) ■ learning new research methods/skills 4. Did you establish any new relationships as a result of attending the convening? a. If so, with whom? (e.g., other researchers, USAID, host country) How did those relationships come about? b. If not, why not? 141 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 5. Did you develop any new research ideas as a result of attending the convening? a. If so, what ideas? How did those come about? How, if at all, were these ideas responsive to local needs? How do you know? b. If not, why not? 6. To your knowledge, did attending the R4D convening lead to any research becoming policy? a. If so, which research? How did that come about? b. If not, why not? 7. What were the challenges of attending the R4D convening, if any? a. How could the convening be improved? (probe for: relationship development, research idea development, CSFA process) 8. How did the COVID-19 pandemic affect the R4D convening, if at all? Researchers only: Now I’d like to talk about the process of conducting your research/translation work. One of the goals of LASER is to develop locally-relevant research, and it employs several tools to do so. This includes co-creation, embedded research translation… 9. How did the co-creation process at the R4D convening change your initial research design? a. What role, if any, did the co-creation process play in making the research locally￾relevant? Please provide examples. ■ If not, why do you think that was? b. How could the co-creation process better support locally-relevant research? 10. Did the research for this project include a CSFA component? a. If so, how did the research incorporate CSFA? b. What impact did the CSFA process have on your research design? What about research implementation? Dissemination? Utilization? c. From your perspective, was the CSFA process beneficial for this project? If so, how? If not, why not? How could it be improved? ■ Would you recommend using the CSFA process on other research going forward? If so, why? If not, why not? 11. Did the research for this project include an ERT component? a. If so, how did the research incorporate ERT? b. What impact did the ERT process have on your research design? What about research implementation? Dissemination? Utilization? 142 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV c. From your perspective, was the ERT process beneficial for this project? If so, how? If not, why not? How could it be improved? ■ Would you recommend using the ERT process on other research going forward? If so, why? If not, why not? 12. If your research is complete, how have you shared the results? Are you aware of any utilization of the findings? If so, how was it used and by whom? 13. What are some of the initial impacts of your research, if anything? a. Probe: New relationships (researchers, policymakers, LMIC), policy recommendations, new research questions b. How, if at all, did the CSFA process impact these results? c. How, if at all, did the ERT process impact these results? Boundary partners 14. To what extent did the convening lead to increased linkages between researchers and policymakers? Please provide examples. 15. How did attending the convening support your work as a boundary partner, if at all? which aspects of the convening supported your work? a. If not, why not? 16. In what ways was the convening relevant to your work as a boundary partner? a. If not, why not? b. How could the convening be improved? ANNEX 8. KII GUIDE- BUY-INS KIIS LINC, the Cloudburst Group, and the DevLab@Duke have been commissioned by USAID/ITR/R to conduct a midterm performance evaluation of the Long-Term Assistance and Services for Research (LASER) Activity, implemented by Purdue University and its partners. The goal of the evaluation is to improve USAID/ITR/R’s understanding of the strengths and weaknesses of the LASER design to improve current and future ITR/R programming implemented by higher education institutions. Today, we would like to speak with you about your experience conducting research funded through LASER’s buy-in mechanism in [country], funded in [year}. Our conversation will last approximately one hour. I will take a transcript of our conversation, but please be assured that we will carefully safeguard any information you provide and any information you provide will be used exclusively for the purpose of the assessment. No staff will be identified by name or by position. Do you have any questions before we begin? BUY-IN LASER ACTIVITY MANAGER 1. Tell me about your work and your role with the [LASER buy-in project]. 143 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 2. How was the SOW developed? a. Were local partners involved in the development of the research questions? Who? How? 3. How was the research team recruited? a. Were any efforts taken to involve women researchers? b. Were any efforts taken to involve LMIC researchers? 4. Did the research for this project include a CSFA component? a. If so, how did the research incorporate CSFA? b. What impact did the CSFA process have on the research design? What about research implementation? Dissemination? Utilization? c. From your perspective, was the CSFA process beneficial for this project? If so, how? If not, why not? How could it be improved? i. Would you recommend using the CSFA process on other research going forward? If so, why? If not, why not? 5. Did the research for this project include an ERT component? a. If so, how did the research incorporate ERT? b. What impact did the ERT process have on your research design? What about research implementation? Dissemination? Utilization? c. From your perspective, was the ERT process beneficial for this project? If so, how? If not, why not? How could it be improved? i. Would you recommend using the ERT process on other research going forward? If so, why? If not, why not? 6. What are some of the initial impacts of the research conducted through this buy-in, if anything? a. From your perspective, to what extent has this project produced high quality research - in terms of academic rigor/improving the body of knowledge on a topic? b. Has the research from this project been translated into any external products, such as academic papers or conference findings or other presentations (beyond internal contract deliverables for USAID)? If so, which products? How have they been used? c. To what extent has the research conducted through this buy-in led to new collaboration on other projects, research or policies among researchers, policy makers, USAID staff, etc.? i. Can you provide some examples of these new working relationships? How strong and sustainable do you think these relationships are? d. To what extent has the research conducted through this buy-in led to the development of new research questions or research priorities for USAID? i. Can you provide us with examples of these questions or examples of how the priorities have changed? 144 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV e. To what extent has the research conducted through this buy-in motivated changes in USAID program design? i. Has this research informed/influenced any policy discussions in the Agency? 7. Now I’d like you to reflect on the process as a whole. What about the LASER buy-in mechanism has worked well? Why? a. Probe for scope development, mission engagement, management, and collaboration with academic partners 8. Beyond what we’ve already discussed, what are some challenges you’ve experienced with the LASER buy-in mechanism? a. Probe for scope development, mission engagement, management, and collaboration with academic partners b. What could be done to address these challenges? c. How could the buy-in process be improved? 9. How do you think the buy-in component of the LASER design helps or hinders the achievement of LASER’s objectives? Would you recommend including buy-ins in future mechanisms? BUY-IN USAID MISSION POCS 1. Tell me about your work and your role with the [LASER buy-in project]. a. How did you learn about the Buy-In Mechanism? b. Why did you choose this mechanism? 2. How was the SOW developed? a. Were local partners involved in the development of the research questions? Who? How? 3. Did the research for this project include a CSFA component? a. If so, how did the research incorporate CSFA? b. What impact did the CSFA process have on your research design? What about research implementation? Dissemination? Utilization? c. From your perspective, was the CSFA process beneficial for this project? If so, how? If not, why not? How could it be improved? i. Would you recommend using the CSFA process on other research going forward? If so, why? If not, why not? 4. Did the research for this project include an ERT component? a. If so, how did the research incorporate ERT? b. What impact did the ERT process have on your research design? What about research implementation? Dissemination? Utilization? 145 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV c. From your perspective, was the ERT process beneficial for this project? If so, how? If not, why not? How could it be improved? i. Would you recommend using the ERT process on other research going forward? If so, why? If not, why not? 5. What are some of the initial impacts of the research conducted through this buy-in, if anything? a. From your perspective, to what extent has this project produced high quality research - in terms of academic rigor/improving the body of knowledge on a topic? b. Has the research from this project been translated into any external products, such as academic papers or conference findings or other presentations (beyond internal contract deliverables for USAID)? If so, which products? How have they been used? c. To what extent has the research conducted through this buy-in led to new collaboration on other projects, research or policies among researchers, policy makers, USAID staff, etc.? i. Can you provide some examples of these new working relationships? How strong and sustainable do you think these relationships are? d. To what extent has the research conducted through this buy-in led to the development of new research questions or research priorities for USAID? i. Can you provide us with examples of these questions or examples of how the priorities have changed? e. To what extent has the research conducted through this buy-in motivated changes in USAID program design? i. Has this research informed/influenced any policy discussions in the Agency? 6. Now I’d like you to reflect on the process as a whole. What about the LASER buy-in mechanism has worked well? Why? a. Probe for scope development, mission engagement, management, and collaboration with academic partners 7. Beyond what we’ve already discussed, what are some challenges you’ve experienced with the LASER buy-in mechanism? a. Probe for scope development, mission engagement, management, and collaboration with academic partners b. What could be done to address these challenges? c. How could the buy-in process be improved? 8. Overall, are you satisfied with the work produced in this buy-in? probe: Cost, timeliness, quality of research, diversity of researchers, usefulness of findings 9. What benefits has the Mission received from participating in this buy-in? 146 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 10. Do you work on any other USAID-funded research activities? Under what mechanisms? How does working with LASER compare? a. Ease/speed of the procurement process b. Inclusivity for women/minority/LMIC researchers c. Efficiency and effectiveness of the co-creation process d. Co-creation process e. Emphasis Locally-relevant research BUY-IN PRINCIPAL INVESTIGATORS 1. How did you become involved in this buy-in? probe: applied, recruited by LASER staff, referred by Mission, referred by colleague 2. Were you involved in the development of the research scope? In what ways? probe: developing research methodology, research questions, research design 3. Did the research for this project include a CSFA component? a. If so, how did the research incorporate CSFA? b. What impact did the CSFA process have on your research design? What about research implementation? Dissemination? Utilization? c. From your perspective, was the CSFA process beneficial for this project? If so, how? If not, why not? How could it be improved? i. Would you recommend using the CSFA process on other research going forward? If so, why? If not, why not? 4. Did the research for this project include an ERT component? a. If so, how did the research incorporate ERT? b. What impact did the ERT process have on your research design? What about research implementation? Dissemination? Utilization? c. From your perspective, was the ERT process beneficial for this project? If so, how? If not, why not? How could it be improved? i. Would you recommend using the ERT process on other research going forward? If so, why? If not, why not? 5. How has COVID-19 impacted your research? What challenges have you faced, and how have you adapted? a. How has the LASER team supported adaptation? b. How do you think these adaptations will impact the final result of your research? 6. What are some of the initial results of your research? 147 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV a. Probe: New relationships (researchers, policymakers, LMIC), policy recommendations, new research questions 7. What are some of the initial impacts of the research conducted through this buy-in, if anything? a. From your perspective, to what extent has this project produced high quality research - in terms of academic rigor/improving the body of knowledge on a topic? b. Has the research from this project been translated into any external products, such as academic papers or conference findings or other presentations (beyond internal contract deliverables for USAID)? If so, which products? How have they been used? c. To what extent has the research conducted through this buy-in led to new collaboration on other projects, research or policies among researchers, policy makers, USAID staff, etc.? i. Can you provide some examples of these new working relationships? How strong and sustainable do you think these relationships are? d. To what extent has the research conducted through this buy-in led to the development of new research questions or research priorities for USAID? i. Can you provide us with examples of these questions or examples of how the priorities have changed? e. To what extent has the research conducted through this buy-in motivated changes in USAID program design? i. Has this research informed/influenced any policy discussions in the Agency? 8. Now I’d like you to reflect on the process as a whole. What about the LASER buy-in mechanism has worked well? Why? a. Probe for scope development, mission engagement, management, and collaboration with academic partners 9. Beyond what we’ve already discussed, what are some challenges you’ve experienced with the LASER buy-in mechanism? a. Probe for scope development, mission engagement, management, and collaboration with academic partners b. What could be done to address these challenges? c. How could the buy-in process be improved? 10. How do you engage with the LASER network? a. Did you take any of the training LASER developed? Which ones? How did they impact your research? b. Have you found any new research partners through the network? c. What would make you more likely to engage with the network? BUY-IN BOUNDARY PARTNERS 148 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 1. Please tell me about your role with the buy-in research. a. Involvement in scope/design b. Involvement in conducting research c. Involvement in findings dissemination/utilization 2. What are some of the initial impacts of the research conducted through this buy-in, if anything? a. From your perspective, to what extent has this project produced high quality research - in terms of academic rigor/improving the body of knowledge on a topic? b. Has the research from this project been translated into any external products, such as academic papers or conference findings or other presentations (beyond internal contract deliverables for USAID)? If so, which products? How have they been used? c. To what extent has the research conducted through this buy-in led to new collaboration on other projects, research or policies among researchers, policy makers, USAID staff, etc.? ■ Can you provide some examples of these new working relationships? How strong and sustainable do you think these relationships are? d. To what extent has the research conducted through this buy-in led to the development of new research questions or research priorities for USAID? ■ Can you provide us with examples of these questions or examples of how the priorities have changed? e. To what extent has the research conducted through this buy-in motivated changes in USAID program design? ■ Has this research informed/influenced any policy discussions in the Agency? 3. In what ways is the buy-in research relevant to your work? How do you anticipate using the findings? 4. Now I’d like you to reflect on the process as a whole. What about the LASER buy-in mechanism has worked well? Why? a. Probe for scope development, mission engagement, management, and collaboration with academic partners 5. Beyond what we’ve already discussed, what are some challenges you’ve experienced with the LASER buy-in mechanism? a. Probe for scope development, mission engagement, management, and collaboration with academic partners b. What could be done to address these challenges? c. How could the buy-in process be improved? 149 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 6. Do you have any ideas for how the LASER team could involve policymakers, CSOs, and other local partners in the development of research ideas? 7. Do you have any ideas for how the LASER team could involve policymakers, CSOs, and other local partners in the sharing of research ideas? 150 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX 9. KII GUIDE - COMPARISON MECHANISM GUIDE LINC, the Cloudburst Group, and the DevLab@Duke have been commissioned by USAID/ITR/R to conduct a midterm performance evaluation of the Long-Term Assistance and Services for Research (LASER) Activity, implemented by Purdue University and its partners. The goal of the evaluation is to improve USAID/ITR/R’s understanding of the strengths and weaknesses of the LASER design to improve current and future ITR/R programming implemented by higher education institutions. To better understand the strengths and weaknesses of the LASER mechanism, the evaluation team has selected four other USAID research mechanisms. Today, we would like to speak with you about the successes and challenges of the [mechanism], with an emphasis on how research is designed and shared, as well as how the mechanism addresses capacity building and relationship building. Our conversation will last approximately one hour. I will take a transcript of our conversation, but please be assured that we will carefully safeguard any information you provide and any information you provide will be used exclusively for the purpose of the assessment. You will not be identified by name or by position. Do you have any questions before we begin? 1. Can you tell me a bit about the [mechanism]’s biggest successes to date? a. What do you think has contributed to the success? 2. How is research solicited through this mechanism? Buy-in, grants, other? 3. EPIC and LER II: What are some of the advantages of a sector-specific research mechanism? 4. LER II: What are some of the advantages and disadvantages of a stand-alone mechanism for buy￾in research from USAID M/B/IOs? a. Advantage/disadvantages to clients (speed, ease, access to researchers) b. Advantages/disadvantages to researchers 5. How are research topics or scopes of work developed? a. Who do you consult? listen for local actors b. What is the process like? c. How long does it take? 6. PEER and HESN: How do researchers apply for funding under this mechanism? a. Are any groups of researchers prioritized? Which groups? How do you ensure these groups apply for/receive funding? 7. EPIC and LER II: How are researchers chosen to carry out the research? a. Are any groups of researchers prioritized? Which groups? How do you ensure these groups are represented? 8. HESN, PEER, and EPIC: How does the network component of the activity aid in collaboration and utilization of research results? 9. What types of capacity building or training are offered under this mechanism? To whom are they offered? a. Which do you feel are most effective? b. Do you prioritize any groups to receive training or capacity building? Which groups? How do you ensure these groups are included? 10. What steps are taken under this mechanism to promote dissemination and utilization of research findings? 151 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV a. What products? To what audiences? b. What has been most effective? Least effective? 11. What are some of the lessons learned from conducting research under this mechanism? 12. Is there anything else you would like the ET to know about the mechanism? ANNEX 10. EVALUATION REPORT OUTLINE 1. Abstract (250 words) 2. Executive Summary (2-5 pages) 3. Evaluation Purpose 4. Background on the Context and the Strategies/Projects/Activities being Evaluated 5. Evaluation Questions 6. Methodology 6.1. Qualitative 6.2. Quantitative 7. Limitations to the Evaluation 8. Findings, Conclusions, and Recommendations 9. Annexes 9.1. KII list 9.2. Survey Results 9.3. Data collection tools (if requested) 9.4. Table of evaluation findings, conclusions, and recommendations 152 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX IX. EVALUATION BRIEF 153 | LASER MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV