EX-POST IMPACT EVALUATION PEER PROGRAM January 2021 DISCLAIMER This publication was produced for review by the United States Agency for International Development’s Global Development Lab. It was prepared independently by the University of Notre Dame’s Pulte Institute for Global Development, part of the ERIE project under contract AID-OAA-A-16-00025. SAMPLE SMART HOME, PEER VISIT, MOROCCO Report prepared by Danice Brown Guzman, Associate Director Lila Khatiwada, Senior Research Associate Paul Perrin, Evidence and Learning Director Jacqueline Shrader, Independent Consultant Jess Young, Data Scientist, Notre Dame Center for Research Computing Pulte Institute for Global Development University of Notre Dame 3150 Jenkins Nanovic Halls Notre Dame, IN 46556 Acknowledgements The authors would like to extend their thanks to Maria Paula Unigarro Alba (Consultant, Colombia), Ayoub Ait Ali (Consultant, Morocco), Shahzoda Alikhanova (USAID Uzbekistan), Shahina Bakhramova (Consultant, Uzbekistan), Marlyn Faure (Consultant, South Africa), Paulo Gomez (USAID Colombia), Salma Kadiri (USAID Morocco), Anna Lande (University of Notre Dame MGA candidate), Haseeb Rahimi (Consultant, Afghanistan), and Randy Sanchez Suarez (Consultant, Colombia) for their support throughout the project. Special thanks also goes to the National Academy of Sciences and the PIs, universities, government officials, and USAID officers for participating in interviews. ABSTRACT This evaluation assesses the impact of the Partnerships for Enhanced Engagement in Research (PEER) activity on the outcomes of program and policy impact, generation of new knowledge, and capacity building of researchers and students. The retrospective impact evaluation of the first five cycles of the PEER programming adopts a mixed-methods approach. Quantitative analysis of survey and bibliometric data utilized a quasi-experimental matching methodology. We find that the program had an effect on: researchers’ career development, level of respect from colleagues, ability to purchase equipment, and capability to support a few students deeply. Qualitative findings show there were programmatic and policy effects—ranging from local level practices to curriculum changes, opportunities for student learning, and involving local and national policymakers in research. However, in the analysis of quantitative data, we did not detect hypothesized impact on: publication output, production of technology, funding, program, or policy outcomes. In fact, we observe a negative relationship between participation in the PEER program and production of technology and patents. We explore the implications of positive, negative, and null findings. We recommend linking PEER grants with a country’s development strategies with local Missions’ support in selecting, coordinating, and overseeing PEER researchers and sharing research results. TABLE OF CONTENTS EXECUTIVE SUMMARY 4 A. RESEARCH 4 B. RESULTS 4 C. RECOMMENDATIONS 5 1. INTRODUCTION 9 2. BACKGROUND 10 A. DESCRIPTION OF THE PEER PROGRAM 10 B. TIMELINE OF PEER IMPLEMENTATION (HISTORICAL TO PRESENT) 11 C. PEER THEORY OF CHANGE 12 D. GEOGRAPHIC SCOPE OF PEER AND EVALUATION 12 3. METHODS AND LIMITATIONS 13 A. OVERVIEW OF THE EVALUATION DESIGN 14 B. EVALUATION METHODOLOGY 14 C. EVALUATION IMPLEMENTATION 16 D. DATA ANALYSIS 19 E. EVALUATION RISKS AND LIMITATIONS 21 4. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS 22 A. SUMMARY STATISTICS AND BALANCE TABLES 22 B. EVALUATION QUESTION 1 25 C. EVALUATION QUESTION 2 30 D. EVALUATION QUESTION 3 33 E. EVALUATION QUESTION 4 36 F. EVALUATION QUESTION 5 39 G. EVALUATION QUESTION 6 41 H. CONCLUSIONS AND RECOMMENDATIONS 43 REFERENCES 53 ANNEXES 54 A. EVALUATION SCOPE OF WORK 54 B. SURVEY TOOLS 68 C. SUPPLEMENTARY TABLES 90 D. INFORMATION SOURCES 92 1 | PEER EX-POST EVALUATION USAID.GOV LIST OF TABLES TABLE 1: EVALUATION TOPICS AND FINDINGS 5 TABLE 2: EVALUATION QUESTIONS AND METHODS USED FOR DATA COLLECTION 14 TABLE 3: SAMPLE FROM ONLINE SURVEY AND WEB-SCRAPING ACTIVITY 17 TABLE 4: QUALITATIVE DATA COLLECTION 19 TABLE 5: PEER APPLICANT DESCRIPTIVE STATISTICS, BY BALANCE TEST RESULTS 24 TABLE 6: GENERATION OF KNOWLEDGE IN LAST TWO YEARS, SURVEY DATA 26 Table 6A: Generation of knowledge, bibliometric data (cut-off =1 year post application) 90 Table 6B: Generation of knowledge, bibliometric data (cut-off = 2 year post application) 90 Table 6C: Generation of knowledge, bibliometric data (cut-off = 3 year post application) 91 TABLE 7: GENERATION OF KNOWLEDGE, BIBLIOMETRIC DATA 26 TABLE 8: PANEL DATA ANALYSIS OF PEER PI’S PUBLICATIONS 27 TABLE 9: PROGRAM AND POLICY IMPACT, SURVEY DATA 32 TABLE 10: CAPACITY AND LEVERAGED FUNDING 37 TABLE 11: EFFECT SIZES OF SELECTED OUTCOMES 45 TABLE 12: EVALUATION QUESTIONS 48 USAID.GOV PEER EX-POST EVALUATION | 2 LIST OF FIGURES FIGURE 1: PEER STAKEHOLDER SUMMARY 10 FIGURE 2: PEER TIMELINE AND FOCUS AREAS 11 FIGURE 3: PEER THEORY OF CHANGE 12 FIGURE 4: RESPONSE RATE, QUANTITATIVE DATA SOURCES 23 FIGURE 5: CAREER PROGRESS OF PIS, OVER TREATMENT STATUS 28 FIGURE 6: PEER PI’S PERCEPTION OF THE BENEFITS OF THE PROGRAM 39 FIGURE 7: MOTIVATION TO PURSUE RESEARCH QUESTION 40 FIGURE 8: ALTMETRIC SCORE OF PEER AND NON-PEER RESEARCHERS, PANEL ANALYSIS 42 3 | PEER EX-POST EVALUATION USAID.GOV ACRONYMS AOR USAID Agreement Officer’s Representative CEM Coarsened Exact Matching CDCS USAID Country Development Cooperation Strategy CDR Center for Development Research COVID-19 2019 Novel Coronavirus DOI Digital Object Identifiers ERIE Expanding the Reach of Impact Evaluation KII Key Informant Interview Lab/CDR USAID Global Development Lab’s Center for Development Research LAC Latin America and the Caribbean MDM Mahalanobis Distance Matching MENA Middle East and North Africa NAS National Academy of Sciences NASA National Aeronautics and Space Administration NGO Non-Governmental Organization NIH National Institute of Health NOAA National Oceanic and Atmospheric Administration NSF National Science Foundation PEER Partnerships for Enhanced Engagement in Research PI Principal Investigator PIRE Partnerships for International Research and Education PIRS Performance Indicator Reference Sheets ToC Theory of Change US United States of America USAID United States Agency for International Development USDA United States Department of Agriculture USG United States Government USGS United States Geological Survey USAID.GOV PEER EX-POST EVALUATION | 4 EXECUTIVE SUMMARY The U.S. Agency for International Development’s (USAID’s) Partnerships for Enhanced Engagement in Research (PEER) is a grant program that funds international local scientists in USAID partner countries to partner with U.S. government-funded researchers. The PEER program seeks to build the capacity of local researchers to generate critical evidence, lead international collaborations, and apply evidence to program and policy. The program has sponsored 350+ research projects that aim to fill evidence gaps to inform programmatic and policy decision-making. The intended impact of the PEER program is to increase the capacity of USAID partner countries to use research to address development challenges. This study retrospectively evaluates PEER programming to assess the extent of policy, capacity and research impacts. A. RESEARCH An evaluation team, working through the Expanding the Reach of Impact Evaluation (ERIE) mechanism, utilized a quasi-experimental matching methodology to evaluate the outputs of PEER-funded researchers. As the theory of change implies a long timeline for academic and policy impact to be realized, this study focuses on researchers in Cycle one to five, who applied to PEER between 2012 and 2016. At least four years passed between application to PEER and measurement of impact. The research team addressed six evaluation questions, which focused on the impact of the PEER program in terms of program and policy changes, production and influence of academic products, and capacity advancement of researchers and students. ERIE reviewed 1,937 total PEER applicants, both rejected and accepted, across 54 countries to compare research output between 2012 and now. The evaluation process included: document reviews, an online survey of all PEER applicants, web-scraping, bibliometric analysis, key informant interviews, and focus group discussions. Quantitative analysis employs a quasi-experimental matching methodology. The ERIE team traveled to three countries (Uzbekistan, Colombia, and Morocco) to conduct in-person interviews and site visits. Researchers from one additional country (Afghanistan) flew in to participate in interviews. Due to travel restrictions related to COVID-19, the research team completed qualitative data collection remotely with researchers in South Africa and Mozambique. B. RESULTS The results of the evaluation are mixed. The qualitative research team found effects of the program on policy and programming, academic output, production of academic output, and career advancement of both researchers and students. Analysis of the quantitative data did not produce such results. With the notable exception of career advancement, we are not able to detect a statistically significant impact on any of the outcomes analyzed. 5 | PEER EX-POST EVALUATION USAID.GOV PEER grantees are 10 percent more likely to have received a promotion, an increase in salary, or an additional paid or advisory role compared to PEER applicants that did not receive funding. This result could reveal an element of prestige and respect gained by PEER researchers. PEER researchers have the ability to purchase high-quality equipment, bring in funding from the United States, and collaborate with U.S. government-funded researchers to produce high-quality output. There is, however, a negative relationship between participation in the PEER program and the production of technology or patents. This could be due to the focus of the research selection process on development and policy-related issues. In addition, the focus of the PEER program activities may have prioritized policy-related output over technology or patent production. Our inability to detect an effect in the quantitative analysis on the other indicators including academic output, policy impact, receiving additional funding, or other capacity building outcomes could imply that there is truly no effect of the program on these indicators. Another explanation is that there is a true effect but that we do not have sufficient statistical power to detect it. We explore both possible explanations in the conclusions section. C. RECOMMENDATIONS The PEER program has an ambitious theory of change which posited outcomes on grantees’ academic outputs, influence, policy impact, and career progression of themselves and their students. We observed that the PEER grant had an impact on career progress, but we do not observe the other posited impacts in this evaluation. Several PEER grantees identified a misalignment between their research agenda and the priorities of government officials. PEER researchers may achieve greater policy impact if their research bridged the gap between outputs geared towards academic research audiences and the needs TABLE 1: EVALUATION TOPICS AND FINDINGS EVALUATION TOPICS QUANTITATIVE FINDINGS QUALITATIVE FINDINGS 1 & 2. Program and Policy Impact No effects on publication output/influence Negative effect of the program in technology and patents produced Factors for policy and program success: relationship with government, ability to conduct outreach, and administrative support 3. Evidence influencing policy- makers Presenting data in non-technical terms, workshops, and presentations 4. & 5. Capacity Level Outcomes Positive effect on career progress No effect on students’ careers No effect on leveraging funding Factors influencing capacity: Equipment and infrastructure Masters and doctoral-level students 6. Motivation of researchers to apply for funding Some variation by region Most frequent motivations: solve a development challenge or academic interest Similar variation by region Frustration when research priorities of PI did not align with those of the host country government USAID.GOV PEER EX-POST EVALUATION | 6 of government development priorities. Drawing upon the evaluation findings, we offer the following recommendations by evaluation question: EQ 1 AND 2. PROGRAM AND POLICY IMPACT • Support principal investigators (PIs) to form policy suggestions that would have a greater success of uptake. This could include advocating policy suggestions on behalf of the PI or providing training to PI’s on how to translate scientific data into policy briefs. Moreover, PIs face constraints in transforming the results of their research into academic publications, mostly due to other work load. USAID could provide support to the PIs in the form of workshops or paid time for producing both academic or policy-related research products. • Solicit transparent input on policy priorities. Otherwise involve host government and/or USAID staff at the proposal stage and beyond so PIs could apply research that directly supports development or policy priorities. A USAID or government collaborator could participate in the proposal and research design to ensure there is overlap between relevant data and development goals within the country to create effective policy and programming, or to message evidence for policy and programming more effectively to local stakeholders. • Base PEER calls for proposals on USAID’s Country Development Cooperation Strategy (CDCS) for the country to increase research utility. • For research projects with the highest policy relevance, work with host governments and, if funding is available, leverage government research dollars for PIs to sustain or build upon their research post-PEER. This can act as a double incentive by allowing USAID to incentivize PIs who do policy-relevant research with potential for future funding while also ensuring that the policy-makers themselves are invested in the research outcomes. EQ 3. EVIDENCE INFLUENCING POLICY-MAKERS • Support governments in their capacity to analyze data in order to create effective evidence-based policy. This could include inviting technical specialists from different ministries within a government to capacity￾building workshops led by PIs or receiving direct support from USAID. • Create a research database hosted by Missions that would include a contact directory where other researchers could find out what work was done within their specific field. This database would be housed within their government so technical staff and other researchers could access it to discover past researchers and their findings. EQ 4 AND 5. CAPACITY LEVEL OUTCOMES • Develop a prior agreement between the procurement department and PEER to ensure that the research process is easy and efficient for the PIs, prior to research. PEER could also do capacity strengthening on research administration with host institutions to ensure streamlined procurement policies, systems, and approaches are in place. 7 | PEER EX-POST EVALUATION USAID.GOV • Streamline the reporting process to allow more time for PIs to focus on research rather than reporting. This could include either less frequent reporting (i.e. every trimester or every six months) or fewer demands within the quarterly reports. Upon receiving the grant, the supporting university staff in the grant or department offices could receive training to help provide a base introduction to USAID￾specific grants and troubleshoot any common challenges. • Increase the interactions and collaborations for increased utility of research between USAID staff and PIs. If the proposals are in line with the CDCS and if there is a provision that connects local USAID Mission staff with PEER research based on his/her sectoral expertise. This would increase the interaction and collaboration between USAID Mission and PEER PIs. • Establish a mechanism within PEER to connect prospective in-country PIs with U.S. based researchers. Some respondents said that the requirement of partnering with a U.S. government-funded researcher is not easily achievable if they did not have ongoing collaborations. This also limits their ability to prepare a strong proposal, which seems problematic for new faculty. For EQ 6, (Regional Variation in motivation of researchers to apply for funding), we did not detect statistically significant differences in regional motivation of researchers to apply for PEER funding, and our findings were not always consistent with the qualitative data. Therefore, we do not recommend targeting researchers in different regions according to their motivations for conducting research. Instead, we recommend targeting research questions which relate to government priorities or USAID’s CDCS, as described in Recommendation #3 under EV 1&2, and Recommendation #3 for EQ 4 & 5. THEORY OF CHANGE AND EVALUATION- RELATED RECOMMENDATIONS: • Revise the PEER Program Theory of Change (ToC). The ToC could more fully integrate the role that policy-makers play in achieving the program goal of improved development outcomes through their need for, access to, and utilization of research outputs. Revisions to the ToC could be based on a variety of frameworks, including public goods or principal-agent, depending on the viewpoint of the programmatic team. A second point to consider in a revised ToC is that research capacity-level outcomes, such as purchases of needed equipment, university infrastructure support, career progress, and individual student support could also be included as a development-level outcome. • Refine outcome variables for future evaluations. This evaluation included a large number of dependent variables, some were not necessarily aligned with PEER programming. Testing many potential dependent variables can reduce survey response rate and cause statistical issues from multiple hypothesis testing. We recommend a reduction in the dependent variables for future evaluations. Future studies will be able to focus on outcomes that are reasonable to PEER program’s ToC. • Revisit a PEER evaluation after all cycles are complete. Our analysis demonstrated that the impact of some dependent variables could not be detected due to an under-powered study. Academic output such as number of publications and Altmetric score could have been detected with a reasonably larger sample. Researchers who apply to PEER have many publications over a period of up to twenty years; therefore, we recommend a time series analysis to take advantage of this nature of the data. USAID.GOV PEER EX-POST EVALUATION | 8 Future PEER programming should be aware of how the relationships between PIs and USAID Missions, hosting higher education institutions, and governmental/non-governmental organizations affect intended research impacts upon policy. The recommendations above aim to support PEER in achieving this goal. 9 | PEER EX-POST EVALUATION USAID.GOV 1. INTRODUCTION Over the course of a nine-year cooperative agreement, PEER has funded over 300 scientists and engineers across geographies and thematic areas of research. The PEER program directly supports scientists in USAID partnered countries through 2-3 year institutional research awards ranging from $40,000 to $80,000 per year. Understanding the underlying capacities of researchers, research institutions, and support system staff that lead to a successful PEER project can help inform the current PEER Program as well as future planning and execution for a potential future iteration of the PEER program. The PEER mid-term evaluation report (2017) provided early conclusions about the extent to which PEER has achieved program goals. The report, however, was not able to fully assess PEER’s broader impacts and could not draw conclusions about attribution. In this context, the USAID Global Development Lab’s Center for Development Research (Lab/CDR) is interested in understanding the contribution of PEER research to local policy, development programming, and improved research capacity. Through the ERIE mechanism, CDR commissioned an evaluation of the PEER portfolio.1 The evaluation engaged researchers--including PEER-funded researchers and non-PEER-funded researchers in the same field and/or university-- along with affiliated ecosystem actors, to better understand the impact of the program in the key areas of knowledge production, career development, and program and policy change. Specifically, the following research questions were explored through the present evaluation: 1. What factors increase the likelihood of achieving program and policy impacts? How has the PEER grant helped to generate new knowledge and be used for policy-making? 2. To what extent and how has PEER contributed to program and policy impacts? 3. What kind of evidence is most likely to influence policy-makers? 4. To what extent and how has PEER contributed to capacity-related outcomes? (publications, students with degrees, data production) 5. What factors (at the level of PI and institution) increase the likelihood of achieving capacity￾related outcomes? (publications, students with degrees, data production) 6. Are there regional differences of motivational factors for involvement in PEER funded activities? a. Which region has higher involvement of researchers motivated by solving specific development problems? b. Which region has higher involvement of researchers motivated by academic interest? 1 ERIE is an approach to conducting retrospective long‐term impact evaluations of completed USAID interventions. For more information on ERIE see: https://www.usaid.gov/GlobalDevLab/MERLIN/ERIE USAID.GOV PEER EX-POST EVALUATION | 10 2. BACKGROUND A. DESCRIPTION OF THE PEER PROGRAM The PEER program’s conceptual model identifies three distinct stakeholders (Figure 1). Represented at the middle of the model are supporting stakeholders, which include USAID Agreement Officer’s Representative (AOR) and PEER Relationship Managers, the U.S. scientific community, and University administrators who provide financial and other support to PEER researchers. The PEER program extends its reach across the U.S. scientific community to include the following partner agencies: National Aeronautics and Space Administration (NASA), National Institute of Health (NIH), National Oceanic and Atmospheric Administration (NOAA), National Science Foundation (NSF), United States Department of Agriculture (USDA) Agricultural Research Service, USDA Forest Service, USDA National Institute of Food and Agriculture, United States Geological Survey (USGS), and the Smithsonian Institution. In addition to these, there are many US based universities that collaborate with researchers. Producers of research comprise the Principal Investigators (PIs) and Co-PIs in universities and research institutions who receive support from the aforementioned supporting stakeholders to conduct research. These actors also directly collaborate with the users of the research. Finally, there are users of research who need data generated by research to make informed academic, policy or program level decisions. These actors directly or indirectly work with the research producers to understand their needs and generate and supply the data that is important for affecting policy change. All three entities work together for the success of PEER research. We included all the stakeholders in our evaluation by reaching out to all three categories of people during the data collection. FIGURE 1: PEER STAKEHOLDER SUMMARY 11 | PEER EX-POST EVALUATION USAID.GOV B. TIMELINE OF PEER IMPLEMENTATION (HISTORICAL TO PRESENT) PEER started its funding cycle in 2012 with a focus on science-related research. Under Cycle 1 in 2012, there were two focus areas: child survival research and Indonesia-focused research. Altogether, 41 projects were funded during Cycle 1. In between the Cycle 1 and 2 there was a special PEER-PIRE (Partnerships for International Research and Education) Cycle, supported by NSF, and three projects were funded under this provision. Following are the details about the subsequent Cycles: • During Cycle 2, the focus was PEER Science and 54 projects were funded; • During Cycle 3, again the focus was PEER Science and 38 projects were funded (during this cycle there were two additional funding opportunities through NIH focusing on health); • During Cycle 4, 45 projects were funded; • During Cycle 5, 50 projects were funded; • During Cycle 6, 27 projects were funded, • During Cycle 7, 26 projects were funded; and, • During Cycle 8, 33 projects were funded. • Currently, Cycle 9 applications are being accepted. During its first five years, PEER placed an emphasis on increasing the number of projects, improving the relationships with Missions/buy-in, and streamlining the solicitation and awards processes (Figure 2). In the ongoing second half of the program, PEER emphasizes project impact, translation of results to key stakeholders, and capacity building activities in partner institutions. From PEER Cycle 5 onwards, a section on Government & NGO partner collaboration was added and the development impact section expanded in the application. Grants awarded from 2016 to 2018 included opportunities to apply for evidence to action or travel grant funding for U.S. partners. FIGURE 2: PEER TIMELINE AND FOCUS AREAS Source: Internal documents (PEER Team Planning PowerPoint) USAID.GOV PEER EX-POST EVALUATION | 12 C. PEER THEORY OF CHANGE Figure 3 depicts the PEER Theory of Change as developed by the program team, beginning with input activities in the form of financial support provided to the PIs and institutions. PEER provides resources to researchers for generating evidence which might have policy implications. Funding builds their capacity to develop and implement high quality research in development priority areas. They also receive capacity building support with project management and reporting. As an output of these activities, research projects are designed and conducted. Research activities funded through PEER lead to new evidence that will improve researchers’ country’s development objectives. This research will have long-lasting development impacts if applied successfully in policy-making processes. Development outcomes can be achieved through many mechanisms; in some cases, these mechanisms are policy changes at the national or regional level. In that case, one barrier to achieving development outcomes could be misaligned priorities between policy-makers and researchers, in terms of which topics should be researched or which policies which should be implemented as a result of the research. Researchers may face barriers when they attempt to get policy-makers to change priorities in response to findings. In addition, this theory of change does not consider the capacity building of researchers, students, or even universities as an impact, only as an input. Some literature demonstrates that grant funding can have an impact on career progress. For example, Ganguli (2017) investigates the impact of grants on the persistence of scientists in the career path of scientific research, which she defines as the extensive margin. This report discusses such challenges in alignment towards the goal of achieving policy change, and explores a wider array of potential impact of the PEER program. FIGURE 3: PEER THEORY OF CHANGE Source: Internal documents (PEER Performance Management Plan) D. GEOGRAPHIC SCOPE OF PEER AND EVALUATION To date, the PEER Program has spanned over 54 countries: 16 in Asia, 21 in Africa, seven in Latin America and the Caribbean (LAC), five in the Middle East/North Africa, and five in Europe and Eurasia— 13 | PEER EX-POST EVALUATION USAID.GOV all of which have been eligible for general call. Only specific groups of countries were eligible for target calls, such as a family planning-related research call that pertained to six countries in Asia. The evaluation focused on all the countries that have PEER grants from PEER Cycles 1 to 5. All Cycle 1- 5 applicants, regardless of acceptance, were contacted and asked to participate in our survey. We also selected four countries to host in-person qualitative interviews with various stakeholders, with interviews in two additional countries involving fly-in PIs. In selecting the countries for in-person visits, we aimed to achieve geographical diversity across the regions, and to include countries where Mission or stakeholder investment was high and those where it was not. 3. METHODS AND LIMITATIONS To address the research questions, our evaluation team adopted a mixed methods ex-post evaluation approach that aims to assess the contributions that PEER made to research production, researcher capacity improvements, and the production of evidenced-based policy. This evaluation also aims to provide a deeper understanding of how factors at the university, Mission and researcher levels contributed to differences in outputs and outcomes. The ex-post evaluation draws on a variety of methods: notably a desk review, online surveys, interviews with key stakeholders, web scraping, research product document reviews, and analyses of administrative data. Below we describe the approach, data sources and implementation of both the quantitative and qualitative data collection. USAID.GOV PEER EX-POST EVALUATION | 14 A. OVERVIEW OF THE EVALUATION DESIGN B. EVALUATION METHODOLOGY The evaluation employed a mixed methods approach. Quantitative data for this evaluation includes existing data from PEER applications, responses from an online survey, and data collected on all PEER applicants’ research output through a web-scraping activity. Our analytical approach for these data sought to answer the evaluation questions through an approximation of a counterfactual. In other words, the counterfactual attempts to answer the question: “What would have been the outputs of PEER researchers in the absence of PEER funding?” SAMPLING APPROACH We cannot adopt a true experimental approach to approximate the counterfactual, as this ex-post evaluation lacks an established control group. Instead, we utilize a quasi-experimental matching TABLE 2: EVALUATION QUESTIONS AND METHODS USED FOR DATA COLLECTION EVALUATION QUESTIONS BIBLIOMETRIC ANALYSIS ONLINE SURVEY FOLLOW-UP PHONE INTERVIEW IN-PERSON INTERVIEWS 1. What factors (at the levels of PI, institution, and country) increase the likelihood of achieving program and policy impacts? How has the PEER grant helped to generate new knowledge and be used for policy￾making? X X X X 2. To what extent and how has PEER contributed to program and policy impacts? X X 3. What kind of evidence is most likely to influence policy-makers? X 4. To what extent and how has PEER contributed to capacity-related outcomes? (career progress, students with degrees, data production) X X X 5. What factors (at the levels of PI, institution, and country) increase the likelihood of achieving capacity￾related outcomes? (publications, students with degrees, data production) X X X 6. Are there regional differences of motivational factors for involvement in PEER funded activities? X X 15 | PEER EX-POST EVALUATION USAID.GOV methodology. We have the ability to adopt such a methodology for two reasons. First, the program maintained a database of all applicants to the PEER program— both those applications which were funded and those which were not. Second, the program maintained meaningful documentation on the selection process of the applicants, including scoring and recommendation procedures by review panels. We are able to employ quasi-experimental matching methods as described below using these points of information. To approximate the counterfactual, we observe the outputs and outcomes of PEER applicants who ultimately did not receive funding from the PEER program.2 These are individuals who selected into the cohort by applying to the program— meaning they believed they met the basic qualifications for the program. These individuals were rejected from the PEER program for a variety of reasons. Some applications were likely rejected because they were not competitive relative to the successful applicants. Others were rejected due to a lack of sufficient resources to fund all competitive grant applications in a cycle. On average, the non-funded applicants are not comparable to the PEER grantees in terms of: year of application, field of research, and panel reviewer score (see Table 5). These differences may affect our ability to estimate the true impact of the program. We explored various matching methodologies to ”prune” the data toward a more balanced sample, in which individual treatment observations are compared to controls that are on average more similar in a set of key potential confounding indicators that are specified by the analyst. Such methods have been used in observational studies or in primary research where randomization is not feasible in various fields of research including education, health, epidemiology, water and sanitation, and social welfare studies.3 The evaluation team reviewed the data from the PEER applications provided by the National Academy of Sciences (NAS). The application data set includes: the title of the project, the sector, the gender of and highest degree earned by the principal investigator, the country of the research, the budget of the project, and reviewers’ scores of the application. We first ran balance tests to confirm that the raw data demonstrates that balance is not achieved between the treatment group (PEER-funded researchers) and the comparison group (non-PEER funded researchers) on some of these indicators. We then explored three matching methods. First, we employed coarsened exact matching (CEM), which improves balance by an amount specified by the analyst through temporarily coarsening the variables used for matching and matching on the coarsened variables (Iacus et al 2012). Second, entropy balancing employs a reweighting method to achieve balance on the specified matching variables while preserving as many of the observations as possible (Hainmueller 2012). Third, Mahalanobis distance matching (MDM) calculates a “distance” between each treatment and comparison observations, which then pares down the data to matched pairs among the nearest matches (Cochran & Rubin, 1973). The Findings section first presents statistics on how all PEER researchers compared with all non-PEER researchers (the population), and then compares the balanced group of PEER and non-PEER researchers selected by the matching methodologies (the analytical sample). 2 Note that the comparison group did not receive funding from PEER, but they could have received funding from another source. This is discussed in the findings and conclusion sections. 3 For a summary of existing studies using one popular matching methodology, propensity score matching, see the introduction to Guo and Fraser 2010. USAID.GOV PEER EX-POST EVALUATION | 16 QUALITATIVE METHODOLOGY The qualitative methodology utilizes semi-structured interviews with key questions that were informed by mid-term interview guides, the predefined evaluation questions, and preliminary results from the online survey. The semi-structured interview guides were reviewed and revised by key stakeholders at USAID, and translated into local languages when necessary. The evaluation team sought to interview all PEER-funded PIs within the selected country that were funded in Cycles 1-5, as well as, corresponding research users identified for each PI. No qualitative interviews were conducted with non-PEER researchers (the comparison group). The research team selected countries that vary in region and economic development (low and middle income), while prioritizing countries with a larger count of PI’s. Within the region, we also included countries where there are co-funding mechanisms from the government, e.g., South Africa and Morocco. Additionally, we looked at other opportunities (e.g., possibility of fly-in PIs from neighboring countries, as in the case of fly-in PIs from Afghanistan in Uzbekistan). Based on these criteria, we selected following countries: • Asia: Uzbekistan (with fly-in PIs from Afghanistan), • Africa: S. Africa (with fly in PIs from Mozambique if feasible), • Middle East: Morocco, and • Latin America: Colombia. The analysis of qualitative information is based on the variability and uniqueness of responses, using Atlas-ti software for consolidating qualitative information. Codes were developed based on the evaluation questions and the narrative content to guide the analysis. In addition to presenting stand￾alone qualitative analysis results based on common patterns, we also identified quotes that reinforced the findings of corresponding quantitative results in the report. The findings from the qualitative analysis are aggregated across the countries, although we identify particular countries when providing illustrative examples. C. EVALUATION IMPLEMENTATION This section describes the process of implementing the evaluation through the lens of each data collection modality within the overarching mixed methods approach. Some phases were undertaken concurrently, but this section aims to present the process of implementation in roughly chronological order. DOCUMENT REVIEW The evaluation team first reviewed existing documentation of the PEER program, and other literature which informed our quasi-experimental design. We reviewed the PEER application data, the Performance Indicator Reference Sheets (PIRS), and the midterm evaluation report (Benschoter et al, 2017). This informed the selection of the evaluation questions, the design of the online survey tool, and selection of key variables to be used for matching. We also identified and reviewed relevant literature for evaluating research grants programs. This research was identified through a search by lead researchers. Several studies attempt to quantify the effects of research funding on various outcomes, such as: career progress, number of publications, 17 | PEER EX-POST EVALUATION USAID.GOV citations, and migration (Arora et al, 2005; Ganguli, 2017; Jacob and Lefgren, 2007). This literature informed the selection of outcomes, the selection of the matching methodologies, interpretation of results, and the data cleaning methods. The evaluation team was grateful to build upon the existing body of literature that evaluates global impacts of research funding. ONLINE SURVEY The online survey was drafted by the evaluation team in consultation with key stakeholders. We aimed to ensure not only that survey questions were consistent with those asked in the midterm evaluation, but also that they were relevant to both PEER and non-PEER funded researchers so they could be asked to both groups. Due to concerns about recall, some questions were asked in reference to two time periods— since applying to the grant and in the last two years. The survey was drafted on paper, and then programmed in Qualtrics for online distribution. The survey was opened to respondents twice, in October 2019 and January 2020, to increase response rate. Response rates are reported in Table 3 below, and visualized in Figure 4. WEB-SCRAPING AND BIBLIOMETRIC ANALYSIS To gain more information about research output, we also conducted a web-scraping activity that searched for all research output of each researcher in the dataset. We created a triangulated list of digital object identifiers (DOIs) of all publications from researchers in our dataset. Our first source of information for this was the list of PEER researcher output provided by the PEER program itself. Next, we requested this information through the online survey, where we asked researchers to upload a CV, or provide a link to a research page on Google Scholar or ResearchGate. Publication information was pulled from these sources. Finally, if the researcher did not respond to the survey, we manually searched for their Google Scholar or ResearchGate pages. Once the researchers’ Google Scholar IDs or ResearchGate websites were determined, their publications were scraped from their profiles. The R package scholar was used for Google Scholar profiles and the R package rvest was used to help scrape the ResearchGate profiles. DOIs were scraped from these sites using Python or R. In order to find DOIs, the CrossRef database was used via the rcrossref package in R. TABLE 3: SAMPLE FROM ONLINE SURVEY AND WEB-SCRAPING ACTIVITY PEER GROUP NON-PEER GROUP Total Applicants 251 1686 Online Survey (% of total) 128 (51%) 431 (26%) Researchers with publications found (% of total) 145 (57%) 950 (56%) Publications with DOIs 2703 23056 Publications with Altmetric scores 780 5796 USAID.GOV PEER EX-POST EVALUATION | 18 The list of publication DOIs was then provided to Altmetric’s API via the Altmetric package for bibliometric analysis. Altmetric is an online platform which provides dynamic data on outcome including number of citations and the number of mentions in news and social media. Altmetric then creates a comprehensive influence score for each publication. An example of an Altmetric dashboard can be found here. Altmetric also produces an Attention Score for each publication. The score is derived from an automated algorithm, and represents a weighted count of the amount of attention which has been picked up for a research output through various mechanisms (online, citations, social media). You can find more information about how this score is calculated here. PHONE FOLLOW-UP OF ONLINE SURVEYS We used in-depth phone interviews with some positive outliers who responded to our online survey— people who were demonstrably successful in publishing, working with policy makers, and training more students. The phone interviews followed the same script that was used for qualitative PI interviews, with the addition of a select number of questions to understand how the PIs were able to achieve these positive results. We selected 10 PIs for the phone interviews and were able to speak with nine of them. The same analysis process and coding structure was employed for these transcripts as were used for the key-informant interviews. KEY-INFORMANT INTERVIEWS AND FOCUS GROUPS (IN COUNTRY/ZOOM) The evaluation team visited three out of four selected countries for collecting primary data. All in￾person interviews with the PIs and other stakeholders were scheduled before the evaluation team left the U.S. Members of the evaluation team visited the working locations of the PIs and other respondents. In Uzbekistan, the evaluation visit corresponded with a PEER event that brought all the PIs from Uzbekistan and Afghanistan into a common location. When possible, questions were shared with the interviewees ahead of time so that they could be prepared for the interview. Each interview lasted for approximately an hour. Interviews were recorded, transcribed, and translated when necessary. For each PI, we also interviewed university officials who provided research administration support, in addition to a demand-side stakeholder, such as a policy-maker from the government or a non-governmental organization representative with whom the PIs worked with to share the research findings. All the interviews were recorded with the respondent’s permission. The evaluation team adapted its strategy to the context of COVID-19. The team could not visit South Africa because of travel restrictions caused by COVID-19. The evaluation team worked with a local consultant who scheduled and conducted interviews using remote options. If time permitted, the evaluators from the U.S. also joined the call. The cancellation of the South Africa visit also meant that we could not fly PIs from Mozambique to South Africa for interviews. We compensated by conducting Mozambique interviews remotely from the U.S. Overall, we planned 28 in-person interviews in South Africa. After switching to remote interviews, we were able to conduct 17 interviews. We interviewed 14 policy makers from five countries: two from Uzbekistan, two from Afghanistan, four from Colombia, four from South Africa and one from Morocco. In Uzbekistan, we spoke with the heads of the related ministry and department. In Afghanistan, we spoke with a Project Manager of Climate Change Adaptation Project from the Ministry of Agriculture, and an Advisor of the Dam and River project in the Ministry of Energy and Water. In South Africa, we spoke with a Senior Manager of the Conservation Management department, a South African National Parks official, and a Principal Scientist 19 | PEER EX-POST EVALUATION USAID.GOV in a Municipality. In Colombia, we spoke with a ranger of a national park, head of an environment management department, and local NGO representatives. In Morocco, we spoke with an official from the Higher Council of Education, Training, and Scientific Research. D. DATA ANALYSIS QUANTITATIVE DATA Quantitative data analysis focused on understanding how various independent variables predicted dependent variables (or outcome variables). The key independent variable is the treatment variable (receiving the PEER grant). Covariates included time since receiving the grant, gender of the researcher, field of study, geographic region, overall reviewer score of the PEER application, and mission relevance score. The specific dependent variables to be included in the analyses, along with their corresponding data sources, are specified at the beginning of each section in the findings section. Dependent variables of interest include: research outputs (in terms of policy, training materials and academic output), the ability to leverage funding to earn more grants, research influence (citations, Altmetric score, mentions in social media), student success, career promotion, and self-perceived efficacy of research output. In the survey data, we aimed to avoid multiple hypothesis testing by creating indices and summative responses when possible. For example, when asked about research output, the respondent could specify counts of many types of research outputs (such as datasets, patents, technology, papers, curriculum materials, and others). These categories of research output were classified and totaled into categories, such as: academic, training, policy and technology. For perception-based questions, we created indices on certain topics such as, Self-Assessed Efficacy of Research. To assess the goodness of fit of these TABLE 4: QUALITATIVE DATA COLLECTION DATA SOURCE STAKEHOLDER GROUP SAMPLE FRAME TOTAL COMPLETED (%) Key Informant Interviews and Focus Groups USAID Missions 6 5 (83%) USAID Operating Units 1 1 (100%) PIs/Co-PIs 28 22 (78%) Students 1 (NA) University officials 28 13 (46%) Policymakers 28 14 (50%) Follow-Up Phone Interviews PIs/Co-PIs 10 9 (90%) Country visit Select group of countries 4 3 (75%) USAID.GOV PEER EX-POST EVALUATION | 20 indices, we calculated and examined the Cronbach’s alpha of the questions to ensure that an index was relevant. We present the alphas of each index created in the relevant findings sections. In the bibliometric data, each publication was coded as “before” or “after” applying to the PEER grant. Because the timeline for publications is long, we explored different cut-offs (immediately upon receiving the grant, one year, two years, or three years after). The data from both the bibliometric analysis and the online surveys includes extreme outliers. Data for all continuous variables has therefore been top-coded at the 99th percentile, meaning that any response which falls above that value is re-coded to equal the value which is at the 99th percentile. ANALYTICAL MODEL Our basic model examines the effect of receiving a PEER grant on any given outcome variable we call here y1. In all models we control for the variables we used for the matching exercise, as well as a time variable which indicates years since the application was submitted. For the time variable we also include a squared term because we hypothesize that the relationship between time and publication output is not linear. Finally, we include fixed effects for each cohort, and 𝜀𝜀 represents the error term. Basic Model: Survey Data 𝑦𝑦1 = 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃 + 𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 + 𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇2 + 𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶 + 𝑐𝑐 ℎ 𝐹𝐹 + 𝜀𝜀 In the bibliometric analysis, we control for the pre-treatment level of the outcome variable as well. For example, if the outcome is the count of publications after applying for PEER, we control for the count of publications before PEER. Basic Model: Bibliometric Data 𝑦𝑦1 = 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃 + 𝑦𝑦0 + 𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 + 𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇2 + 𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶 + 𝑐𝑐 ℎ 𝐹𝐹 + 𝜀𝜀 For each outcome variable we run three versions of the model. First, we run the model using the raw data. Next, we repeat the same analysis on the weighted dataset created by entropy balancing. Finally, we repeat the model a third time on the dataset created by the MDM method, clustering by the matched pairs. Clustering by the matched pair is recommended by literature on matched pair analysis (Abadie and Spiess, 2019; De Chaisemartin and Ramirez-Cuellar, 2020). Most questions in the survey target PEER and non-PEER researchers alike. However, a few questions are relevant to PEER researchers only. These include perceptions of the utility of various PEER activities such as the networking opportunities. For data collected on PEER researchers only, the more rigorous analytical approach described above is not relevant. Therefore, only summary statistics are produced. Summary statistics are disaggregated by region4 or cohort when relevant. These statistics are presented with the results of the qualitative data collection, to provide a richer understanding of the outcomes of the PEER program. 4 Regional definitions were created to track onto the regions listed by the PEER program on their website: Sub￾Saharan Africa, Asia/Pacific, Europe/Eurasia, Latin America and Middle East/North Africa (MENA). 21 | PEER EX-POST EVALUATION USAID.GOV QUALITATIVE DATA The qualitative data analysis largely followed a concept-driven coding process—meaning that prior to analysis, key codes were prepared based on the evaluation questions. During the coding, transcripts were reviewed carefully and segments of the responses that fit one of the pre-defined criteria were coded accordingly. Also, if new information was discovered that was not covered by the concept-driven codes, a new open code was generated, assigned, and added to the overall codebook. Atlas.ti qualitative software was used for the coding exercise, which provides an opportunity to create the summary codes to observe the patterns emerging in the data. We used summary codes for interpreting the results. E. EVALUATION RISKS AND LIMITATIONS In this section, we present limitations to this evaluation, including limitations to the design and the effects of COVID-19 on our fieldwork. We note below if we believe that these risks could bias our results, if they vary by treatment status. These should be taken into consideration when interpreting the findings of this study. RESPONSE RATES TO SURVEY Low response rates likely introduce bias to our data. We sent regular emails reminding individuals to participate in the survey, and repeatedly followed up with invited in-depth interview respondents, but at the end of the day, the respondents decide if they participate in the evaluation. We see that response rates to the survey do vary by treatment status: a larger proportion of the PEER-funded researchers responded to our survey than non-funded PEER applicants. We have attempted to mitigate this risk through the use of matching methods in the quantitative analysis, which aim to create two comparable groups, but it should still be considered when interpreting the results. In addition, the web-scraping activity can be subject to bias, particularly when searching for authors’ names from alternative writing systems which may have variations in spellings when transliterated to the Latin alphabet, or for individuals with very common names. This is a common flaw in the method which has been cited in other literature (Ganguli 2017). However, we do not see a reason why this bias would vary by treatment status—it’s likely equally probable that common or difficult-to-spell names exist in the treatment and the comparison groups respectively. Indeed, we find that we are able to find some Altmetric data on a similar proportion of the treatment and comparison groups, supporting our theory that this bias does not vary by treatment status, and therefore should not affect our estimations. RECALL BIAS Any long-term evaluation is bound to face some issues of recall bias. For example, we asked researchers how many times they have posted online about their research since they applied to PEER (up to eight years ago). Because the event of actually receiving a PEER grant could be considered a salient or memorable event, recall bias could vary by treatment status. We have mitigated this risk in two ways. First, we asked the questions with respect to a given year, and do not refer to the PEER application itself in the question. Second, we triangulate researchers’ responses with similar outcomes from the bibliometric data where possible. USAID.GOV PEER EX-POST EVALUATION | 22 COVID-19 Fortunately, our evaluation team completed almost all of the scheduled field visits before the cessation of travel in mid-March due to COVID-19. We were able to visit Uzbekistan and Colombia as planned in October 2019 and early March 2020, respectively. We also visited Morocco in mid-March; however, a small number of these scheduled interviews were moved to phone or Zoom at the last minute due to respondents’ hesitation to meet in person in light of the then rapidly-emerging COVID-19 crisis. COVID-19 mainly affected our scheduled field visit to South Africa. Interviews from respondents in South Africa and Mozambique were moved to Zoom or phone. This change could introduce bias into our data due to the digital divide—only respondents who had access to the internet and strong connectivity were able to participate in these interviews. We are likely missing qualitative data from those who had less access to technology during lockdowns in South Africa and Mozambique. We attempted to mitigate this risk by working with our local consultant who followed up with the PIs with emails and phone calls. The consultant provided various options for interviews: by landline, WhatsApp, Zoom, or Skype— based on key informants’ convenience. The consultant followed up for 18 interviews and was able to schedule 14 interviews. In addition, we were not able to interact with students or visit field sites in Southern Africa, meaning we missed contextual insight with regards to sub-Saharan Africa. 4. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS The findings section will first present summary statistics, then intermix qualitative and quantitative data in response to each evaluation question, as relevant. The section below notes which questions from the PEER survey were requested to be included in response to feedback. A. SUMMARY STATISTICS AND BALANCE TABLES SURVEY RESPONDENT CHARACTERISTICS AND BIBLIOMETRIC DATA The sample for quantitative analysis is included above in Table 3, and depicted visually below in Figure 4. The response rate to the survey varied by treatment status. In the treatment group, about half of PEER PIs completed the survey. In the comparison group, the response rate was about 26%, which comprises over 400 individuals. The bibliometric activity capture rate was more consistent between the two groups. We were able to capture at least one publication of 56% to 57% of researchers in both groups. Our average researcher had 20 publications; so, with over 1,000 researchers, this totals to over 23,000 publications. 23 | PEER EX-POST EVALUATION USAID.GOV FIGURE 4: RESPONSE RATE, QUANTITATIVE DATA SOURCES BALANCE TESTS The results of the balance tests are shown in Table 5. Before performing any matching techniques, we found that the sample is unbalanced on the indicators of: year of application, field of research, and review panel’s score of application. Non-funded applications have a score of approximately 2.6 on average, while funded applications have a score of 3.9. This indicates that matching methods are needed to create a more balanced sample. Entropy balancing method retained a sample of 1096 researchers and achieved balance on all indicators included from the application data. Coarsened exact matching also achieved balance on all indicators, but this method reduced the sample to 227 researchers. Finally, Mahalanobis distance matching resulted in a sample of 342, but over-corrected on the indicator of reviewer score— the comparison group has a higher average reviewer panel score than the treatment group (p<.05). USAID.GOV PEER EX-POST EVALUATION | 24 In the following sections, we present the results of our analysis with regards to each evaluation question. We present regression coefficients from analysis on three samples: 1) the raw data, 2) data created through entropy balancing (EB), and 3) data created from Mahalanobis Distance Matching (MDM). We present the regression coefficients for the independent variable of interest, receiving the PEER grant, TABLE 5: PEER APPLICANT DESCRIPTIVE STATISTICS, BY BALANCE TEST RESULTS Raw Data funded=251, non-funded=1686 Entropy Balanced Data (EB) Funded=249, non-funded=847 Mahalanobis Distance Matched Data (MDM) Funded=171, non-funded= 171 Variables Full Sample Funded Non- Funded p￾value of diff Full Sample Funded Non- Funded p￾value of diff Full Sample Funded Non- Funded p￾value of diff Treatment (0,1) 0.13 1 0 0.5 1 0 0.5 1 0 Gender (% Female) 28.18 30.52 27.83 0.29 30.52 30.52 30.52 1 27.65 28.82 26.47 0.65 Year of Application 2014 2014 2015 0 2014 2014 2014 1 2014 2014 2014 0.42 Region Africa (%) 27.57 25.1 27.94 0.54 24.9 24.9 24.9 1 26.47 26.47 26.47 1 Latin America (%) 18.53 14.74 19.1 0.36 14.86 14.86 14.87 1 15.29 15.29 15.29 1 Asia/Pacific (%) 34.9 43.03 33.69 0.1 43.37 43.37 43.37 1 42.94 42.94 42.94 1 Europe/Eurasia (%) 5.42 5.58 5.4 0.94 5.62 5.62 5.62 1 4.71 4.71 4.71 1 MENA (%) 13.58 11.55 13.88 0.5 11.24 11.24 11.24 1 10.59 10.59 10.59 1 Field of Research Agriculture (%) 12.39 5.18 13.46 0 5.22 5.22 5.22 1 5.29 5.29 5.29 1 Biodiversity (%) 13.01 16.73 12.46 0.12 16.87 16.87 16.87 1 17.06 17.06 17.06 1 Environment (%) 10.38 11.55 10.2 0.55 11.65 11.65 11.65 1 11.76 11.76 11.76 1 Health (%) 15.02 13.55 15.24 0.48 13.65 13.65 13.65 1 12.35 12.35 12.35 1 Water (%) 11.87 18.33 10.91 0.03 17.67 17.67 17.67 1 15.88 15.88 15.88 1 Budget (USD) 191,960 197,572 190,311 0.29 198,346 198,351 198,340 1 194,740 195,777 193,703 0.82 Panel Score (0-9) 2.75 3.93 2.58 0 3.92 3.92 3.92 1 4.2 4.04 4.37 0.01 25 | PEER EX-POST EVALUATION USAID.GOV and note which covariates were included below each table. We also present results from the qualitative data collection, and note where results align and where they diverge. Because Table 5 indicates that the raw data is biased, we focus our interpretation of findings on indicators where we observe effects which are consistent in terms of statistical significance and direction across both matching methods. Occasionally we observe a larger and significant effect with the MDM method only. However, if this effect is not consistent with that found using the other matching method, it is likely spurious. This could be due to outliers in the small sample maintained after use of that matching method. These instances are noted below in the findings section, but our discussion will focus on findings which are consistent across both matching methods. B. EVALUATION QUESTION 1: HOW HAS THE PEER GRANT HELPED TO GENERATE NEW KNOWLEDGE? From the quantitative data, Table 6 presents the results on the topic of generation of new knowledge. We asked researchers to report counts of various types of research output they’ve created in the last two years (see Q23 in Annex B). New technology includes patents and technology produced. Training materials include: courses, training tools and resources, workshops, and blogs. Policy output includes: briefs, program strategies, business plans and market analyses. Academic output, labelled research in the table, includes: peer-reviewed papers, conference presentations, posters, and scientific reports. For each of these we’ve totaled up the outputs identified by the researcher. The results of the analysis do not provide evidence of a positive impact of PEER on the production of training materials, academic output, or policy documents. We do, however, observe a difference between PEER and non-PEER researchers in terms of production of new patents and technology, although not in the direction that might be expected. PEER researchers have produced slightly fewer patents and technology when compared with the non-PEER researchers. Multivariate regression output using the raw data demonstrates an impact of the PEER program in terms of training materials produced. This impact, however, is not sustained when matching methods are used. Using the bibliometric data, we examine the count of publications produced, and several indicators of how much “buzz” these publications caused—the Altmetric Attention score, the number of times it was tweeted, the number of Mendeley readers of the publication, the count of times it was cited in other academic papers, and the number of times it was cited in a patent or a policy document. For all of these indicators, our outcome of interest is the total per researcher, after applying for the PEER grant. We also control for the outcome variable prior to application to PEER. Consistent with the survey data, we see PEER research cited in fewer patents compared with non-PEER researchers. Other than this difference, we do not observe an impact of the PEER program on any of the outcomes coming from the bibliometric data. USAID.GOV PEER EX-POST EVALUATION | 26 Note: *** p<0.01, ** p<0.05, * p<0.1 Source: Survey data. Controls included: region, years since application, reviewer score, gender of applicant, field of research, and cohort fixed effects TABLE 6: GENERATION OF KNOWLEDGE IN LAST TWO YEARS, SURVEY DATA VARIABLE COEFFICIENT: RAW DATA N COEFFICIENT: EB N COEFFICIENT: MDM N New patents and technology -0.46*** 342 -1.05*** 208 -0.82** 85 New Open Access Datasets 0.13 342 0.01 208 -0.08 85 New training 2.14** 342 1.67 208 -0.39 85 New policy output 0.26 342 0.26 208 -0.29 85 New research -0.32 342 -2.43 208 -3.41 85 TABLE 7: GENERATION OF KNOWLEDGE IN LAST TWO YEARS, SURVEY DATA VARIABLE COEFFICIENT: RAW DATA N COEFFICIENT: EB N COEFFICIENT: MDM N Count of publications, after application† 0.04 847 0.17 422 -1.33 130 Average Altmetric score, overall‡ 2.80 1,088 2.14 591 5.34 185 Average Altmetric score, after application†‡ 3.61 847 3.22 422 2.20 130 Count of Mendeley readers, after application†‡ 29.62 847 21.80 422 4.48 130 Count of Tweets, after application†‡ 6.68 847 9.97 422 7.52 130 Count of Patents, after application†‡ -0.02 847 -0.03** 422 -0.07* 130 Count of Policies, after application†‡ -0.07 847 -0.03 422 0.10 130 Count of Citations, after application†‡ 12.09 847 14.88 422 9.17 130 27 | PEER EX-POST EVALUATION USAID.GOV Note: *** p<0.01, ** p<0.05, * p<0.1 Source: Altmetric data. Controls included: region, years since application, reviewer score, gender of applicant, field of research, and cohort fixed effects † baseline value of outcome ‡ count of publications in time period To perform a check on these findings, we examined the publication dataset as a panel dataset. Examining the data as a panel helps us increase our statistical power because we have multiple observations for each researcher. Literature demonstrates that multiple time periods can increase the statistical power of a study (McKenzie 2012). Using the bibliometric analysis, we have publications for time periods of up to 19 years, and over 3000 observations (researcher-years) in total. When we examine the data as a panel, we find that the program produced a statistically significant bump in Altmetric score (p<.1). However, this difference does not continue to grow over time. After experiencing the bump in Altmetric score, the growth rate of PEER PI scores is not statistically different from the growth rate of non-PEER PIs. See Table 8 and Figure 5 below for analysis and graphical representation of this analysis. Standard errors in parentheses, data weighted using Entropy Balancing Method *** p<0.01, ** p<0.05, * p<0.1 TABLE 8: PANEL DATA ANALYSIS OF PEER AND NON-PEER PI’S PUBLICATIONS (1) Average Altmetric Score (2) Count of Publications per year PEER Funded 9.25* -0.27 (5.56) (0.66) Time 0.57*** 0.30*** (0.18) (0.11) Funded X Time -0.47 -0.13 (1.20) (0.14) Constant 9.34*** 3.94*** (1.07) (0.20) Observations (Individual-Years) 1,665 1,665 Observations (Individuals) 368 368 R-squared 0.02 0.08 Individual Fixed Effects Yes Yes USAID.GOV PEER EX-POST EVALUATION | 28 FIGURE 5: ALTMETRIC SCORE OF PEER AND NON-PEER RESEARCHERS, PANEL ANALYSIS HOW WAS THE PEER GRANT USED FOR POLICY-MAKING? While the quantitative analysis did not demonstrate any statistical difference in terms of the quantity of policy outputs between grantees and non- grantees, the qualitative interviews with PIs, institution leadership, policy makers, and USAID staff provided rich insight into the meaningful policy-making experience of PEER grantees. The key informants highlighted a number of factors that they felt increased the likelihood of affecting program and policy impact at the levels of the PI, institution, and country. For example, several PIs in Colombia, Afghanistan and Brazil (in a follow up phone call), seeking to influence local practices reported holding evidence-based skills-building workshops that produced tangible results in shifting practices. In Ghana, a PI was working alongside farmers to find solutions to parasites damaging or destroying maize crops. These solutions were designed to increase maize production, improve farming practices, and open up potential new export markets around the region, thereby increasing farmers’ income. Though the PI was still in the process of generating results, he reported that his evidence-based collaboration with farmers was already improving the crop yield. This would help improve issues around food scarcity by increasing the nutritional value of maize, and also offer future economic and policy impact. PIs who saw themselves as in a position to apply their research to issues facing their society frequently reported generating not only journal publications, but also tools and other research products for 29 | PEER EX-POST EVALUATION USAID.GOV communities and governments. About half of the PIs we spoke with expressed their desire to produce such tools and research products. A PI in Morocco wondered what she could do to address the question of high unemployment rates amongst youth. She then developed a study to analyze the skills that job requirements desired versus those that were being cultivated in university courses. Her contacts within the government have allowed her to successfully interact and present to several government officials on data demonstrating clear paths to addressing unemployment. In South Africa a similar experience is shared as ongoing relationships between the PI and a national park official helped to implement some research finding in park management in the Kruger National Park. Similarly, in Uzbekistan, a PI networked with government officials and was eventually invited into meetings to present findings on improved irrigation efficiency. Per his recommendations to policymakers, Uzbekistan is now subsidizing farmers to adopt drip-irrigation, which benefits the farmer and the environment with lower carbon emissions and water usage. Multiple PIs noted that a strong professional and institutional relationship with government actors, in addition to administrational transitions of officials impacted their ability to influence programmatic and policy change. Those who were invited onto commissions or into meetings felt more confident that their findings would have impact— either currently or in the future. PIs sometimes see themselves as creators of innovative solutions to problems that hinder sustainable development progress in their countries. Yet, they felt that if they are not able to be in direct contact consistently with government officials, then their work risks having little impact on policy. Thus, networking or collaborative opportunities facilitated a greater chance of having measurable impact. The PEER grants which have facilitated the generation of new knowledge are often connecting multiple demographics and categories of stakeholders. In Colombia, a PI who is studying disaster humanitarian response is working with both local communities to find creative solutions while collaborating on a national level to find solutions alongside NGOs and government officials. We observe that, at their best, the PEER grants encourage PIs to facilitate interaction and information exchange between researchers, local communities, and national-level strategic policy-makers. WHAT WERE THE CHALLENGES? Respondents readily pointed out factors that lower the potential for research-finding-based policy impacts. For one, focusing PEER calls on sensitive or multinational topics might reduce the prospect of getting the attention of government officials, much less ensuring the application of findings in programming and policy. For example, one research user pointed out that a transboundary water PEER project in Afghanistan was not likely to contribute to policy impact due to ongoing transboundary water tensions with Pakistan that the government officials did not feel comfortable discussing. Additionally, some PIs in Afghanistan and Colombia reported that their government needed a strong infrastructure, a platform that facilitates research and data sharing. Their governments also require capacity for receiving, analyzing and acting upon PI research. For example, multiple interviewees working in Afghanistan commented that the national government had little infrastructure and experience processing data to inform policy. Thus, PEER data that was generated specifically for policy-makers was not often utilized. For example, one PI is working with Afghani women to improve water management solutions, while also generating new data to advise the government. She has not succeeded in changing water resource policies due to the sensitive geopolitics between USAID.GOV PEER EX-POST EVALUATION | 30 Afghanistan and Pakistan. Furthermore, the government did not know how to handle the data she produced. While she is creating opportunities for scientific research for women in Afghanistan, there were few outlets to create tangible impacts. This is also related to culture in the country as one PI mentioned the government does not trust women and their capacity of doing research and producing results. They don’t get much support from the government. Similarly, there were also time constraints. A South African PI stated there were limitations in what they were able to complete in certain time frames due to misalignment with the academic cycle. Another South African PI noted that the reporting process was different between the two different organizations that gave him the grant money, doubling his time spent creating reports. In Colombia, a PI spent time translating documents into Spanish for their university administration in the grants office. At an institutional level, university staff and faculty noted two factors that generated programmatic impact were the existence of support staff assisting the PI and an institution with capacity to support the PI in each step of research, from proposal submission to grant and financial management, research implementation, and sharing of results. PIs reported being overwhelmed by managing the logistics of balancing their course load with research activities. Certain institutions that had some level of supporting staff to assist in managing the logistics allowed PIs to focus more on their actual research. Additionally, host institutions that had strong existing relationships with the government, NGO sector, and other universities opened the door to increased and improved knowledge sharing opportunities for the PIs. USAID officials indicated that they want to see the PEER projects more directly contribute to USAID’s country development cooperation strategy (CDCS). A USAID official in Colombia commented that when researchers use grants to advance specific scientific research, this becomes challenging to align with existing development goals and implementation. If there was better coordination between USAID local missions and prospective and current PIs, this might allow for synergizing policy and practice impact. If the research findings carried out established development strategies, they would have higher chances of being integrated into USAID programming.5 C. EVALUATION QUESTION 2: TO WHAT EXTENT AND HOW HAS PEER CONTRIBUTED TO PROGRAM AND POLICY IMPACTS? The results of the analysis do not provide evidence of a statistically significant impact of the PEER program on program and policy changes in the quantitative data. However, we find evidence that these results were achieved at some level in the qualitative data. We present both findings in this section. We examined whether PEER researchers are sharing the outputs of their research more frequently than their non-funded counterparts. We totaled the number of trips taken to present research (See Q24 in Annex B), the number of online posts regarding their research, and the number of academics or policy￾makers with whom the research was shared (see Q26 in Annex B). We created several sets of indices 5 Although we heard this comment only from a USAID official in Colombia, we find this feedback to be particularly pertinent to PEER’s objective of linking research with development. 31 | PEER EX-POST EVALUATION USAID.GOV to assess self-reported utility and efficacy of research to learn if PEER researchers hold different opinions on these topics than their non-PEER counterparts. In the survey, we inquired about various ways that their research is used—for example, to give guidance to communities or to bring more grants for research (see Q27 in Annex B). Respondents could select multiple uses of their research from the list. We also used this question to construct a policy impact score which coded policy-related impact of all applicants’ research according to the PEER program’s definitions of low, medium, and high influence, defined by the program’s Performance Indicator Reference Sheets. 6 Another set of questions inquired if research has contributed to policy changes at various levels—from national to village, private corporations, or multilateral donors (see Q28 in Annex B). Respondents could answer each question on a Likert scale (from “no impact or don’t know” to “impacted a lot”). For both sets of questions, we found a strong correlation between responses (ɑ>.8), meaning they could be averaged together to create indices of perceived research efficacy and perceived policy impact. Our quantitative analysis does not provide consistent evidence of an impact of the PEER program consistently on indicators of program and policy impact, across the two matching models (see Table 9). For example, we find that PEER researchers took more trips to present their research on average, when compared with non-PEER researchers without using a matching method. However, when data are analyzed using the matching methods, we observe a different direction and magnitude of this effect, indicating that the raw data was biased on this indicator. In addition, a lack of consistent effect across the two matching methods means we cannot conclude that the PEER program had an impact on this indicator. For other indicators, including counts of times research was shared online or with an academic audience, perceived policy impact of research, and the USAID policy score, we cannot distinguish the effect of the PEER program from zero using any of the models employed in this study. There are a few indicators where we see potential promise of an effect. Notably, there is a small but significant effect when analysis was conducted using raw data in the case of perceived efficacy of research. Researchers are 7% more likely to indicate that they believe their research has efficacy. When limiting the data to the samples created by either matching method, this effect is sustained in terms of the same direction and magnitude; however, it is no longer statistically significant. This could indicate a potential effect of the program on researcher self-perceived efficacy. In a somewhat similar situation, we observe that PEER PI’s were potentially more likely to share their research with policy-makers--this coefficient is consistently large in magnitude, and positive in sign. However, we only observe a statistically significant coefficient with one model, and this effect is only significant at the 10% level. 6 From the PEER Program’s PIRS: High Impact refers to those PPC's whose combined influence and implementation scores reach a score of 6 or higher Low/Small (1 point) = Local gov't, a few villages, individual (smaller) NGO/CSO's, universities, Medium (2 points) = Provincial gov’t, or major urban centers (1,000,000+), combinations of the same change over multiple individual orgs, or multiple districts across a region, USAID Mission program High (3 points) = National gov't policy or core business of multinational Institution or corporation USAID.GOV PEER EX-POST EVALUATION | 32 Consequently, we cannot conclude with confidence that the PEER program had an effect on policy￾maker audience size. We investigated the relationship between the Mission Relevance score given to applications, and the policy-relevant outcomes listed here. We observe a positive and statistically significant relationship between the score and the USAID policy impact score. PEER applications with higher scores of Mission Relevance are more likely to have high policy impact (p<.05), but we observe no significant impact when this score is interacted with treatment status (not shown). Note: *** p<0.01, ** p<0.05, * p<0.1 Source: Survey data. Controls included region, years since application, reviewer score, gender of applicant, field of research, and cohort fixed effects In the qualitative data collection, PIs reported multiple types of programmatic and policy impacts, ranging from local level practices to curriculum changes, or from opportunities for student learning to involving national policy-makers in their research. Depending on which level that the PI is focused on, interviewees shared a number of success stories that demonstrate the wide perceived impact influenced by PEER grant products. PEER grants have supported young professionals in cultivating experiences that enriched their careers and allowed them to answer questions that are relevant to development and TABLE 9: PROGRAM AND POLICY IMPACT, SURVEY DATA VARIABLE COEFFICIENT: RAW DATA N COEFFICIENT: EB N COEFFICIENT: MDM N Traveled to present research in last 2 years (count) 2.41* 293 -0.72 176 -6.47** 74 Count of times share research output in last 2 years: Online (count) 0.75 301 -0.75 183 -4.68 74 With academic audience (count) -34.32 300 -11.88 184 55.93 75 With policy audience (count) 32.13 281 52.58 176 173.05* 75 Index of perceived efficacy of research 0.07** 352 0.06 176 0.07 75 Index of perceived policy impact of research -0.04 346 0.02 214 -0.11 90 USAID policy score -0.03 346 0.16 210 -0.01 90 High USAID policy score (%) -0.04 346 -0.02 210 -0.03 90 33 | PEER EX-POST EVALUATION USAID.GOV policy officials. Collaborations between universities, NGOs, and governments allowed for proactive applied research and solutions to affect programming and policy challenges. The following illustrative examples represent different types of programmatic and policy impacts. RELEVANCE OF THE RESEARCH OUTPUTS AND LINKS TO DECISION-MAKING Research outputs were used for programmatic or policy change when there was a perceived need of evidence informed by research. In Armenia, the themes of nuclear energy and radiation issues are pressing to the national agenda. One PI stated that he represented the sole institute within the country with the capacity to assess radiation issues. Thus, he is consistently providing policy briefs and maintains regular contact with government officials to support revising policies and examining best practices. He has produced four peer-reviewed publications, improved his international recognition, and forged networks within the government that are allowing him to continue his research. With the equipment that was purchased through PEER funding, he will continue to advance his research and produce results that will impact policy for years to come. USE OF PEER OUTPUTS AMONG STAKEHOLDERS There is evidence that the PEER research outputs are being used by various stakeholders, including: government, NGOs, and individual researchers. For example, PEER research output is being used by South African park authorities to manage a national park. In Colombia, one PI worked alongside firefighters to better shape fire response policy. Her work caught the attention of Brazilian counterparts who were asking similar questions. This connection provided opportunities for future local and international collaborations to address fire management and practices. Within Colombia, the research finding provided feedback to the government to bring a bill to set up a national advisory commission on the issue of forest fires and fire management. Furthermore, in the Philippines, a PI attended Philippine Congressional meetings twice concerning the coastal wetland and coastal Greenbelt law. Some aspects of the law were informed by PEER project outcomes. Similarly, a South African PI, whose research looked at community and regional food system promotion, worked alongside the Department of Rural Development and Agrarian Reform towards creating farmer markets that could be scaled to other regions within the country. He worked alongside local and regional officials to create markets that fostered local economic stimulation for farmers. An example of how effective collaboration can be between government officials and PIs perhaps is best displayed in a case in South Africa. The PI had a relationship with a government official prior to applying to the PEER grant. The two shared an understanding of the need to address fire issues facing a national park. The PI applied for the PEER grant using a question co-created by her and the government official. As a result, the research was built from the ground-up with government use in mind. This collaboration led to research products that included multiple trainings for staff and policy development for better fire management. The official commented that their office’s lack of ability to collect and analyze data was overcome by the PI and her students. The PI allowed them to create data-driven policies that would not have existed otherwise. D. EVALUATION QUESTION 3: WHAT KIND OF EVIDENCE IS MOST LIKELY TO INFLUENCE POLICY-MAKERS? USAID.GOV PEER EX-POST EVALUATION | 34 PERCEPTIONS OF PEER BY USAID MISSION PARTNERS Overall, the interviews with USAID Mission officials in some cases revealed little interaction between the Mission and PIs beyond addressing simple logistical concerns. In some interviews, Mission officials knew very little about PEER and its goals, suggesting that PEER was a program that corresponded with USAID headquarters rather than the individual missions. Two USAID officials from separate countries indicated that they would be interested in supporting the dissemination of PEER research findings to government policy makers. They offered ways to support presenting information or advocating to government officials on behalf of PIs’ research. Greater interaction between PIs and USAID missions could increase awareness of the specific projects, which could help alleviate the challenges of sharing data and policy suggestions with government officials for PIs. Additionally, if communication between the missions and PIs are bolstered, there may exist a way to align PIs’ research relevance with USAID Mission development goals. This was also reflected in the interviews with PIs, many of whom stated that they had very little interaction with USAID, other than to obtain information around reporting requirements or with acquiring equipment. While PIs overwhelmingly reported positive interactions, they were minimal. INTERACTION WITH GOVERNMENT AND POLICY-MAKERS According to a government official, the most influential presentations of evidence to policy-makers are short memos that use accessible terminology.7 An official in Uzbekistan also suggested scientists write precise and practical recommendations which can be disseminated easily. A government official from Uzbekistan suggested: “Materials design in the form of presentations could be very useful. Also modelling and designing possible scenarios would be effective. For example, [a] model of water management. Of course, the models themselves cannot be applied into practice, as they should be first localized and adjusted to the requirements and conditions of certain fields.” Overall, officials requested that only well-vetted data be provided to them, meaning that the research has undergone rigorous testing using a complete methodology. While government officials did not comment specifically on conferences, PIs and university officials said that they held multiple workshops and conferences where government officials were invited to attend. While there were mixed results reported from these workshops, PIs felt that they at least allowed them to present their findings to an audience, network, and build relationships with policy-makers. One Colombian PI reported that after giving a workshop to a local community, he was invited into a longer￾7 An exception we observed was that some government officials in South Africa were also scientists/experts in their respective fields. These policy-makers preferred journal articles. 35 | PEER EX-POST EVALUATION USAID.GOV term relationship with local organizations and municipalities. He then contributed to new development plans in the region. In one case in South Africa, a PI reported that she was able to successfully offer data that aligned with government interests and has subsequently been involved in shaping water management policy and programming. This PI was able to develop partnerships within South Africa’s government water research commission. As a result, two publications have been produced, with an additional three forthcoming. The project has attracted the attention of the Danish private sector and other local universities to create solutions with the commission for groundwater management. The consistent communication around both the problems and solutions contributed to creating a transnational, collaborative response that will have future positive outcomes. While there was consensus between PIs, USAID officials, and government officials that the need to share relevant data to pressing policy issues is a first priority, there was discord between the best methods of data sharing. Various PIs commented how they are challenged to translate their data into language and formats that are approachable to government officials. As a result, several PIs suggested that they would find it helpful to receive training on how to create policy briefs or other documents that are easily understood by government officials. RELEVANCE FOR COUNTRY STRATEGIES AND MISSION NEEDS A couple of USAID officials stated that one of their primary research-related challenges is formulating the correct research question that will obtain necessary and timely data to address their priorities. For example, in Colombia, the quality of existing data around migrant demographics is low. Thus, their efforts to support migrant children is limited in absence of data. Instead, the USAID official said: “I want to know what's happening now, but really, the answer to these questions... you need a couple of years to look back, and we just don’t have that luxury right now.” We find that by having clear questions that provide guidelines for strong data acquisition, there is a foundation for stronger overall capacity-related and policy-making outcomes. A couple of Mission staff suggested that if the PEER PIs involve the USAID Mission from an earlier stage in the process, potentially during research design, there will be greater impact on programming and policy making. However, there might be staff bandwidth issues as it may add additional burden to Mission staff. CHALLENGES OF USING OUTPUTS AT MISSION LEVEL As of now, there are few mechanisms that exist within the missions we interviewed that actively include PEER outputs in working toward their CDCSs. The perception that PEER is operated out of Washington D.C. headquarters is a common sentiment among Mission staff. If current and future iterations of PEER are seeking to influence USAID policy and practice in a given country, a major challenge would be to bolster awareness and ownership of the PEER grant within the mission. A mechanism would need to be established within, both in the application and implementation phases of research that considers— alongside a PI’s research capacity and the merits of their research—the CDCS information needs within the PI’s home country. USAID.GOV PEER EX-POST EVALUATION | 36 In addition, there is no consolidated directory or database accessible to the public or USAID staff that has in-depth descriptions of PEER research projects in each country. There is a consolidated directory for the public and USAID staff, but PEER researchers were not aware of it. Thus, some PIs noted that there could be a risk that similar research questions have been asked, either within that country or in a similar context, which could mean that a PI is repeating research. Additionally, a database would allow government or Mission officials access to research that informs policy and programming. E. EVALUATION QUESTION 4: TO WHAT EXTENT AND HOW HAS PEER CONTRIBUTED TO CAPACITY-RELATED OUTCOMES? CAPACITY-RELATED OUTCOMES ● CAREER PROGRESS ● STUDENT CAPACITY ● LEVERAGING FUNDING ● SELF-PERCEIVED RESEARCHER CAPACITY ● TRAVEL FOR TRAINING OR EDUCATION The results for capacity related outcomes are presented in Table 10. In the survey, we asked a series of questions on the career progress of the researchers— if they had received a promotion, increase in salary, or taken on an additional paid or advisory role (see Q18 and 19 in Annex B). We created a summary variable coded as one if they answered yes to any of these, and zero if otherwise. For this variable we see that in the raw data and the EB weighted data, PEER PIs are 11-12% more likely to have progressed in their career than non-funded counterparts. The effect is slightly smaller in magnitude and not statistically significant when using the MDM method, however it maintains the positive sign. This somewhat consistent finding demonstrates a positive impact of the program on career progress of the PIs. Next there was a series of questions on self-perceived capacity as a researcher. Researchers reported if they’d improved in ability to perform various aspects of their job—from collecting data, to communicating results, teaching or accessing needed infrastructure (see Q51 and 52 in Annex B). The respondent could answer on a 5-point Likert scale (from strongly disagree to strongly agree). Again, these scores were found to fit well together (ɑ>.8), so a Perceived Capacity Development Index was created. We do not find an effect of the PEER program on this indicator. Next, we asked about travel related to capacity development for researchers and students. This included travel to attend a conference, take a training, or pursue a degree (see Q24 in Annex B). We totaled up the number of trips for PIs and for their students. We do not find an effect of the PEER program on researcher or student travel for capacity development. We also asked a series of questions about student capacity—receiving funding, working in their professor’s area of research or working in international development more broadly (see Q25 in Annex B). We inquired about the number of students supported (from the level of undergraduate to doctoral) since applying to PEER, to measure if PEER attracted more students to the researchers’ classes (see Q34 in Annex B). We do not find a consistent effect of the PEER programming on these indicators. We observe a potentially negative relationship between PEER funding and students’ careers in PI’s area of 37 | PEER EX-POST EVALUATION USAID.GOV research after graduation. However, because this effect is not consistently significant across both matching methods, we cannot be sure if this is a true effect. Lastly, we looked at other grants received since applying to PEER (see Q8 in Annex B). It was hypothesized that PEER researchers may use the grant as leverage to pursue other funding sources, so we aimed to assess if PEER researchers received more grants than the comparison group.8 We do not detect a positive relationship between PEER funding and receiving additional grants. In fact, there are indications of a negative relationship between PEER funding and receiving additional funding; however, this relationship is not consistent across both matching methods. Consequently, we cannot say with confidence that there is a true negative effect. 8 We constructed this variable in two ways--first, we did not include the PEER grant for the treatment group. Next we included the PEER grant as one of the grants received. In the first model, PEER was correlated with receiving 2-3 fewer grants (p<.001) when analyzed using the matching techniques. But in the second case, there was no difference between the PEER and non-PEER group in terms of grants received. TABLE 10: CAPACITY AND LEVERAGED FUNDING VARIABLE COEFFICIENT: RAW DATA N COEFFICIENT: EB N COEFFICIENT: MDM N Career progress since application 0.11** 557 0.12** 318 0.07 125 Perceived capacity index -0.06 320 0.03 192 -0.18 82 Count of trips taken in last 2 years to attend a conference, take a training or pursue degree Researcher 0.09 278 0.45 168 -0.11 70 Students 5.91 223 -0.98 143 -7.13 60 Any students have received funding for their research since application -0.02 357 -0.08 216 -0.17 90 Many or all students work in PI’s area of research since graduation -0.05 329 -0.06 202 -0.27** 84 Many or all students work in international development since graduation -0.05 309 -0.01 191 -0.13 79 Count of students supported since applying to PEER -17.18* 338 -16.88 205 -14.32 89 USAID.GOV PEER EX-POST EVALUATION | 38 Note: *** p<0.01, ** p<0.05, * p<0.1 Source: Survey data. Controls included region, years since application, reviewer score, gender of applicant, field of research, and cohort fixed effects In our in-depth interviews, however, PIs consistently reported success in supporting their students in both gaining research experience and obtaining degrees. In a number of cases, this extended towards students applying for and being accepted into fully funded doctoral degree programs. Multiple PIs reported that their students won internships, gained funding towards their tuition, received valuable research experience, and assisted in writing the journal publications. A masters student from Colombia stated: “I was enrolled in the project as a Master student. Through the project I financed my Master’s degree, and I had the opportunity to travel to the United States with the USAID department. The project was really important in my research career, because I could learn how to investigate, research the project. Also, I enrolled in the doctoral program, because with these projects I was very interested in the research and in the humanitarian logistics field.” In cases such as this one, students went on to postgraduate programs as a result of their involvement with PEER-funded research. Additionally, three PIs stated that their students were studying toward a Ph.D. at the institution of their U.S.-based research partner. For the students, the experience of having worked on a PEER grant increased their competitiveness for fully-funded scholarship programs to continue educational pursuits in the field. Furthermore, PEER funded students were hired quicker than their non-PEER funded students. A South African PI noted that PEER funded students are more experienced in research. Other PIs mentioned finding post graduate opportunities, including Google fellowships and postdoc opportunities in the U.S. Multiple PIs that considered themselves early in their careers reported that they felt confident that the PEER grant experience would open up their career to new opportunities and recognition. One PI in Lebanon mentioned that she had gained recognition in her own institution, which facilitated access to more opportunities within her department. Additionally, she and her co-PI in the U.S. continued applying for funding successfully in other large grants to continue their research. Overall, through the experiences of obtaining and implementing a competitive grant, PIs widely reported that there were new possibilities presented to them. The vast majority of PIs interviewed were able to either publish and/or present in conferences from local to international levels. In addition, PIs from many countries—notably Lebanon, Armenia, Ghana and the Philippines—reported that their data would continue to serve as a basis for future publications, research, and potential policy and programmatic suggestions. A number of PIs stated that they anticipated further programmatic or policy changes to come with subsequent data and stronger reports. PIs attributed this to both having access to the tools and necessary equipment to complete the research, which either was not available in their country or was too expensive before receiving the PEER grant. Count of grants received since applying to PEER (including PEER grant) 0.42 467 -0.85 270 -2.05** 111 39 | PEER EX-POST EVALUATION USAID.GOV F. EVALUATION QUESTION 5: WHAT FACTORS (AT THE LEVEL OF PI AND INSTITUTION) INCREASE THE LIKELIHOOD OF ACHIEVING CAPACITY-RELATED OUTCOMES? QUANTITATIVE MATCHING STUDY RESULTS Since the only indicator we found to be significant for research question 4 was career advancement, we investigated if there is any regional variation in response to this question. We do not have statistical power to detect differences by region, but we present Figure 6 below to demonstrate that career progress is potentially driven by differences in Asia/Pacific region and Latin America. FIGURE 6: CAREER PROGRESS OF PIS, OVER TREATMENT STATUS QUALTRICS SURVEY RESULTS In the survey, we asked PEER researchers how PEER has helped them in their capacity development (Q29 in Annex 2). As these questions are not relevant to the comparison group, we present summary statistics in Figure 7. For most categories, responses were quite positive, with 80% or more indicating that PEER helped them to leverage new funding, advance their career, achieve recognition as a leading scientist, gain recognition from their colleagues, and communicate with decision-makers. In particular, career advancement is a strong impact which we see both in the quasi-experimental data analysis, and in the self-reported impact of PEER presented here. One potential driver of this is recognition from colleagues as a leading scientist. Other impacts reported by PEER researchers, such as leveraged funding or increased communication with decision-makers, are not backed up by quasi-experimental analysis. USAID.GOV PEER EX-POST EVALUATION | 40 Nevertheless, it is important to note that according to PEER researchers themselves, the program did have an impact in these areas. One response category was selected with notably less frequency— “attracting students to my classes.” For this category, only about 50% of respondents answered positively. This is consistent with the qualitative findings, where researchers spoke at length about the value of the grant to support a few of their students deeply. Typically, PEER researchers worked with one to three students very closely; these students were often further in their studies (masters or doctoral level). But the researchers do not indicate that they were able to have larger classes or work with large cohorts of students through the PEER grant. Class size may not be a relevant indicator for the PEER program; we discuss this further in the Recommendations section. FIGURE 7: PEER PIS’ PERCEPTION OF THE BENEFITS OF THE PROGRAM QUALITATIVE RESULTS The qualitative data provided insight around factors related to capacity outcomes at different levels. At a government level, some officials and PIs agreed that governments need a strong infrastructure for processing data. For example, in Afghanistan, interviewees from various stakeholder groups noted that data production and analysis was a new process in the country. Thus, officials and PIs were still trying to find effective ways of communication, collaboration, and presenting data. In countries like South Africa or Colombia, where the infrastructure exists to collect, analyze, and act upon data, the likelihood for uptake was much higher. Within the institution, there are multiple factors that contribute towards capacity-related outcomes. For example, universities that have programs for masters and doctoral students allow them to spend more time consistently in one research project and provide greater technical knowledge to support PIs. In 41 | PEER EX-POST EVALUATION USAID.GOV addition, universities that contain thematic institutes within their academic structure can provide additional funding to students that coalesce with their PEER experience. Thereby helping students focus on their degrees, rather than having to work part-time to fund their degree. At the level of the PIs themselves, there are also multiple opportunities to capitalize in capacity-building. The relationship with the U.S.-collaborator matters greatly to not only creating high-quality final products to present findings, but also to ensure that the research process is distributed and engages multiple perspectives to analyze and troubleshoot issues. Those that expressed close relationships and trust with their U.S. collaborator often showed examples of internships and exchanges between their students, commitment to publishing results together, and presenting at conferences. In terms of student involvement, PIs spoke highly of the value of the grant for the capacity building of the few students with whom they worked closely. For example, one Colombian PI stated that in order to obtain a masters degree, students normally struggle to finish within a reasonable amount of time because they struggle to finance their education. Yet, through the PEER funding, a few students were able to continue studying full-time to obtain their degrees without having to take on a second job. One student was able to successfully apply to a Ph.D. program due to the funding and research experience they obtained. Other PIs noted that due to certain constraints, their collaborations with their U.S.-based counterparts were not as fruitful as they could have been. Some PIs wished that there had been an easier avenue towards either obtaining a U.S. visa or ensuring funding for their U.S. co-PI to allow for travel to their country to promote in-person collaboration and build deeper rapport. PIs consistently said that having the correct tools and equipment allowed them to fully engage in their research. In Ghana, the PI reported that a piece of equipment was broken that he could not acquire within the country. The U.S. Embassy helped find a replacement and shipped it to Accra, where he was able to pick it up and continue with his research. PIs from Lebanon, Armenia, Kenya, Brazil, Morocco, and South Africa said that the ability to procure the correct equipment facilitated their ability to research, supported students in their technical training, and improved data acquisition. Finally, there is a gender aspect of achieving capacity-related outcomes. Women might be deterred from applying for PEER grants that have timelines for research that do not allow for maternity leave. If the grant process had flexibility for women who might become mothers, more women could apply. Some PIs suggest that the research field itself in many developing countries is dominated by male researchers, who are always in a better position to get grants from external sources. To mitigate this issue, some PIs suggest providing opportunities for female researchers, including: providing training on grant writing, research management, and reporting. G. EVALUATION QUESTION 6: ARE THERE REGIONAL DIFFERENCES OF MOTIVATIONAL FACTORS FOR INVOLVEMENT IN PEER FUNDED ACTIVITIES? IN THE QUANTITATIVE DATA, WHICH REGION HAS HIGHER INVOLVEMENT OF RESEARCHERS MOTIVATED BY SOLVING SPECIFIC DEVELOPMENT PROBLEMS? WHICH REGION HAS HIGHER INVOLVEMENT OF RESEARCHERS MOTIVATED BY ACADEMIC INTEREST? In the survey, we asked researchers why they pursued the research topic listed in the PEER grant application (see Q12 in Annex B). Respondents could select from many options or write in their own USAID.GOV PEER EX-POST EVALUATION | 42 responses. These were later coded as relating to three larger themes: 1) development problems or policies, 2) academic interests, or 3) career development of self or students. We present mean frequencies of responses to each category among PEER grantees in Figure 8. We do not observe many statistically significant differences by region, but we see some general trends. In particular, Latin America/Caribbean and Asia/Pacific researchers are potentially more motivated by academic interests than researchers from Sub-Saharan Africa (SSA) or Europe/Eurasia. Finally, the third category, which includes career advancement, cross-border collaborations, and student support, is the least frequently identified reason for pursuing the research question across most regions. FIGURE 8: MOTIVATION TO PURSUE RESEARCH QUESTION IN THE QUALITATIVE DATA, WHICH REGION HAS HIGHER INVOLVEMENT OF RESEARCHERS MOTIVATED BY SOLVING SPECIFIC DEVELOPMENT PROBLEMS? WHICH REGION HAS HIGHER INVOLVEMENT OF RESEARCHERS MOTIVATED BY ACADEMIC INTEREST? Consistent with the quantitative data, we found that in African low-income countries, many of the PIs linked their research directly to developmental questions that could have positive economic and climate impacts in their regions. In addition, they reported that their work was a continuation of their doctoral work— indicating that their research questions are associated with development themes that are consistently important in these researchers’ careers. PIs consistently researched issues of food security, crop production, and water management. They used their findings to produce journal articles, but also sought to find tangible ways to share knowledge with local farmers and communities affected by their findings. For example, in Uganda, a PI studying the effects of drought on crops created policy recommendations that have been enacted by the government, alongside creating a phone application 43 | PEER EX-POST EVALUATION USAID.GOV that helps farmers predict patterns that could affect their crops. These types of studies that engaged local and national levels offered both programmatic and policy impact. In the MENA region, researchers equally responded that they were shaping their research questions, according to development and academic questions. In one instance, a PI stated that as a scientist trained in generating and analyzing actionable data, she should base her research off of national questions. Thus, she created a project to address employment issues in Morocco: “And one of the problems they're talking about is that there is a huge skill mismatch in the local [economy] and it causes huge youth unemployment… I'm a data scientist, how can I help?” Other PIs in the region took an approach of pursuing research questions that could eventually align with development questions, but reported that they were not in the position to impact policy within their country. For example, one PI was studying how to expand energy access and transform consumption in rural areas through innovative solar panel solutions. They stated that this should be a development priority for the government. The researcher then acknowledged that the cost was too high for the government to consider widely scaling the innovative technological solutions that were identified through the research. Latin American PIs reported a mix of motivations for engaging in research, with research questions that represented both academic and pressing developmental issues. Four Colombian PIs and one Brazilian PI stated that they were employing research methodologies to solve development issues. However, they noted that their particular research question may not be a top priority within their current government administration. For example, PIs commented on how overfishing or climate change were leading towards issues within local communities, yet their effects were yet to be drastically felt. One Colombian PI stated: “I realize that the climate projections are affecting our hydrology. For example, we also know that we will have droughts seasons more extreme than usual. And since the city just has one source for drinking water, we are vulnerable. But climate change is not a phenomenon which is waiting for a technical solution. Climate change is a phenomenon that is changing the way we see ourselves, our society, our institutions, our children, our educational systems and so on.” Thus, while their research may not be affecting the most pressing developmental policy on a national scale, they are focusing on preventative measures for the future, local community level challenges. In Asia, PIs were studying issues around climate change that have effects on tourism, agriculture, and local fisheries. Similar to the Latin American PIs, they reported that while their research interests were relevant to development problems and policies, the topics were not necessarily representative of government priorities. Instead, they noted that their research generated findings that were useful to local communities directly impacted by climate change. H. CONCLUSIONS AND RECOMMENDATIONS USAID.GOV PEER EX-POST EVALUATION | 44 SUMMARY OF FINDINGS AND HYPOTHESIZED EXPLANATIONS In summary, on the quantitative side, we observe little statistical impact of the PEER program with the notable exception of career progress. We fail to detect a difference between PEER grantees and comparable PEER applicants in the categories of: generation of knowledge (training materials, policy output, or academic research), academic influence, sharing research, perceived efficacy, perceived policy impact, participating in capacity-developing activities, leveraged funding, or career progress for students. The only two impacts we observe are a positive impact on career development—PEER grantees are over 10% more likely to have had a promotion, increase in salary, or taken on an additional paid or advisory role since applying to the grant. And we observe a slightly negative relationship between participating in the PEER program and the production of technology, datasets, or patents. We present hypotheses to explain these findings here. In terms of the negative finding for technical output, PEER researchers may focus on policy or community outreach through the program’s priorities. If this is the case, it is understandable that they have not produced as much technology or patents as researchers who have a purely academic focus to their work. This is a potential trade-off to consider when prioritizing research which applies to policy or development. In terms of career progress, this positive effect could demonstrate that the researcher gained prestige from receiving a US-based grant, partnering with a US-based researcher, or from the ability to purchase infrastructure. For example, several researchers indicated, when interviewed, that the ability to purchase high quality research equipment helped them gain professional recognition. Our observations of career progress could indicate the effects of prestige from partnership with U.S. researchers or funding from the U.S. government. NULL FINDINGS We investigate our null findings further, in terms of anticipated timing of outcomes, statistical power, and potential substitution. First, one potential explanation relates to various aspects of timing. It could be that we are examining these outcomes too early, and that the program will not realize its impact for several more years. The midline report for PEER presents a literature review of the theory of change for policy impact, and posits that policy impact typically takes 10 or more years to come to fruition (Benschoter et al, 2017). In the in-depth interviews, many PEER PIs expressed frustration that the current policy priorities of their governments do not align with their research topics. This could explain our lack of a finding for policy￾related impacts. Regarding academic output, review of the literature indicates that the timeline we are examining (from 4-8 years after receiving a grant) is a reasonable timeline for the completion of academic research. This timeline is one of the reasons we only included PEER cohorts 1-5 cycle in this study, because later cohorts would not have had enough time to realize these impacts. While some researchers from Cycles 4 and 5 have not yet completed their research, it’s likely that researchers in the comparison group (PEER applicants from Cycles 4 and 5) are in a similar position, so this issue would not bias our results. While it appears that the timeline for realizing academic input is reasonable, what about the cut-off which defines what research is “after PEER?” Our original model defines any publication dated after 45 | PEER EX-POST EVALUATION USAID.GOV receiving the PEER grant, even by one month, as the dependent (outcome) variable, and anything published before PEER as a covariate. Given the life cycle of research, it’s unlikely that a publication which appears in a journal a few months after receiving a grant can be attributed to that grant. So as a robustness check, we re-ran our analysis using different cut-offs—defining “after PEER” as 1 year, 2 years, even 3 years after receiving the grant. Our results change in magnitude due to changes in sample size when adjusting these cut-offs, but the results remain the same in terms of a lack of statistical significance. A second explanation relates to statistical power. It could be that there is a true effect of the PEER program, but our study is underpowered to measure it. To examine this further, we calculated the effect size we observe in this study. An effect size is defined as the coefficient divided by the pooled standard deviation of treatment and comparison groups. Standard rules of thumb are that an effect size of .2 is considered small, .5 is considered medium, and .8 is considered large. Effect sizes are important to examine both in terms of understanding statistical power to detect a difference, and policy relevance of an effect. Table 11 presents the effect sizes of our key indicators, and includes both the necessary sample size needed to detect an effect and the sample size we have in this study before performing matching. We see that for some indicators (number of citations or student areas of employment, for example), differences between treatment and comparison groups are so small that they both require an unreasonable sample size to measure, and are likely not policy relevant. For other indicators such as new research in the past two years or perceived research efficacy, however, the effect size is fairly reasonable and our study seems to be underpowered to detect an effect. This indicates that there could be an effect of the program but we do not have sufficient observations to detect the effect. One solution to this could be revisiting the data in a few years, at which point applicants from Cycles 6-9 could also be included in the sample. TABLE 11: EFFECT SIZES OF SELECTED OUTCOMES OUTCOME EFFECT SIZE, EB WEIGHTED DATA SAMPLE SIZE NEEDED TO DETECT AN EFFECT SAMPLE SIZE, THIS STUDY Outcomes from bibliometric data # of publications .196 800 422 # of citations .069 6,666 422 Altmetric score, after publication .147 1,464 422 # of policies .053 11,372 422 # of Mendeley Readers .025 49,594 422 Outcomes from survey data EQ 1: Knowledge Generation New technology in last two years .402 634 208 New training in last two years .151 1,382 208 USAID.GOV PEER EX-POST EVALUATION | 46 Finally, we examine the question of substitution. One reason we do not observe an effect on publication output is that researchers who are rejected from PEER simply receive other grants. This could be the case since we observe no difference between PEER and non-PEER researchers in terms of number of grants received. We see that 13% of non-PEER researchers applied for and received funding for the exact same research project. 71% received at least one other grant since applying to PEER. 56% have collaborated with their US-based PI since applying to PEER (not shown). This evidence points to the possibility that comparison researchers are simply acquiring funding elsewhere and realizing the same outcomes as the PEER researchers. Jacob and Lefgren (2007) similarly note that the market for scientific funding is competitive, so the loss of one grant simply causes a shift to another. This could be one explanation for why we do not observe a difference in the number of research products created. In addition, with the exception of publication data, we do not have an indicator which measures the quality of research outputs (training materials or policy briefs for example). PEER and non-PEER researchers could have produced the same number of these outputs, but not the same quality. In some literature evaluating research funding programs, equilibrium models were presented for the market of research grants. Jacob and Lefgren (2007) created an equilibrium model examining if NIH grants displaced other funding sources. Other literature examines the elasticity of scientists and they found that the cost of convincing a scientist to engage in new research topics is quite high (Myers 2018). In the future, a market analysis could help provide an understanding of the dynamics at play—is there a surplus of grant funding in PEER research countries or sectors? Or is there a surplus of qualified researchers who cannot get funding for their projects? What is the elasticity of demand for research funding or the supply of scientists? Such supply and demand, or private goods frameworks may or may not be relevant to the full Theory of Change of the PEER program, but these frameworks have been presented in some of the literature considering the impact of grant-making programs which was New policy in last two years .149 1,414 208 New research in last two years .210 712 208 EQ 2: Program and Policy Change Index of perceived efficacy of research .231 596 214 Index of perceived policy impact of research .180 976 210 Many or all students work in PI’s area of research since graduation -.037 22,386 202 EQ 4-5: Capacity Building Career Progress .249 438 318 Many or all students work in international development since graduation .055 8,064 191 Count of grants received since applying to PEER (including PEER grant) .245 524 270 47 | PEER EX-POST EVALUATION USAID.GOV reviewed in preparation for this research. It could be relevant to review such literature in considering how the PEER program may be interacting with the market for researchers and research products. INTERVIEW FINDINGS The PEER grant offers resources to scientists that enable them to focus on their research, which in some cases simultaneously addresses development questions. PIs reported overwhelming support and gratitude for the PEER mechanism, suggesting that it provided them with a wide array of opportunities that might not otherwise have been available to them. PIs reported that they advanced their career by having a prestigious grant on their resume, publishing related journal articles, and having dedicated resources for their research. Also, the tools and equipment purchased by the PEER grant would facilitate future research or expansion upon the PIs’ original PEER questions. Additionally, PEER created opportunities to cultivate partnerships with governments, US-based researchers, and other development actors in order to produce tangible results towards advancing programming and policy. Many PIs suggested that they were able to provide specific support and opportunities to their students to advance their careers. This included co-authoring publications, presenting at international conferences, and gaining valuable experience in carrying out research and analyzing raw data. A select number of students continued to pursue their PhDs, either at the same university or at a university in the United States. Responses identified gaps in the PEER grant’s efficacy. The responses suggest that the program is not reaching its full potential in terms of affecting programming or policy change. For example, the openness of the grant for any researcher creates challenges in finding ways to limit a wide range of research to finding solutions for development issues. In addition, PIs are not trained in presenting information to a policy-minded audience. Thus, finding effective ways to communicate with government officials has been challenging. Furthermore, the volatility of government administration turnovers or overall interest displayed by the government towards PIs’ research can be cumbersome. Interviews with two of the four USAID Missions suggested that there was not much awareness of the PEER grant or PIs’ work, primarily due to staff turnover. Similarly, one official indicated that there is a lack of well-maintained communication between PEER and Mission staff. One USAID Mission official commented that better alignment of research proposals and development goals could create more opportunities for collaboration. This entails support from USAID Mission in either translating data into briefs suitable for a policy-making audience, advocating on behalf of the PIs to the government, or increasing data-analysis infrastructure within the government. RECOMMENDATIONS The PEER program has an ambitious theory of change which posited outcomes on PI’s academic outputs and influence, policy impact, and impact on the career progress of both the PIs and their students. Through the analysis of survey and bibliometric data, we observed that the PEER grant had an impact on career progress, but we do not observe the other posited impacts in this evaluation. One of the key insights of the evaluation is that while the PEER Stakeholder Summary acknowledges the importance of the interplay of demand-side and research-supporting stakeholders with the PIs. The PEER operational USAID.GOV PEER EX-POST EVALUATION | 48 model focuses squarely on interventions targeting the supply-side stakeholders. In the absence of resources and activities specifically targeting the other key stakeholders in the equation, the assumptions connecting the levels of the PEER theory of change may remain unmet in many contexts. Along with this observation, our evaluation revealed other key insights for recommendations that could support the successful manifestation of the theory of change. We have grouped these recommendations by the evaluation questions with which they are associated. For reference, an abbreviated version of the evaluation questions can be found below in Table 12. EQ 1 AND 2. PROGRAM AND POLICY IMPACT Some academic researchers are experiencing an ongoing disconnect with the policy environment. While some PIs are able to influence policy action, others mentioned that government officials and other policy-makers are unable or unwilling to heed the findings of academic research, so there is little chance of PEER research to impact policy. While government culture cannot be changed overnight, some PIs and USAID officials offered suggestions. 1. The USAID Mission could assume a role of supporting PIs in translating their research results to form policy suggestions that would have a greater chance of uptake. This could include providing training on how to translate scientific data into policy briefs and other policy￾relevant formats, providing feedback to PIs on policy products, and advocating policy suggestions on behalf of the PI. Further, PIs face constraints in transforming the results of their research into academic publications. This could be due to the focus on teaching or getting funding at their institutions. USAID could provide support to the PIs in the form of workshops or paid time for producing both academic or policy-related research products. 2. USAID, PIs and government representatives commented on how the priorities of these actors may not align. Furthermore, USAID staff commented that their development strategies tend to be broad, allowing the mission to seek to address numerous challenges within a given country. While researching a specific question in this context, PIs may reach conclusions that are too narrow and specific to apply to any one development goal or implementation strategy. Suggestions for ameliorating this type of situation included either soliciting transparent input within each country on policy priorities or otherwise involving government and/or USAID staff TABLE 12: EVALUATION QUESTIONS 1. What factors increase the likelihood of achieving program and policy impacts? How has the PEER grant helped to generate new knowledge and be utilized for policy-making? 2. To what extent and how has PEER contributed to program and policy impacts? 3. What kind of evidence is most likely to influence policy-makers? 4. To what extent and how has PEER contributed to capacity-related outcomes? 5. What factors increase the likelihood of achieving capacity-related outcomes? 6. Are there regional differences of motivational factors for involvement in PEER funded activities? 49 | PEER EX-POST EVALUATION USAID.GOV at the proposal stage and beyond so that PIs could do applied research within their field that directly supports and aligns with development or policy priorities. This might mean that a USAID or government collaborator is included in the PI’s proposal from the beginning, to ensure that there is overlap between relevant data and the development goals within the country for effective policy and programming. This might also entail including supplemental country-specific policy information to prospective applicants within that given country to complement the general call for applications. 3. One challenge of utilizing PEER research findings in USAID programming is when PEER research areas do not match with USAID’s sectoral priority for the country. A USAID official commented that they “needed data yesterday for today’s problems.” USAID staff felt that research utility would increase if the PEER call for proposals were based on USAID’s CDCS for the country. 4. PIs and government officials were concerned about sustaining their research after the close of the grant. Many PIs desired to not only maintain the PEER research efforts, but also expound upon initial studies as new research questions and avenues came to light. To mitigate this concern, USAID mission staff can work with host governments to explore avenues to leverage government research dollars for PIs to sustain or build upon their research post-PEER if the research findings achieve policy relevance. This can act as a double incentive by incentivizing PIs who do policy-relevant research with future funding while also ensuring that the policy￾makers themselves are invested in the research outcomes. EQ 3. EVIDENCE INFLUENCING POLICY-MAKERS 1. Many PIs and government officials agreed that the infrastructure for analyzing data and creating new policies does not exist within some contexts. One potential solution would be to invest resources and implement activities that support governments in their capacity to use data to create effective evidence-based policy. This could include inviting technical specialists from different ministries within the government to capacity-building workshops led by PIs, through providing direct technical or material support to government policy-making agencies from USAID. 2. Many PIs expressed that they had a limited understanding of the development research landscape within their countries, namely in terms of what research had been or was being carried out on similar topics and by whom. PIs felt that having access to this knowledge might allow them to build upon previous research and explore collaborations in-country, thereby ensuring better research outcomes. PIs suggested that USAID missions support the creation of a country-level research database that would include contact and other relevant information that would allow researchers to learn what research has already been done on development issues of interest within their country. This could be housed within a relevant government ministry so that technical staff and other researchers could access it. EQ 4 AND 5. CAPACITY LEVEL OUTCOMES 1. Some PIs pointed out that the research administration processes within their host university is inefficient. For example, these PIs experienced delays in buying the necessary tools and USAID.GOV PEER EX-POST EVALUATION | 50 equipment, and human resources and finance department services that hampered their ability to stay on schedule. For PEER grants, developing an agreement with the procurement department could be made to ensure that the process is both easy and efficient for the PIs prior to engaging in research activities. PEER could also do capacity strengthening on research administration with host institutions to ensure more streamlined procurement policies, systems, and approaches are in place. Such an endeavor could begin with a research administration rapid capacity assessment or checklist that is included either as part of the proposal process or following award. This assessment would then inform the need for capacity-strengthening investments in this area for the PI’s host institution. 2. PIs and university officials commented that the rigorous PEER reporting demands created new challenges for those involved in the process. This problem could be mitigated through a more consultative reporting agreement between the researchers and the USAID entities. Both PIs and university officials suggested that there could be ways to streamline the reporting process that allowed more time for them to focus on research rather than reporting. This could include either less frequent reporting (i.e., every trimester or every 6 months) or fewer demands within the quarterly reports. Upon receiving the grant, the supporting university staff in the grant or department offices could receive training to help provide a base introduction to USAID-specific grants and troubleshoot any common challenges. 3. PIs reported constraints inherent to the tight time frames imposed upon their research. PIs in South Africa and Ghana stated that due to unforeseen environmental challenges, they desired a flexible end-date so that they could have met their research objectives fully rather than rushing their results to meet mandated timelines. A PEER grant end date could be negotiated to d accommodate completing research activities if unforeseen environmental or other external challenges arose during the research period. 4. Many PIs expressed low levels of interaction and collaboration between the PEER PIs and USAID Mission staff when the research projects were ongoing. Likewise, some USAID Mission staff did not have much familiarity with the PEER grants, including their purposes or processes. If USAID Mission staff are seen as a key stakeholder in achieving the PEER theory of change, it is imperative to provide opportunities and incentives to increase the interactions and collaborations between USAID staff and PEER PIs. If the proposals are in line with the CDCS and if there is a provision of connecting local USAID Mission staff with PEER research based on his/her sectoral expertise, these would increase the interaction and collaboration between USAID Mission and PEER PIs. USAID could invite researchers to informally or formally present on research progress and findings to Mission staff throughout the year. 5. Some respondents said that the requirement of connecting with a U.S. based collaborator is not easy for some PIs if they did not already have ongoing collaborations, and may leave some potentially strong PIs with non-competitive proposals. This also limits their ability to prepare a strong proposal, which seems problematic for new faculty. Establishing a mechanism within PEER to connect prospective in-country PIs with U.S. based researchers would assist many researchers in fostering collaborations that might not otherwise be possible. 6. Many PIs expressed the sentiment that collaboration with their U.S.-based researchers was not as fruitful as it could have been. PIs felt that it wasn’t a lack of will, but rather resource 51 | PEER EX-POST EVALUATION USAID.GOV constraints that inhibited true collaboration. PIs recommended that this could be mitigated if their co-PI could receive some sort of stipend or travel benefit to allow for in-person planning, rapport, and interaction. They stated that this could improve communication, offer incentives to busy researchers to serve as co-PI, and ultimately, improve their research findings and outputs.9 For EQ 6, (Regional Variation in motivation of researchers to apply for funding), we detected few statistically significant differences in regional motivation of researchers to apply for PEER funding, and our findings were not always consistent with the qualitative data. Therefore, we do not recommend targeting researchers in different regions according to their motivations for conducting research. Instead, we recommend targeting research questions which relate to government priorities or USAID’s CDCS, as described in Recommendation #3 under EQ 1&2, and Recommendation #3 for EQ 4 & 5. Future PEER programming should be aware of how the relationships between PIs and USAID Missions, hosting higher education institutions, and governmental/non-governmental organizations affect intended research impacts upon policy. The above recommendations aim to support PEER in achieving this goal. THEORY OF CHANGE AND EVALUATION-RELATED RECOMMENDATIONS 1. REVISE THE PEER PROGRAM THEORY OF CHANGE (TOC). In conducting this evaluation, our evaluation team revisited the ToC and its associated frameworks. We noted that the ToC as it exists now does not successfully integrate the role that policy-makers play in achieving the program goal of improved development outcomes through their need for, access to, and utilization of research outputs. In particular, the important step of research translation and uptake could be added between the outcomes level (new evidence) and the impact (improved development outcomes). The involvement of policy-makers could also be integrated at earlier levels of the ToC, indicating their role in incentivizing and supporting policy-relevant research. The ToC could be based on several frameworks, including public goods or principal-agent, depending on the viewpoint of the programmatic team. A second point to consider in a revised ToC is that research capacity-level outcomes, such as purchases of needed equipment, university infrastructure support, career progress, and individual student support could also be included as a development-level outcome. The ability of researchers at local universities to conduct policy-relevant research could be considered a strong development impact. 2. REFINE OUTCOME VARIABLES FOR FUTURE EVALUATIONS: This evaluation included a large number of dependent variables, some of which were not necessarily aligned with the programming that PEER provides. For example, we observed a negative relationship between participating in the PEER program and production of technology or receiving patents. This is reasonable if PEER prioritized and supported policy-relevant research. Secondly, it’s unlikely that engaging in a large-scale research project would increase the number of students reached by the PIs in their coursework. In fact, it’s more likely that PEER PIs may reach fewer students in the classroom due to a reduced course load when conducting research. These are simply two examples. Testing so many potential dependent variables can reduce 9 We learned that this concern may have been addressed in later PEER grant cycles, but it was a notable concern from the PIs we interviewed so we have included it regardless. USAID.GOV PEER EX-POST EVALUATION | 52 survey response rates, and cause statistical issues from multiple hypothesis testing. We recommend a reduction in the dependent variables for future evaluations, in order to focus future studies on outcomes which are reasonable to expect based on the PEER program’s theory of change. 3. REVISIT A PEER EVALUATION AFTER ALL CYCLES ARE COMPLETE. Our analysis of effect sizes demonstrates that our inability to detect impact on some dependent variables could be due to an under-powered study. In particular, academic output such as publications and Altmetric score could have been detected with a reasonably larger sample. Finally, given that the researchers who apply to PEER have many publications over a period of up to twenty years, we recommend a time series analysis in order to take advantage of this nature of the data. 53 | PEER EX-POST EVALUATION USAID.GOV REFERENCES Abadie, A., & Spiess, J. (2019). Robust Post-Matching Inference. 34. Arora, A., & Gambardella, A. (2005). The Impact of NSF Support for Basic Research in Economics. Annales d’Économie et de Statistique, 79/80, 91–117. https://doi.org/10.2307/20777571 Benschoter, E., Fonseca, C., Caro, D., Nordehn, C., & Orvedal, I. (2017). Enhanced Engagement in Research: Mid-Term Evaluation of the Partnerships for Enhanced Engagement in Research (PEER) Program. Cultural Practice, LLC and The QED Group, LLC. Cochran, W. G., & Rubin, D. B. (1973). Controlling Bias in Observational Studies: A Review. Sankhyā: The Indian Journal of Statistics, Series A (1961-2002), 35(4), 417–446. de Chaisemartin, C., & Ramirez-Cuellar, J. (2020). At What Level Should One Cluster Standard Errors in Paired Experiments, and in Stratified Experiments with Small Strata? 49. Ganguli, I. (2017). Saving Soviet Science: The Impact of Grants When Government R&D Funding Disappears. American Economic Journal: Applied Economics, 9(2), 165–201. https://doi.org/10.1257/app.20160180 Guo, S. Y., & Fraser, M. W. (2009). Propensity Score Analysis: Statistical Methods and Applications (1 edition). SAGE Publications, Inc. Hainmueller, J. (2012). Entropy Balancing for Causal Effects: A Multivariate Reweighting Method to Produce Balanced Samples in Observational Studies. Political Analysis, 20(1), 25–46. https://doi.org/10.1093/pan/mpr025 Iacus, S. M., King, G., & Porro, G. (2012). Causal Inference without Balance Checking: Coarsened Exact Matching. Political Analysis, 20(1), 1–24. https://doi.org/10.1093/pan/mpr013 Jacob, B., & Lefgren, L. (2007). The Impact of Research Grant Funding on Scientific Productivity. NBER Working Paper, 13519, 73. https://www.nber.org/papers/w13519.pdf McKenzie, D. (2012). Beyond baseline and follow-up: The case for more T in experiments. Journal of Development Economics, 99(2), 210–221. https://doi.org/10.1016/j.jdeveco.2012.01.002 Rubin, D. B. (1980). Bias Reduction Using Mahalanobis-Metric Matching. Biometrics, 36(2), 293–298. JSTOR. https://doi.org/10.2307/2529981 USAID.GOV PEER EX-POST EVALUATION | 54 ANNEXES A. EVALUATION SCOPE OF WORK EX-POST EVALUATION OF CDR’S RESEARCH ENGAGEMENT Version: April 5, 2019 Prepared by: Danice Brown Guzman, Lila Khatiwada and Tom Purekal (Pulte Institute) Audrey Moore and Delia Welsh (MPR) INTRODUCTION This memo responds to a request by the Center for Development Research (CDR) to evaluate two mechanisms, Higher Education Solutions Network (HESN) and Partnerships for Enhanced Engagement in Research (PEER), and to offer insights and lessons for the evaluation of a new program, Long-Term Assistance and Services for Research (LASER). This final version of the memo presents the context, proposed methodologies, costs, and key personnel for the evaluation option which was selected by the CDR team. CONTEXT Partnerships for Enhanced Engagement in Research (PEER) PEER is a competitive awards program that invites scientists in developing countries to apply for funds to support research and capacity-building activities on topics of importance to USAID and conducted in partnership with U.S. Government (USG)-funded and selected private sector partners. The program is supported by USAID, and implemented by the U.S. National Academies of Sciences, Engineering, and Medicine (referred to as the National Academies). To date, the PEER program has supported more than 300 projects involving participants in 54 countries for a total investment of nearly $59 million. The PEER mid-term evaluation report (2017) found that over one-third of surveyed researchers believed that their research products are being used to change policy at the national level. However, this report indicated that there is no clarity on how to define policy engagement of PEER funded grant. Amid such findings, CDR is interested in understanding the utility of research products. Specifically, CDR requests an evaluation that would: • Examine policies, barriers and opportunities for the success of PEER projects at PEER funded universities. • Understand underlying capacities of researchers or research institutions that lead to research uptake and increase development impact. • Do a cohort analysis to better understand investment opportunities (bang for buck type of projects or researchers) 55 | PEER EX-POST EVALUATION USAID.GOV • Do a comparative analysis of researchers in the same department: those receiving PEER funds vs those who did not (controlled for gender, seniority, training, etc.) • Understand capacity training and resources delivery that have had the highest impact on research capacity and research output (publications, students with degrees, data uptake and use) • Gather information on the relevance and use of the research from Missions, policy makers, and other research users Higher Education Solutions Network (HESN) The Higher Education Solutions Network (HESN) is a partnership between USAID and seven top universities, designed to channel the ingenuity of university students, researchers, and faculty towards global development. At the launch of the partnership in 2012, each university established a Development Lab - including two at MIT - and created a network of researchers with a mission to revolutionize development through science and innovation. The HESN Development Labs research, incubate, test, and accelerate solutions in partnership with local universities, organizations, and communities to deliver the most significant impact. The network has created a vibrant framework of cooperation between USAID Missions, local actors, development professionals, and academics to tackle the complexities of modern-day development challenges. Soon after initiation of their awards, the HESN 1.0 labs began working with USAID Missions to address Mission-relevant development challenges. These projects were funded entirely by HESN, solely by the Mission, or the projects were co-funded by both. A midterm evaluation of HESN showed that agency and Mission staff were interested in the easy access to academic research and evaluations that programs such as HESN provided to them. The evaluation also found that collectively the HESN Labs were performing well in the field across its three objectives. The Labs met or exceeded 76 percent of their indicator targets in the first three years with the strongest performance coming in the areas of improving data quality, access, and analytics to advance evidence-based development decision-making (Amula et al. 2016). Moreover, 90 percent of the stakeholders interviewed during the midterm evaluation confirmed the benefits that the Labs provided to them. New CDR Programs As the current programs have shifted more heavily towards Mission engagement, CDR is interested in understanding the ultimate impact/utility of HESN 1.0 projects on the Mission partners as well as the conditions and models that generate effective partnerships with USAID Missions and produce policy impact. Like HESN and PEER, the new CDR programs also engage researchers and universities in co-created, long-term, technical and research assistance with USAID. They are designed to allow Missions, Bureaus, or Independent Offices rapid access a network of international universities and associated researchers interested in refining and solving development challenges in partnership with USAID and local stakeholders. The new CDR programs are based, in part, on lessons learned through Mission engagement and CDR modeled elements of the new programs to facilitate access to good researchers USAID.GOV PEER EX-POST EVALUATION | 56 and grants. However, it is unclear whether and how USAID operating units use the outputs generated by the research grants to help them with future programming and decision-making. An improved understanding of the relationships between the various elements of the partnerships (funding source, type of buy-in, etc.) and the utility of the outputs for the Mission could greatly benefit the new programs. RESEARCH QUESTIONS CDR has requested an evaluation of the PEER and HESN to answer the following research questions: PEER Research Questions 1. What factors (at the levels of PI, institution, and country) increase the likelihood of achieving program and policy impacts? How has the PEER grant helped to generate new knowledge and be used for policy-making? 2. To what extent and how has PEER contributed to program and policy impacts? 3. To what extent and how has PEER contributed to capacity-related outcomes? (publications, students with degrees, data production) 4. What factors (at the levels of PI, institution, and country) increase the likelihood of achieving capacity-related outcomes? (publications, students with degrees, data production) 5. Are there regional differences of motivational factors for involvement in PEER funded activities? a. Which region has higher involvement of researchers motivated by solving specific development problems? b. Which region has higher involvement of researchers motivated by academic interest? HESN Research Questions 1. To what extent have Mission partners applied learnings from HESN 1.0 research or outputs to their programs? a. What was the usefulness of the buy-ins to their programs (i.e., output was pertinent, relevant and timely)? b. What was the utility of the buy-ins to their programs (i.e., degree to which the outputs were used by policymakers and other stakeholders)? c. To what extent did the usefulness and utility of the buy-ins vary by funding source? 2. What was the perceived utility of the partnership among stakeholders (i.e. USAID Mission, LABs, policymakers)? 3. What structural/procedural elements of the partnership contribute to different levels of usefulness to Mission programming and decision-making? Utility? 57 | PEER EX-POST EVALUATION USAID.GOV 4. What process elements of the partnership contribute to different levels of usefulness to Mission programming and decision-making? Utility? 5. To what extent has HESN contributed to changes at higher education institutions or in their networks that increases their engagement in international development? a. To what extent would any changes be sustained? Why? Or Why not? METHODOLOGY To answer the research questions described above, we plan to conduct a mixed-methods ex-post process evaluation that will (1) explore how project funds were used to generate research and the utility and usefulness of the research to policy; (2) explore whether the funding mechanism (e.g., CDR funded, Mission funded or co-funded) contributed to any differences in the type or perceived usefulness of the research; (3) help contextualize outcome results by describing the geographic, social, and policy environment in with several labs and researchers; and (4) provide a deeper understanding of how Mission engagement around the engagements contributed to differences in outputs and outcomes. The ex-post process evaluation draws on online surveys, interviews with key stakeholders, document review and administrative data. For the HESN evaluation, research will focus on three funded labs (to be chosen from AidData, MIT, Makerere University, and Berkeley). For the PEER evaluation, we will focus on the first five cycles of PEER implementation. Focusing on research which was funded between 2012 and 2016 will allow for sufficient time to have passed since funding in order to observe results such as publications and policy￾level effects. RESEARCH ACTIVITIES We will perform the following activities for the PEER and HESN Evaluations. • Document review: Both the PEER and HESN evaluation teams will begin the work by conducting a document review. The teams will review all the available documents related to both programs (program documents, program plan, progress reports/quarterly reports, evaluation reports, sustainability plans, original application submitted for funding, publications from PIs, policy documents, conference proceedings, etc.) and synthesize the findings from the documents into a memo that can be coded and analyzed. Where possible and relevant, the bibliometric approach will be utilized in the document review. During the document review process, we will create a list of all the PEER and HESN funded stakeholders with the contact information (email, phones, Skype) for future use in data collection. We will also develop a workable design to cost-effectively collect all the necessary data to respond to the evaluation questions. For the HESN document review, we will also link the research study/grant conducted at each Lab with the type of funding mechanism (CDR, Mission, cofounded) to explore any potential patterns in usage of the mechanism and type of research. We will develop summaries of the main documents and code the summaries for qualitative analysis. • Online survey: Both evaluation teams will use the document review as a basis for identifying the themes and data we want to collect from stakeholders. These themes will serve as the basis of an online survey. The online survey will include a series of multiple choice questions and Likert scales, which can be converted into quantitative data. The questions will be pilot tested and refined in the country to ensure that wording and syntax are culturally appropriate. Following the pilot USAID.GOV PEER EX-POST EVALUATION | 58 assessment, we will use existing platforms (e.g. Survey Monkey, google forms, Enterprise) to implement the online surveys. The surveys will then be sent out to USAID Mission staff, Key staff at the Labs (HESN only), and researchers. We will download and clean the data on an ongoing basis during the data collection period. For both HESN and PEER, the online surveys will focus on understanding the use of USAID TIPS, tools, approaches, and mechanisms (i.e. DIV, MERL, GCD), stakeholder perceptions around the utility and usefulness of the research grants. We will also explore the extent to which networking opportunities, meetings, and convenings strengthened partnerships in international development, how these partnerships changed over time, strengths and challenges of the mechanisms, and their influence on both policy and USAID Mission programming. The PEER online survey will document the motivational factors, type of research activities performed, research outputs, collaboration, publications, engagement of policymaking bodies and linkages between research and policy changes. The HESN online survey will also explore core themes such as the utility of each type of funding mechanism (CDR funded, Mission funded, co-funded), the ease of utilizing the funding, and linkages between research and policy changes. We will also gather quantitative data related to indicators such as the number of research activities for undergraduate and graduate students related to international development; the number of development-focused collaborations across academic disciplines, research institutes, and student service units; and the number of new faculty demonstrating interested in international development. The HESN team will conduct a separate online survey with each HESN stakeholder group so that we can triangulate their perceptions on similar issues. The online survey will be administered at the beginning of the fieldwork so that we can use preliminary results to inform our selection of countries and the development of interview protocols. Based on the results of the online survey, both the HESN and the PEER research teams will develop qualitative protocols to explore these same themes (or aspects of the themes) in more depth through interviews and focus groups as described below. • In person interviews: With regards to policy makers and key stakeholders at the national level, the ERIE team recognizes that in-person interviews are the ideal way to collect information from this group. The ERIE team will select three to four countries to visit for in person data collection. For PEER, we will select four countries (one from Africa, one from Asia, one from Middle East and one from Latin America) with higher number of PEER grants and travel there to meet with the USAID, researchers, and other stakeholders such as local Mission and government counterparts to understand PEER’s contribution to policy. The HESN evaluation team will select three countries (one from Africa, one from the Middle East and one from Latin America) --two countries that show high levels of engagement among USAID Missions and one country that had low levels of utilization among Mission staff. We will travel to each country to meet with USAID staff, researchers, and other stakeholders such as local Mission and government counterparts to understand why Mission staff chose to use the mechanism and the contributions of the research conducted by local researchers. We will develop a semi-structured interview protocol with key questions and conduct in-depth interviews with the selected stakeholders. We will also gather examples of how the Missions have used USAID tools and approaches to increase the use of research and influence policy. We anticipate conducting six to ten interviews in each country. The interviews will gather in-depth information on similar topics to the online survey (which will be used to develop the interview protocols). For HESN, interviews will focus on themes such as perception of the HESN program and tools, awareness and visibility of the role of science and engineering in improving conditions for people living in poverty, and how the mechanism contributed to stronger partnerships. The interviews and site visits will also explore the extent to which HESN labs have sought external 59 | PEER EX-POST EVALUATION USAID.GOV funding since the mid-term evaluation. If there are a sufficient number of local researchers in the country, we may choose to conduct a group interview or focus group to take advantage of multiple perspectives in the same interview. Travel for this activity will be coordinated between PEER and HESN teams evaluation teams if the same countries are selected and scheduling allows. • Phone Interviews: In conjunction with the in-person interview process, we will select a sample of additional USAID Mission staff and researchers who completed the online survey, and conduct phone interviews based on the scripts for in-person interviews. The HESN team estimates that they will conduct between 10 and 15 additional interviews, which accounts for nearly 25 percent of the grants allocated under the HESN mechanism. The PEER team will conduct 20-25 phone interviews with a subsample of Mission staff and researchers. We will use maximum variation sampling to select participants, drawing from Missions that had high, medium and low usage of the grants. This process allows us to learn more about factors that played into the decision-making process for selecting the mechanism. We will conduct those interviews by phone or video conference call. We will record all in-person and phone interviews as well as take detailed notes during the interview process. Based on the availability of resources, we will transcribe and code key interviews. If we are unable to transcribe all the interviews, we will write summaries of the interview findings and analyze the summaries in NVIVO or Atlas-ti. • Data analysis and reporting: We will use both quantitative and qualitative software for data analysis and reporting. The analysis of qualitative information will be based on the variability and uniqueness of responses, using software for consolidating qualitative information. Codes will be developed to guide the analysis. In addition to presenting qualitative analysis results and frequencies, quotes will be used to support the findings of quantitative results in the report. We will prepare two reports (one for HESN and one for PEER) and share with CDR for their review and feedback. In addition to these analyses, we will explore using USAID’s Program and Policy Change Score Card to examine how PEER and HESN research contributed to policy change in LMICs. The results would contribute to USAID’s indicator, SO #1: High impact program or policy changes made by public sector, private sector, or other development actors that are influenced by Lab-funded research results or related scientific activities. We will further explore the possibility of doing a cost￾effectiveness analysis of USAID investments against outcomes. The feasibility of this will be assessed in the detailed evaluation designs. The report will address if and how the findings demonstrate that the PEER and HESN research contributed to the Lab’s scientific objectives, specifically IR1 (improved quality of research), IR2 (increased locally-focused R&D for development) and IR3 (strengthened science ecosystem). The final version of the report will address all feedback from CDR and other USAID stakeholders. • Preparation of case studies: We will also prepare several case studies, which offer a more in￾depth look at several PEER-funded research activities which resulted in policy-level changes, and the most utilized HESN grants to highlight how the mechanism contributed to international development and the factors that most affected the utility of the mechanism for USAID Missions. The ERIE team will prepare three to five case studies for PEER and two case studies for HESN. These case studies will most likely be generated from data collected during fieldwork, which would provide the best opportunity to understand the policy-level effects of the program. These case studies will be included in the reports described above. USAID.GOV PEER EX-POST EVALUATION | 60 Sub-Activity I: PEER Counterfactual Analysis The structure of the PEER activity presents a unique opportunity to gain a deeper understanding of potential effects of the PEER funding in terms of research outcomes including publications, citations, students with degrees, and data production. In the case of PEER, demand (researcher applications) far exceeded supply (available funding); therefore, these competitive applications can be used as a comparison group of researchers. To better address PEER research questions 3 and 4, we will develop a quasi-experimental design which utilizes this substantial information provided from PEER applications. For creating the counterfactual, we will get the basic information (demographic and institutional affiliation) from PEER applications of all the PIs who applied for funding and whose applications were ranked as competitive. The counterfactual analysis would help us provide the following information: What would be the outputs (research findings, policy change, publication, citation, policy engagement) of the researchers in the absence of PEER funding? We will perform following additional steps for the counterfactual analysis of the PEER program: • Prepare a list of non-PEER Researchers: Using the data from applications submitted to PEER for funding we extract names, gender, country, position and institutional affiliation of the applicants and create a list for gathering information using web-scraping. • Web-scraping: We will use data-mining tool to gather information from the publicly available profiles of PEER and non-PEER researchers. We will investigate the position held, number of publications, number of citations, conferences attended, number of funded grants, patented products, and professional networking such as number of contacts. Using R, we will collect this data from the web, compile all the data together into one source, and analyze the data using regression. Regex functions will be used to pinpoint and extract the data of interest from the webpages gathered about each researcher. Ideally, each researcher's Google Scholar page will be used, but LinkedIn and other websites like university/company profile pages may also be used to gather data about papers, presentations, and conferences that the researchers may have written/attended after submitting for a grant. After collating all the data, linear regression will be used for modeling the data. Information collection through web-scraping will serve as outcome variables in the model. This approach will be limited to publicly available outcomes such as publication output. Another challenge is that non-PEER PIs may have been awarded funding from other sources to research if s/he did not get a grant from PEER and this information may not be available publicly in all cases. • Online Survey of non-PEER PI’s: We will adapt and send the online survey to the non-PEER PIs as well. Similar to the PEER PI survey, the non-PEER PI survey will address, motivational factors, type of research activities performed, research outputs, collaboration, publications, engagement of policymaking bodies and linkages between research and policy changes. However, it will not reference PEER funding itself, or USAID TIPS, tools, approaches, and mechanisms (i.e. DIV, MERL, GCD), as these researchers did not have any interaction with these approaches or funding mechanisms. • Comparing the PEER Researchers vs. non-PEER Researchers: If a numeric score for each PEER applicant can be provided by USAID, the research team will employ a regression discontinuity design (RDD) to compare the outcomes achieved by PEER researchers with the PEER applicants who 61 | PEER EX-POST EVALUATION USAID.GOV were “near misses” the grant (non-PEER researchers). If a numeric score is not available, we will use an instrumental variable, matching or other analytical approach to select the groups. Sub-Activity II: Policy Brief for LASER The findings from these evaluations can inform the evaluation design for LASER. We will develop a 5-7 page policy brief that brings together the conclusions of the PEER and HESN evaluations. The policy brief will compare the recommended elements from the review of PEER and HESN with the current version of LASER and identify gaps for future research. The policy brief will also provide broad design options for the LASER evaluation. BUDGETING The ERIE team plans to coordinate to the extent possible in terms of development of qualitative and quantitative tools, and hope to leverage support from each other during travel as schedules allow. We will also collaborate during data analysis and report-writing in order to deliver coherent messages which can easily be compared and cross-referenced. The budget presented here reflect this collaboration. Pulte Institute: Leading PEER Evaluation, contributing to HESN and LASER Recommendations MPR: Leading HESN Evaluation and LASER Recommendations Total Labor $236,000 $229,400 $465,400 Other Direct Costs $133,600 $44,700 $178300 Total $369,600 $274,100 $643,700 USAID.GOV PEER EX-POST EVALUATION | 62 HIGH LEVEL TIMELINE Exact timeline will be agreed-upon in the design document; however, a high-level timeline is presented here as an estimate. PERSONNEL AND STAFFING To implement the proposed evaluation, NDIGD and Mathematica Policy Research has brought together highly experienced experts in designing and implementing impact evaluations in developing countries with extensive experiences in collecting quantitative and qualitative data from primary and secondary sources as well as using machine learning techniques. Our diverse team has experience managing the complex range of tasks required by CDR for these important evaluations. Our proposed project organization reflects the lines of authority and responsibility for accomplishing the work and indicates the leaders who will oversee each evaluation. Key Personnel Proposed Candidates Qualifications Notre Dame University PEER Evaluation Team PI Dr. Lila Kumar Khatiwada Dr. Khatiwada has a Ph.D. in Rural Sociology, more than 20 years of experience designing, implementing and evaluating programs in the area of agriculture, economic development, climate change, and health in global south. He currently involved in evaluation in Malawi, Nepal, East Bank/Gaza, and Uganda. Co-PI Dr. Paul Perrin Dr. Perrin has a Ph.D. in International Health at the Johns Hopkins University, more than 15 years of experience supporting the conceptualization, design, implementation, analysis, and reporting on monitoring and evaluation and implementation science activities in resource-challenged 63 | PEER EX-POST EVALUATION USAID.GOV environments. He currently involved in evaluation in Haiti Bolivia, Bangladesh and India. Co-PI Mrs. Danice Brown Guzman Mrs. Guzman has a Masters in Public Policy at Duke University, more than 10 years of experience in program management and evaluation. She currently involved in evaluation in Malawi, Peru, Uganda, Haiti, Nepal, West Bank/Gaza. Research Support TBD The research analyst (TBD) will bring 5-7 years of experience working on data collection and analysis. The research analyst will have experience developing instruments, overseeing data collection in the field and coding and analyzing qualitative and quantitative data. Data Analyst Ms. Jessica Young Ms. Young has a MS in Applied Statistics, more than 5 years of experience in data mining and analysis. Mathematica Policy Research HESN/LASER Evaluation Team PI Dr. Audrey Moore Dr. Audrey Moore has a Ph.D. in Education Policy and Administration, more than 15 years of experience designing and implementing government programs and evaluations around the world. Dr. Moore is a mixed methods evaluator who brings both extensive quantitative and qualitative evaluation skills. She current leads RCT and performance evaluations in Guatemala, Cabo Verde, and Mozambique. Research Analyst TBD The research analyst (TBD) will bring 5-7 years of experience working on evaluations. The research analyst will have experience deep quantitative experience as well as experience coding and analyzing qualitative data, developing instruments, and overseeing data collection in the field. Quality Assurance Dr. Nancy Murray Dr. Murray has a Ph.D. in Sociology from Johns Hopkins University and more than 25 years of experience designing, managing and implementing performance and impact evaluations. She has served as project director of several large, multi-country evaluations in West Africa, Asia, and Latin America including the USAID-funded LAC READs program evaluation project. Research Support Delia Welsh Ms. Welsh has an M.P.A. and over 15 years of experience working on evaluations and has worked closely with academics pursuing research that has a policy impact for most of her career. She represents Mathematica in the USAID.GOV PEER EX-POST EVALUATION | 64 ERIE consortium and will provide research support to the HESN evaluation and policy brief on both evaluations. Exact roles will be determined upon award; however, key researchers will likely include the following personnel: PULTE INSTITUTE FOR GLOBAL DEVELOPMENT Dr. Lila Khatiwada: Dr. Khatiwada has implemented research work in Burkina Faso, Ethiopia, Ghana, Indonesia, Kenya, Mozambique, Namibia, Nepal, and Uganda. He has developed evaluation designs (mostly using mixed method), prepared sample frame, trained enumerators, developed data collection instruments, supervised data collection work, analyzed data and prepared reports. Further, using participatory approach, Dr. Khatiwada has designed and implemented community health, gender development, agriculture, forestry, microfinance and enterprise development, and institutional development projects in Nepal for ten years. Currently, he is leading a project to evaluate the impact of a UNICEF intervention to reduce child labor in a US Dept. of Labor (USDOL) funded project in Nepal where he is using mixed method in impact evaluation. In Malawi, he is involved in evaluating a long-term impact of USAID funded WALA project in food security where he is assisting in implementing qualitative part of study. In Ghana, he designed and oversaw qualitative and quantitative data collection work for MCC in a WASH evaluation. He is proficient in using qualitative software, ATLAS.ti. Dr. Khatiwada holds a Ph.D. in Rural Sociology and a Graduate Certificate in Geographic Information System (GIS) from the University of Missouri and an MS in Rural Sociology and Environment Science from the University of the Philippines. He has published his research works in American Behavioral Scientists, Journal of Regional Analysis and Planning, Maternal and Child Health, Journal of International Tourism Research, Africa Today, Eurasian Geography and Economics journals and has reviewed for the World Development, Journal of International Tourism Research, and Community Informatics. Dr. Paul Perrin: Dr. Perrin is the Director of Monitoring and Evaluation at the Notre Dame Initiative for Global Development and a Concurrent Associate Professor of the Practice in the Keough School of Global Affairs. Dr. Perrin’s work has spanned academic, government, and non-governmental settings and has largely focused on supporting the conceptualization, design, implementation, analysis, and reporting on monitoring and evaluation and implementation science activities in resource-challenged environments. Currently, he is leading the Haiti Reads project, where he leads the research team to implement EGRA literacy assessments in French and Creole of students in over 250 schools in Haiti. Prior to joining Notre Dame, Dr. Perrin served as the Director for Monitoring, Evaluation, Accountability, and Learning (MEAL) at Catholic Relief Services (CRS). In this role, Dr. Perrin oversaw the development and implementation of a global MEAL strategy, which has led to the incorporation of evidence-based learning into CRS programming across the world. Dr. Perrin has also worked as Senior Technical Advisor for Health Research and Evaluation at CRS, as an Associate in the Johns Hopkins Center for Humanitarian Health, and an Information Officer at USAID’s Office of U.S. Foreign Disaster Assistance. Dr. Perrin completed his PhD in International Health at the Johns Hopkins University Bloomberg School of Public Health, with a certificate in Humanitarian Assistance. Paul also received a Masters in Public Health and a Bachelors of Arts in Linguistics from Brigham Young University. Dr. Perrin is proficient in French and will be responsible for providing his expertise in study design, examining project outputs from policy perspective, data analysis, report writing and dissemination. 65 | PEER EX-POST EVALUATION USAID.GOV Mrs. Danice Brown Guzman: Mrs. Guzman has experience as a program manager for education evaluations and in nutrition. Mrs. Guzman has worked on an impact evaluation for a community drinking water project in Burkina Faso, MCC’s WASH project in Ghana and coordinated the gathering of baseline data on the USDA Benin Food for Education Program. Mrs. Guzman has also served as the program manager for designing new USAID evaluation tools, including the $3.4M USAID Expanding the Reach of Impact Evaluation grant. Her education has focused on quantitative evaluation methods such as randomized controlled trials and quasi-experimental methods such as regression discontinuity, difference in difference, synthetic controls, and propensity score matching, including software skills in STATA, ORA and Gephi (social network analysis). Mrs. Guzman has a Master’s of Public Policy from the Sanford School of Public Policy at Duke University and a Certificate in International Development from the Duke Center for International Development. MATHEMATICA POLICY RESEARCH Dr. Audrey Moore is an educational economist with over 20 years of experience working in research, evaluation, educational policy analysis, and analysis of educational development projects focusing on children and youth. She has worked in more than 15 countries across Latin America, Africa, the Middle East, and the U.S. to design, implement and analyze impact and performance evaluations, implementation studies, quasi-experimental, and mixed-methods designs. Dr. Moore also has experience designing and implementing cost-effective and cost-benefit analysis. She is currently the project director for three MCC evaluations including the Guatemala Threshold Education program (Exito Escolar), where she is leading the design and implementation of two performance evaluations and an impact study. As Project Director for the MCC Land Management for Investment Performance evaluation in Cabo Verde and the Mozambique WASH performance evaluations, she is leading the design and implementation of outcome￾based evaluations that assess the extent to which the programs met their objectives. She also works as a senior researcher supporting a performance evaluation of the MCC Morocco secondary education program evaluation and the MCC-funded WASH evaluation in Cabo Verde. Before joining Mathematica in 2015, Dr. Moore spent 14 years leading and supporting USAID projects around the world. Dr. Moore has presented to numerous international and U.S. audiences and has published in major academic journals including Development in Practice, Journal of International Cooperation in Education, World Studies of Education Journal, and Education Review. Dr. Moore is also an adjunct faculty member at Georgetown University’s foreign policy school, where she co-teaches a course on practical evaluation. Dr. Moore is natively fluent in English and Portuguese; fluent in Spanish and has basic, functional proficiency in French. She holds a Ph.D. in Education Policy and Administration. Dr. Nancy Murray has more than 25 years of experience in international development program evaluation, education, adolescent and maternal and child health, and nutrition and the family. She is the International Area Leader, and the Deputy Director of the Center for International Policy, Research and Evaluation (CIPRE). She is project director of the LAC Reads project, a USAID-funded set of impact evaluations of early reading initiatives in Latin America and the Caribbean. She also directs the impact evaluation component of the School Dropout Pilot Prevention (SDPP) project, an impact evaluation of school dropout prevention programs in four Asian countries, in collaboration with Creative Associates. Her past work includes key roles in designing and leading evaluations of international health and education programs in Latin America, the Caribbean, and Africa. Dr. Murray, who joined Mathematica in 2007, was previously the deputy director for the Center for Health Solutions at Futures Group International, where she was responsible for all projects with evaluation and monitoring components. She has published in Studies in Family Planning, Social Biology, International Family Planning Perspectives, USAID.GOV PEER EX-POST EVALUATION | 66 Vulnerable Children and Youth Studies, and Maternal and Child Nutrition. She is fluent in Spanish and proficient in Portuguese and French. She holds a Ph.D. in sociology from the Johns Hopkins University and an M.P.H. and M.A. in Latin American Studies, both from the University of California, Los Angeles. Ms. Delia Welsh has an extensive background in the monitoring, evaluation and learning (MEL) of international development programs. As a senior researcher at Mathematica, she is a member of the MERLIN/ERIE consortium. Delia is also working on evaluations of large infrastructure programs in Ghana and Tanzania, and has developed and has developed several MEL guides. Prior to joining Mathematica, she was a senior staff member of Innovations for Poverty Action (IPA) where she supported the development of a large academic research network and oversight of impact evaluations, policy outreach, and research operations in several developing countries. Prior to joining IPA, Ms. Welsh was a member of the start-up team of the Millennium Challenge Corporation (MCC) and worked with MCC country representatives to develop monitoring and evaluation strategies for several large compacts. Ms. Welsh holds an M.P.A. in Public Affairs from Princeton University. PARTNER ORGANIZATIONS Pulte Institute for Global Development The University of Notre Dame Pulte Institute for Global Development combines the existing world-class teaching and research faculty of the University with a dedicated staff of experienced international development professionals, administrators, and researchers in the areas of 1) impact evaluation; 2) training & capacity building; and 3) applied innovation. Through learning, documentation, and policy development these researchers help address the challenges of building just and equitable societies in fragile nations. The Pulte Institute cuts across academic fields of expertise to produce multidisciplinary knowledge on complex development challenges in areas related to security and peace building; the rule of law; global health; infrastructure; agriculture and food; energy and the environment; commerce and economic development; and education. In addition to analyzing policy, training & capacity building, and disseminating and applying lessons learned, The Pulte Institute monitoring and evaluation specialists have hands‐on experience in quantitative and qualitative research methods; designing field researchers, preparing data collection instruments, supervising surveys; integrating data derived from different sources, manipulating and conducting statistical analyses. They have work experience in implementing food security, poverty alleviation, natural resource management, microfinance, and institutional development projects in conflict situations; using logic models in program implementation and evaluation. NDIGD researchers have expertise in STATA and SPSS programming, manipulating, summarizing, and producing reports from large and complex dataset and conducting econometric analysis using advanced statistics. NDIGD staff also have technical expertise in Geographic Information Systems (GIS). The Pulte Institute is at the forefront of using technology to collect data. Leveraging University of Notre Dame expertise through its Information and Communication Technologies for Development (ICT4D) Specialists, and the Center for Research Computing, the Pulte Institute offers unique approaches to monitoring and evaluation that fuse together technologies such as Global Positions Systems (GPS), Remote Sensing (RS), and Geographic Information Systems (GIS) with fundamental expertise in monitoring and evaluation. Where applicable, the Pulte Institute projects use smartphones or tablets to collect data and GPS coordinates to track respondents. Making use of our many state-of-the-art technologies, Pulte Institute experts are able to more efficiently and effectively measure the progress of projects and measure changes in outcomes. In addition, state of the art technology in monitoring allows us improve project implementation, ensure accountability and transparency, and make smarter policy recommendations. 67 | PEER EX-POST EVALUATION USAID.GOV Mathematica Policy Research (MPR) Committed to improving lives of disadvantaged people, Mathematica has 50 years of experience conducting a wide range of research and evaluation activities, data collection, impact and performance evaluations, and providing training and technical assistance in a large number of sectors domestically and internationally. Mathematica has successfully managed dozens of large and complex $20 million projects with multiple partners and working closely with academics. Our organizational structure includes strong administrative support system that provides well-established management processes and procedures and allows us to easily manage large-scale, complex projects and nimbly respond to client requests. With a staff of more than 1,200 employees, we have a deep bench of researchers who offer a wide range of skills, many of whom do international research. To achieve CDR’s goals for the HESN evaluation, we are proposing a team of highly qualified researchers led by Senior Researcher Dr. Audrey Moore. Our team brings far-reaching experience conducting research studies and using evidence to inform decision-making for USAID and other government agencies. Mathematica has worked in more than 30 countries. Through our expansive geographic footprint, we can identify the appropriate stakeholders, understand any possible data collection challenges, and facilitate any logistical needs. In addition to our team’s extensive experience in multiple countries, our partner EDI provides us with a robust operational platform in Africa through its offices in Mozambique, Tanzania, and Uganda. Mathematica also brings more than 15 years of experience managing working with local researchers around the world. We take a hands-on approach to fieldwork. Our internal teleconference and WebEx systems enable us to connect to globally dispersed teams and provide ongoing support. USAID.GOV PEER EX-POST EVALUATION | 68 B. SURVEY TOOLS INTRODUCTION On behalf of the USAID Partnerships for Enhanced Engagement in Research (PEER) Program, a team of independent researchers from the University of Notre Dame are conducting an ex-post impact evaluation. Your participation and honesty will help us provide a more accurate assessment of PEER so the managers may improve the program. The purpose of the evaluation is to assess the research and policy outcomes achieved by the PEER program. This Evaluation is NOT an audit, and neither is the purpose to assess individual performance of any project or individual. Through this survey, we will be asking you a series of questions about your experience conducting research funded by PEER or other institutions: • outcomes and impacts of your research • collaborative activities • changes in scientific capacity This survey should take 45-60 minutes to complete. Some data you provide in this survey may be unique to you and will allow us to identify you. In addition, your responses will be linked with USAID programmatic data concerning your application or research output for analysis purposes. This identifying information will be kept confidential and will be safely stored so it can only be accessed by the research team. The final evaluation report will present aggregated data only, no individual responses or identifiable information will be reported. The final report will be available through the PEER website, along with an anonymized dataset with any identifiable information removed. We would like to follow up with a few researchers to learn more about their experience with PEER. At the end of this survey, we will ask if you are willing to have a short phone call with our research team. This is not required, but if you agree, you can provide your name and contact information at that time. If you have any questions about this survey or the evaluation, please contact the Senior Evaluation Technical Lead for this Survey – Danice Guzman at [email] We thank you for your time and help! *Before you start, please click on the box to provide consent to participate in this survey.11 -Yes I consent. –No I do not consent. [if “no” to Q1, skip to end] 11 Asterisks indicate the few required questions in this survey. 69 | PEER EX-POST EVALUATION USAID.GOV SECTION 1: PEER AND OTHER FUNDING INFORMATION This section should capture all data on respondents’ applications to PEER and other funding sources, both before and after the PEER application. It will also provide some information on the respondent’s qualifications at time of PEER application. *What cycles did you apply to PEER? [select all that apply] • Science Cycle 1 (awards announced May 2012) • Science PIRE Cycle (awards announced December 2012) • Science Cycle 2 (awards announced June 2013) • Health Cycle 1 (awards announced July 2013) • Health Cycle 2 (awards announced August 2014) • Science Cycle 3 (awards announced August 2014) • Cycle 4 (awards announced August 2015) • Cycle 5 (awards announced 2016) • Not sure *Were you awarded a PEER grant? Yes/No [if no to Q3, skip to Q5] *What cycle were you awarded? • Science Cycle 1 (awards announced May 2012) • Science PIRE Cycle (awards announced December 2012) • Science Cycle 2 (awards announced June 2013) • Health Cycle 1 (awards announced July 2013) • Health Cycle 2 (awards announced August 2014) • Science Cycle 3 (awards announced August 2014) • Cycle 4 (awards announced August 2015) • Cycle 5 (awards announced 2016) [Calculate: $YEARS: Count of years since first application] [if No to Q3] Did you submit a proposal for the same research activity to other funders? Yes/No [if yes to Q5] Was it awarded? Yes/No Have you applied for funding from other institutions for other research activities in the past two years? Yes/No [If yes to Q7] How many other proposals have you submitted in the past [$YEARS]? [integer] [if yes to Q7] Please indicate the largest amount of these proposals in USD. [integer or “I don’t know”] USAID.GOV PEER EX-POST EVALUATION | 70 9a. If respondent does not know exact amount, indicate from below list: • less than $30,000 • $30,000-$49,999 • $50,000-$99,999 • $100,000-$199,999 • $200,000-$299,999 • more than $300,000 [if yes to Q7] How many were accepted? [integer] [if PEER] Is your PEER project completed? Yes/No Why did you choose to explore the research question(s) listed in your PEER application? Select all that apply. • To solve a development problem/issue • Change a policy • General academic interest • Make a contribution to my field • Try a new method • Gain experience in a new academic area • Progress in my career • Other, specify Had you collaborated with a US-based researcher before [$YEAROFAPP]? Yes/No [non-PEER only] Have you collaborated with a US-based researcher since [$YEAROFAPP]? Yes/No Before [$YEAROFAPP] what was the largest grant you had ever received for your research? in USD. [integer or “I don’t know”] 15a. If respondent does not know exact amount, indicate from below list: • Had not received any grants before [$YEAROFAPP] • less than $30,000 • $30,000-$49,999 • $50,000-$99,999 • $100,000-$199,999 • $200,000-$299,999 • more than $300,000 What position did you hold in $YEAROFAPP? • Senior/Full Professor • Junior/Associate/Assistant Professor • Professor University Lecturer (non-tenure track, non-adjunct) 71 | PEER EX-POST EVALUATION USAID.GOV • Visiting or Adjunct Faculty • Research Scientist • Post doctorate • College Dean/Department Chair/College Rector/ or equivalent • Extension Agent/Faculty • Other, specify What position do you hold now? • Senior/Full Professor • Junior/Associate/Assistant • Professor University Lecturer (non-tenure track, non-adjunct) • Visiting or Adjunct Faculty • Research Scientist • College Dean/Department Chair/College Rector/ or equivalent • Extension Agent/Faculty • Other, specify Have you received any promotions (in salary or title) since $YEAROFAPP? Select all that apply: • Increase of salary • New title • Others, specify Have you begun any additional paid or advisory role for government, think tanks, or private sector since $YEAROFAPP? Yes/No [if yes] Who is the employer? • University • Research Institute • Think Tank • Government Agency • Private sector • Non-governmental organization • Other, specify In what country were you living in $[YEAROFAPP]? Select from list of all countries In what country do you live now? Select from list of all countries USAID.GOV PEER EX-POST EVALUATION | 72 SECTION 2: CAPACITY AND POLICY OUTCOMES OF RESEARCH Note: This section will help us understand the impact of PEER on research output, capacity development of the researcher and the policy impact of the research. Have you produced any of the following products? Please consider all research activities you’ve worked on in the past two years. Mark all that apply. PRODUCTS COUNT OF PRODUCTS IN LAST 2 YEARS COUNT OF PRODUCTS SINCE $[YEAROFAPP] patent proposal new courses new trainings or workshops new materials for trainings or workshops blogs/vlogs/videos with scientific instruction paper/poster presentation at a conference paper/poster presentation at my Department/Institution MS/PhD thesis concluded by one of my students MS/PhD thesis concluded for myself scientific report(s) new technology open access data set(s) stakeholder workshops/training technical and/or managerial guidelines/documents market analysis documents/reports policy recommendations/analysis for government agencies business plans programmatic strategies Other, specify 73 | PEER EX-POST EVALUATION USAID.GOV Have you or your students traveled domestically or internationally for any of the following reasons? Select all that apply. PURPOSE SELF IN PAST 2 YEARS (# OF TIMES) SELF SINCE $[YEAROFAPP] (# OF TIMES) STUDENTS SINCE $[YEAROFAPP] (# OF TIMES) Have any of your students received funding for their research since $[YEAROFAPP]? Yes/No 25.a. About how many of your students continue to work in your area of research after graduating? • None • Few • Some • Many • All • Don’t know 25.b. About how many of your students work on topics relevant to international development or policy after graduating? invited presenter/speaker/panelist at academic conference participated in academic conference presented research findings to government, NGO, policy maker audience lead a workshop or training related to your research write or collaborate with other researchers take a training or course pursue a degree USAID.GOV PEER EX-POST EVALUATION | 74 • None • Few • Some • Many • All • Don’t know What have you done with products of your research in the last two years? Select all that apply. COUNT OF PEOPLE/POSTS 75 | PEER EX-POST EVALUATION USAID.GOV In your opinion, how do you think individuals are USING your research product(s)/results? (MARK ALL that apply) • to change policy at the national level • as guidance to communities posted on institutional webpage/newsletter posted on social media (Twitter, Facebook) Shared with the following: NGO or international organization Other people in my scientific community USAID staff US Scientists US Science Government Agencies (examples: NOAA, NSF, NASA, NIH, other) my country's National Science institution(s)/academies national government agency regional government agency local government agency local community Rector/Dean of my university Scientists from my Department(s) of my University/Institution Scientists from other Departments of my University/Institution Scientists from other Universities in my country Non-Profit organizations Private Sector/For-Profit organizations Other, specify USAID.GOV PEER EX-POST EVALUATION | 76 • as information for improved decision-making by my government • to build other products/technologies • to change policy at the community/village level • to improve programs at government agencies • to improve evidence in my field of research • to improve programs at USAID • to improve programs at non-governmental organizations • to bring more grants for the research • no one is using my products do not know How much do you think your research products have contributed to policy changes at the following levels: (select from Not at all, A little, somewhat, a lot, Don’t know) A. University B. USAID missions C. Other US government offices D. National Government E. Regional or Municipal Government F. Provincial or Local Government G. Village H. Non-Government Organization I. Corporations J. Multilateral Institutions [The next five questions are only for PEER recipients] How strongly do you agree with the following statements (select from Strongly disagree, disagree, neither agree or disagree, agree, strongly agree) A. "receiving PEER support has helped me to leverage new funding" B. "receiving PEER support has helped me advance my career" C. "as a PEER participant I have been recognized by my institution as a leading scientist" D. "as a PEER participant I have more students attending my courses" E. "as a PEER participant I have been recognized by my colleagues in my field as a leading scientist" F. “as a PEER participant I can effectively communicate my research findings to decision-makers in policy and development programming” Did you participate in any regional meetings as a part of your participation in the PEER program? Yes/No [if yes to #30] Which did you attend? Select all that apply. • Bangkok • Arusha • Lima • Amman 77 | PEER EX-POST EVALUATION USAID.GOV • Jakarta • Accra • Other, specify [if yes to #36] Please rate the workshop in terms of the following: Select from Not useful at all, not useful, somewhat useful, very useful, or NA • Networking opportunity with other researchers • Networking opportunity with USAID or other funders • Chance to share results of research • Learn new skills in conducting research • Learn new skills in budgeting, grants administration • Other, specify To what extent has PEER changed your relationships with the following: (mark no change, greatly weakened, weakened, strengthened, greatly strengthened) • scientists inside your institution/university/organization • scientists outside your institution/university/organization within your country • scientists from other countries • USAID or other US government agencies • Decision-makers in policy and development programming (following 2 questions for PEER only) 33b. Have you maintained a connection with your US-based partner after completing the PEER research? • Yes • No 33c. Have your students maintained a connection with your US-based partner after completing the PEER research? • Yes • No • Not applicable Since $[YEAROFAPP], how many students have you supported directly with your research? (examples: to work in your lab, support your research, help teach courses, graduate research assistantships, graduate teaching assistantships, paid academic expenses, other) • Undergraduate students (BS, BA, or equivalent): • Graduate Students (MS, MA, or equivalent): • Doctoral Students (PhD or equivalent): • Medical Students (MD Medicine, MDV Veterinary medicine, or equivalent) • Post-doctoral Students USAID.GOV PEER EX-POST EVALUATION | 78 Can you upload an up-to-date CV which lists all your current publications? Yes/No If yes, upload here The following 8 questions are relevant if no CV uploaded Please share your total number of published peer-reviewed journal articles published since $[YEAROFAPP] [integer] [if Q36>0] Please name the journals or publication outlets which approved or published your research? IF more than 1, list all here. [if Q36>0] What are the titles of the article approved or published? IF more than 1, list all here. [if Q36>0] Please include links, year of publication, DOIs or any other information on these articles. IF more than 1, list all here. Please share your total number of books or book chapters published since $[YEAROFAPP] [integer] [if Q40>0] Please name the publisher which approved or published your research? IF more than 1, list all here. [if Q40>0] What are the titles of the books or chapters approved or published? IF more than 1, list all here. [if Q40>0] Please include links, year of publication, DOIs or any other information on these books or chapters. IF more than 1, list all here. Have you written or been quoted in any articles in the national newspaper on the topic of your research in the past two years [yes/no] [if Q44=yes] Please name the newspaper IF more than 1, list all here. [if Q44=yes] What are the titles of the articles? IF more than 1, list all here. [if Q44=yes] Please include links or any other information on these articles. IF more than 1, list all here. Have you participated in any interviews on radio or TV related to your research in the past two years? [yes/no] [If yes to Q48] How many times? Since $[YEAROFAPP] how have you changed your involvement in professional networks in your field of the following? • Greatly Increased, • Somewhat Increased, 79 | PEER EX-POST EVALUATION USAID.GOV • No change, • Somewhat Decreased, • Greatly Decreased Since $[YEAROFAPP] how have you changed your access regarding the following? Select from Greatly Increased, Somewhat Increased, No change, Somewhat Decreased, Greatly Decreased • infrastructure (labs, equipment) • teaching (course/curriculum offerings) • information about collaborative opportunities • international funding opportunities Since $[YEAROFAPP], how have you changed your ability to do the following? Select from Greatly Increased, Somewhat Increased, No change, Somewhat Decreased, Greatly Decreased • collect data • manage or analyze data • write research articles • communicate your results SECTION 3: DEMOGRAPHICS How old are you? [answer in years] [The following 3 questions relevant if no CV uploaded] Have you changed home institutions since $[YEAROFAPP]? Yes/No [if yes] In what year did you change institutions? (drop down box of years) Mark ALL the degrees you have completed: DEGREE YEAR COMPLETED NAME OF INSTITUTION COUNTRY Doctor of Philosophy (PhD) Medical Doctor (Degree in Medicine) Doctor of Veterinary Medicine Juris Doctor or Doctor of Laws (Law Degree) Masters Degree (MS, MA, MBA or equivalent) USAID.GOV PEER EX-POST EVALUATION | 80 Bachelors Degree (BA, BS, AB or equivalent) [for cycle 2 applicants only] Are you male or female? Male / Female We would like to learn more about some of the researchers who complete this survey, either through web searches of your academic output or short phone conversations. If you agree, please provide the following information. You are not required to complete this section. If you do not wish to provide this information you can leave it blank! Name: Email: Phone Number: Google Scholar link: LinkedIn Profile: Thank you for your responses, this completes the survey! 81 | PEER EX-POST EVALUATION USAID.GOV INTERVIEW GUIDES All the interviews will begin with the following introduction: We, [insert Name] are from University of Notre Dame, USA and are evaluating the USAID funded Partnerships for Enhanced Engagement in Research (PEER) Program to understand the policy utility of PEER grants. ● I am on the PEER External Evaluation Team conducting a long-term evaluation of PEER. I/we work for a research team called ERIE, which is a consortium of staff from the University of Notre Dame, AidData at College of William and Mary, Mathematica Policy Research Center, and CEGA at University of California Berkeley. ● Our evaluation team is conducting interviews with key stakeholders, such as yourself. As a PEER [insert TYPE of stakeholder USAID Buy-in partner, USAID/DC Unit participating partner, other] ● Information you provide will be extremely helpful in providing feedback on your experience participating in PEER and suggestions on how to improve the program. ● During our/my conversation with you, we/I will take notes and will also use a voice recorder. No one outside of research team can see the written notes and hear the conversation. ● We will do everything in our power to protect the confidentiality of your responses. Your answers will be grouped with the answers of other people and your name will not be used in the report. ● The interview will run for about 45 minutes. ● If you agree, we will proceed for the interview. 1. Interview Guide: USAID Washington and Mission Staff Background Questions [This set of questions are intended to be brief and less probing. Sets context for answering evaluation questions. 5 to 10 minutes.] What is your current position and role within USAID? 1. How long have you been with USAID and in your current position? 2. What relationship do you have/have you had with the PEER program? [Probe: how are they involved with PEER, are they the Point of Contact (POC) for the PEER buy-in? Do they work with a Mission/OU (operating unit) that has PEER buy-in?] 3. OU Only: How would you describe the PEER project, including how it relates to your Mission/OU Development objectives? Evaluation Questions: Program Design and Implementation 4. Have you been involved with any other projects/program similar to PEER, if yes, which ones? USAID.GOV PEER EX-POST EVALUATION | 82 5. In your opinion, what are PEER’s greatest strengths? 6. What are the biggest challenges you have faced specifically with PEER? Evaluation Question: PEER Program Results 7. OU Only: In what ways, if any, has the PEER Program contributed to your unit’s overall goals and objectives? 8. OU Only: What information/products/outputs from PEER were shared with you? What information/products/outputs from PEER have you used? In what context or for what purposes have you used these scientific products? How have these findings help your unit, if at all? 9. OU Only: What information/products/outputs from PEER have you shared and with whom (NGO, other unit, other)? 10. Please share any particular strengths and challenges with regards to communication or information sharing of PEER products/activities [Probe: both with Washington and field-based stakeholders.] 11. Are you familiar with any outcomes associated with PEER knowledge sharing activities? [Probe: examples such as program or PI-led workshops or presentations.] If yes, which appear to be most effective in increasing awareness about their products? 12. What are some of the best examples of sharing PEER research? 13. What are some examples of ways in which PIs share products/outcomes that could be improved? 14. What examples, if any, can you share about PEER project(s) that have affected programmatic or policy changes? If no changes were affected, why do you think not? Do you anticipate any changes in the future? 15. What are some unintended consequences/outcomes of PEER program (positive/negative)? 16. How have PEER products met or not met your needs in the following areas? o Timing in provision of information o Format of information o Confidence in the information to inform decision-making o Answering evaluation question o Fostering collaboration (for example between science and policy audiences, between the US and host country) 17. Please share examples of how PEER has affected collaboration between scientists in LMIC, including at the level of government, private sector, and/or education and research institutions. [Probe: regional, south-south collaborations and exchanges between PIs, USAID and host country governments] 83 | PEER EX-POST EVALUATION USAID.GOV 18. Are you aware of any of these collaborations that have been expanded upon or that have lasted beyond the life of the PEER grant? If yes, please describe. 19. How did you monitor PEER progress once the research was funded? Evaluation Question: Research Capacity 20. In your opinion/experience, what factors help change research capacity in a developing country? [Probe: In your opinion what is the greatest need in your country/sector?] 21. Are you aware of any examples of how PEER has improved curricula or created new training programs? If yes, please provide examples? 22. What recommendation would you have to increase/improve the outcomes and impacts of PEER for the next five years? Time permitting/Optional questions 23. How might a PI/PEER project better share their findings? What format/process would be most helpful/useful to you. 2. Interview Guide: PEER PIs/Co-PIs Background Questions 1. What is your current position and role within your institution? a. Ask about their rank (tenured, assistant, associate, visiting, research, other) 2. How long have you been at your institution? (as a PEER PI/Co-PI?) 3. How did you learn about PEER? [Probe: who told them about PEER, what mechanism (via email, online, other)?] 4. What were your expectations about what PEER could do for your career and/or research? [Probe: What did you think PEER was about or what it would accomplish?] 5. What research question(s) are/were you trying to answer through PEER funding? 6. Why did you choose this/these research question(s) to explore? [ Probe around whether the PI/Co￾PIs are motivated to conduct this study to solve any development problem in the country or it’s simply an academic research/interest] Evaluation Question 1: Efficiency of Program Implementation 7. Have you been involved with any other projects/programs similar to PEER? If yes, which ones? [Ask for names, nature of research and total value] USAID.GOV PEER EX-POST EVALUATION | 84 8. What are the biggest implementation challenges of projects of this nature? What implementation challenges have you faced specifically with PEER? a. in your own institution? b. in engaging with USAID? c. in your collaboration with your US-based researcher? d. in working with government officials and other policy makers? 9. In your experience with PEER thus far, can you discuss how each of the management entities, if at all, have interacted with you [Probe: what types of interactions, positive/negative; how do you benefit from their leadership/assistance?] a. NAS b. USAID Washington c. USAID Mission d. Other Buy-in funder 10. Please describe changes, if any, in the donor’s administration of the PEER Program--such as in funding or reporting requirements or award stipulations--that have affected your work/outcomes. 11. Concerning the sharing of your PEER products (papers, technology, results, etc.), what role have you played in this? How have you shared these and to whom (assumes they shared products)? 12. Please describe any particular strengths and challenges with regards to your communication with field-based stakeholders/USAID Missions/PIs? Evaluation Question 2: Influence Program/Policy Change? 13. What is, in your opinion, the most valuable, useful, product/outcome of your PEER project? [Probe: types of products produced by PEER, e.g., publication, conference presentation, students degree, thesis/dissertation, policy paper, etc.] 14. What is in your view the anticipated impact of your PEER project? Of PEER in general? 15. Where appropriate, what are some specific examples of how your PEER project(s) has affected programmatic or policy changes? a. Can you share specific documents, innovations, policies, practices, or processes that were influenced by your project’s work? b. If there is no effect, do you anticipate any changes in the future? c. How might PEER improve its program to better target policy change? 85 | PEER EX-POST EVALUATION USAID.GOV 16. What are some consequences/outcomes of PEER of PEER (positive/negative)? What outcomes should be happening next that are not? 17. How would you describe the career trajectory of any students supported using PEER funding as compared to students you have trained or mentored not under a PEER award? Evaluation Question 3: Collaboration 18. How do you communicate with your US-based research partner? How could PEER help you to have better/more effective communication with your partner? [Probe methods; frequency; barriers] 19. What do you think are factors that contribute to a successful partnership with a collaborator from another country? a. What might be some barriers to this collaboration? b. What could PEER do to improve this? 20. Please share examples of how PEER has promoted collaboration between scientists and others, including at the level of government, private sector, and/or education and research institutions. [Probe: regional, south-south collaborations and exchanges between PIs, USAID and host country governments] 21. How has this collaboration affected your work and/or personal life? 22. From this partnership/project, how does this collaboration help you in future collaborations? Expand collaborations with others in your institution? And, outside of your institution, especially US based PEER research partner? Evaluation Question 4: Capacity 23. What types of capacity building efforts have you helped with/participated/provided your host country PI/Co-PI/students? 24. What type of capacity building activities do you think a PEER project needs? 25. How can PEER better support women and gender minorities? Time Permitting/Optional Questions 26. What more would you like to know about PEER? [Intended to help identify other relevant questions for this evaluation, this group might be able to help us.] 27. How might a PI/PEER project better share their findings? What format/process would be most helpful/useful to you? Interview Guide: Host Country Government Officials INTERVIEW QUESTIONS Background Questions USAID.GOV PEER EX-POST EVALUATION | 86 [This set of questions are intended to be brief and less probing. Sets context for answering evaluation questions. 5 to 10 minutes.] 1. What is your current position and role with your organization? 2. How long have you been in your current position? 3. What relationship do you have/have you had with the PEER funded project(s)? (Interviewer: Please provide reference of a specific PI with PEER grant with the name of University) 4. How many PEER projects have you been involved with or were supported in some way by your organization? In what ways? 5. How did you learn about PEER and what were your expectations? [Probe: who told them about PEER, what were their incentives and needs?] 6. What kind of evidence is most likely to influence policy-makers in your country? How would you describe the PEER project and its objectives in making influence in policy making? Evaluation Question 1: Efficiency of Program Implementation 7. Are you familiar with any activities that began under PEER and continued beyond the life of the grants? If yes, what activities? How do you feel these contribute to program effectiveness? Evaluation Question 2: Influence Program/Policy Change 8. What is in your opinion, the most valuable, useful, product/outcome of PEER? a. What types of products are coming out of PEER? b. What outcome should be happening that is not? 9. What are some specific examples of new scientific discoveries, policies, practices, processes, or technological innovations that can be attributed to PEER (cite PEER grant) ? 10. What were the drivers of these discoveries? How have they been used/applied? 11. Do you have any examples of how your PEER project(s) has affected programmatic or policy changes within the country or region? a. If Yes, how? b. If No, why do you think not? Do you anticipate any changes in the future? c. How might PEER improve its program to better target policy change? 12. What are some unintended consequences/outcomes of PEER (positive/negative) [Probe impacts specifically at developing country institutions]? 87 | PEER EX-POST EVALUATION USAID.GOV 13. What information/products/outputs have you shared and with who (within/outside your institutions, with other colleagues, other)? 14. Are you familiar with any outcomes associated with PEER knowledge sharing activities like PI-led workshops or presentations? If yes, please describe them. 15. Has PEER led to any off-shots within your agency? Has your participation in PEER led to your Agency making new funding allocations for things related to PEER or because of PEER? What were the drivers? Evaluation Question 3: Collaboration 16. How has PEER affected cooperation between your operating unit and others broadly within your government? [Probe: specifically around the use of research to solve development challenges.] 17. How has the PEER project led to additional partnerships/collaborative efforts in other areas of your Agency not involved with PEER? Evaluation Question 4: Capacity 18. In your opinion/experience, what factors help change science and innovation capacity in the country? A. C> WHAT ARE THE STRENGTHS IN RESEARCH CAPACITY/SCIENCE, TECHNOLOGY ECOSYSTEM IN YOUR COUNTRY? B. HOW ABOUT THE WEAKNESSES? Time Permitting/Optional Questions 19. How might a PI/PEER project better share their findings? What format/process would be most helpful/useful to you? Interview Guide: University Officials supporting PEER Funded PI [This set of questions are intended to be brief and less probing. Sets context for answering evaluation questions. 5 to 10 minutes.] 1. What is your current position and role with your university or institute? 2. How long have you been in your current position? 3. What relationship do you have/have you had with the PEER project? 4. How many PEER projects have you been involved within? How many total supported by your university or institute? What level of financial support, if any, has the university or institute provided in support of PEER funded research? USAID.GOV PEER EX-POST EVALUATION | 88 5. How did you learn about PEER and what were your expectations? [Probe: who told them about PEER, what made them participate/buy-in, what were their incentives and needs?] 6. How would you describe the PEER project and its objectives? Evaluation Question 1: Efficiency of Program Implementation 7. Have you been involved with any other projects/programs similar to PEER, if yes, which ones? 8. What do you think other similar projects could learn from PEER? What could PEER learn from other similar projects? 9. What are the biggest implementation challenges of projects of this nature? What implementation challenges have you faced specifically with PEER? Evaluation Question 2: Influence Program/Policy Change 10. What is in your opinion, the most valuable, useful, product/outcome of PEER? f. What types of products are coming out of PEER? g. What outcome should be happening that is not? 11. What are some examples of new scientific discoveries or technological innovations that can be attributed to PEER? What were the drivers? How have they been used/applied? 12. Do you have any examples of how your PEER project(s) has affected programmatic or policy changes within the country or region? d. If Yes, how? e. If No, why do you think not? Do you anticipate any changes in the future? f. How might PEER improve its program to better target policy change? 13. What are some unintended consequences/outcomes of PEER (positive/negative) [Probe impacts specifically at developing country institutions]? 14. What information/products/outputs have you shared and with who (within/outside your institution, with other colleagues, other)? 15. Are you familiar with any outcomes associated with PEER knowledge sharing activities like PI-led workshops or presentations? If yes, please describe them. 16. Has PEER led to any off-shots within your institute? Has your participation in PEER led to your institute making new funding allocations for things related to PEER or because of PEER? What were the drivers? 89 | PEER EX-POST EVALUATION USAID.GOV Evaluation Question 3: Collaboration 17. How has PEER affected cooperation between your institute and others nationally or regionally? [Probe: specifically around the use of research to solve development challenges.] 18. How has the PEER project led to additional partnerships/collaborative efforts in other areas of your institute not involved with PEER? 19. Are you aware of any of these collaborations that have been expanded upon or that have lasted beyond the life of the PEER grant? If yes, please describe. Evaluation Question 4: Capacity 20. In your opinion/experience, what factors help change research capacity in your country? [Probe: how do they define change in research capacity?] 21. Are you aware of any examples of how PEER has improved curricula or created new training programs? If yes, please provide examples? 22. How can PEER better support women and minority researchers? Time Permitting/Optional Questions 23. How might a PI/PEER project better share their findings? What format/process would be most helpful/useful to you? 24. For you, what was the best part of PEER? Most challenging? USAID.GOV PEER EX-POST EVALUATION | 90 C. SUPPLEMENTARY TABLES Note: *** p<0.01, ** p<0.05, * p<0.1 Source: Bibliometric data. Controls included: region, years since application, reviewer score, gender of applicant, field of research, and cohort fixed effects † baseline value of outcome ‡ count of publications in time period TABLE 6A: GENERATION OF KNOWLEDGE, BIBLIOMETRIC DATA (CUT-OFF = 1 YEAR POST APPLICATION) VARIABLE COEFFICIENT: RAW DATA N COEFFICIENT: EB N COEFFICIENT: MDM N Count of publications, after application† 1.08 842 0.94 429 -0.78 129 Average Altmetric score, after application†‡ 6.83 842 6.31 429 3.89 129 Count of Mendeley readers, after application†‡ 31.81 842 30.25 429 -7.39 129 Count of Tweets, after application†‡ 3.98 842 7.50 429 5.22 129 Count of Patents, after application†‡ -0.02 842 -0.03** 429 -0.08* 129 Count of Policies, after application†‡ -0.01 842 0.04 429 0.08 129 Count of Citations, after application†‡ 9.34 842 12.67 429 9.89 129 TABLE 6B: GENERATION OF KNOWLEDGE, BIBLIOMETRIC DATA (CUT-OFF = 2 YEARS POST APPLICATION) VARIABLE COEFFICIENT: RAW DATA N COEFFICIENT: EB N COEFFICIENT: MDM N Count of publications, after application† 0.93 833 0.10 431 -2.28 129 Average Altmetric score, after application†‡ 7.06 833 5.40 431 3.94 129 Count of Mendeley readers, after application†‡ 41.23** 833 34.16 431 10.29 129 Count of Tweets, after application†‡ 15.27 833 14.65 431 18.81 129 91 | PEER EX-POST EVALUATION USAID.GOV Note: *** p<0.01, ** p<0.05, * p<0.1 Source: Bibliometric data. Controls included: region, years since application, reviewer score, gender of applicant, field of research, and cohort fixed effects † baseline value of outcome ‡ count of publications in time period Note: *** p<0.01, ** p<0.05, * p<0.1 Source: Bibliometric data. Controls included: region, years since application, reviewer score, gender of applicant, field of research, and cohort fixed effects † baseline value of outcome ‡ count of publications in time period Count of Patents, after application†‡ -0.01 833 -0.02** 431 -0.06* 129 Count of Policies, after application†‡ 0.03 833 0.05 431 0.08 129 Count of Citations, after application†‡ 19.52* 833 18.77 431 23.88 129 TABLE 6C: GENERATION OF KNOWLEDGE, BIBLIOMETRIC DATA (CUT-OFF = 3 YEARS POST APPLICATION) VARIABLE COEFFICIENT: RAW DATA N COEFFICIENT: EB N COEFFICIENT: MDM N Count of publications, after application† -0.32 717 -0.51 399 -2.44 116 Average Altmetric score, after application†‡ 0.85 717 -1.31 399 0.50 116 Count of Mendeley readers, after application†‡ 29.10** 717 18.29 399 11.63 116 Count of Tweets, after application†‡ 9.01 717 4.61 399 -0.79 116 Count of Policies, after application†‡ 0.03 717 0.03 399 0.03 116 Count of Citations, after application†‡ 10.80 717 6.28 399 3.12 116 USAID.GOV PEER EX-POST EVALUATION | 92 D. INFORMATION SOURCES Institutions where PIs and other stakeholders were interviewed either by field visits or remotely: COUNTRY/PI’S INSTITUTION STATUS GOVT OFFICIAL/ POLICY MAKER UNIVERSITY OFFICIAL USAID UZBEKISTAN √ International Water Management Institute √ √ √ International Center for Biosaline Agriculture √ √ √ International Water Management Institute (IWMI) √ √ NA AFGHANISTAN √ AZMA Technical and Vocational Institute √ √ √ Organization of Skill Development and Social Services (OSDSS) √ √ √ COLOMBIA √ Universidad de Antioquia √ √ √ Universidad EAFIT √ NA √ Universidad del Norte √ √ √ Universidad Tecnológica de Pereira √ √ √ 93 | PEER EX-POST EVALUATION USAID.GOV Universidad Nacional de Colombia √ √ √ SOUTH AFRICA √ University of Pretoria √ NA √ University of KwaZulu-Natal √ √ NA University of Witwatersrand √ √ NA University of Fort Hare √ NA NA University of the Witwatersrand √ √ √ University of Pretoria √ NA √ University of KwaZulu-Natal √ √ √ Dept. of Sc and Tech (co-funder) √ MOZAMBIQUE Beira Operations Research Center √ NA NA MOROCCO √ International University of Rabat √ √ √ International University of Rabat √ NA √ USAID.GOV PEER EX-POST EVALUATION | 94 Ecole Nationale de l'Industrie Minérale √ NA NA Ecole Nationale de l'Industrie Minérale √ NA √ NA= Not Available FOLLOW-UP PHONE INTERVIEWS PIS/COUNTRY NO. OF PI Kenya 2 Lebanon 1 Uganda 1 Armenia 1 Sri Lanka 1 Brazil 1 Philippines 1 Ghana 1 95 | PEER EX-POST EVALUATION USAID.GOV E. SUMMARY INFORMATION ON EVALUATION TEAM MEMBERS Lila Khatiwada: Dr. Khatiwada has over 25 years of work experience in the development field as a program implementer and researcher. He has implemented research work in Burkina Faso, Ethiopia, Ghana, Indonesia, Kenya, Mozambique, Namibia, Nepal, Tanzania and Uganda. He has developed evaluation designs (mostly using mixed method), prepared sample frame, trained enumerators, developed data collection instruments, supervised data collection work, analyzed data and prepared reports. Further, using participatory approach, Dr. Khatiwada has designed and implemented community health, gender development, agriculture, forestry, microfinance and enterprise development, and institutional development projects in Nepal for ten years. Currently, he is leading a project to evaluate the impact of a UNICEF intervention to reduce child labor in a US Dept. of Labor (USDOL) funded project in Nepal where he is using mixed method in impact evaluation. In Malawi, he evaluated a long-term impact of USAID funded WALA project in food security where he assisted in implementing qualitative part of study. In Ghana, he designed and oversaw qualitative and quantitative data collection work for MCC in a WASH evaluation. He is proficient in using qualitative software, ATLAS.ti. Dr. Khatiwada holds a Ph.D. in Rural Sociology and a Graduate Certificate in Geographic Information System (GIS) from the University of Missouri and an MS in Rural Sociology and Environment Science from the University of the Philippines. He has published his research works in American Behavioral Scientists, Journal of Regional Analysis and Planning, Maternal and Child Health, Journal of International Tourism Research, World Development Perspectives, Africa Today, Eurasian Geography and Economics journals and has reviewed for the World Development, Journal of International Tourism Research, Experimental Agriculture, and Community Informatics. In this evaluation, Dr. Khatiwada led the qualitative aspect of evaluation, designed the evaluation, visited Colombia and assisted to conduct remote interviews in South Africa, Mozambique, and Afghanistan. He was also involved in drafting the report. Paul Perrin: Dr. Perrin is the Director of Monitoring and Evaluation at the Pulte Institute for Global Development and a Concurrent Associate Professor of the Practice in the Keough School of Global Affairs. Dr. Perrin’s work has spanned academic, government, and non-governmental settings and has largely focused on supporting the conceptualization, design, implementation, analysis, and reporting on monitoring and evaluation and implementation science activities in resource-challenged environments. Currently, he is leading the Haiti Reads project, where he leads the research team to implement EGRA literacy assessments in French and Creole of students in over 250 schools in Haiti. Prior to joining Notre Dame, Dr. Perrin served as the Director for Monitoring, Evaluation, Accountability, and Learning (MEAL) at Catholic Relief Services (CRS). In this role, Dr. Perrin oversaw the development and implementation of a global MEAL strategy, which has led to the incorporation of evidence-based learning into CRS programming across the world. Dr. Perrin has also worked as Senior Technical Advisor for Health Research and Evaluation at CRS, as an Associate in the Johns Hopkins Center for Humanitarian Health, and an Information Officer at USAID’s Office of U.S. Foreign Disaster Assistance. Dr. Perrin completed his PhD in International Health at the Johns Hopkins University Bloomberg School of Public Health, with a certificate in Humanitarian Assistance. Paul also received a Masters in Public Health and a Bachelors of Arts in Linguistics from Brigham Young University. Dr. USAID.GOV PEER EX-POST EVALUATION | 96 Perrin is proficient in French and will be responsible for providing his expertise in study design, examining project outputs from policy perspective, data analysis, report writing and dissemination. In this evaluation, Dr. Perrin was involved in a qualitative part of study and visited Morocco and Uzbekistan for on-site interviews of PEER stakeholders. He provided inputs during the drafting of the report. Dr. Perrin also reviewed the report for quality. Danice Brown Guzman: Mrs. Guzman has experience as a program manager for education evaluations and in nutrition. Mrs. Guzman has worked on an impact evaluation for a community drinking water project in Burkina Faso, MCC’s WASH project in Ghana and coordinated the gathering of baseline data on the USDA Benin Food for Education Program. Mrs. Guzman has also served as the program manager for designing new USAID evaluation tools, including the USAID Expanding the Reach of Impact Evaluation grant. Her education has focused on quantitative evaluation methods such as randomized controlled trials and quasi-experimental methods such as regression discontinuity, difference in difference, synthetic controls, and propensity score matching, including software skills in STATA and Gephi (social network analysis). Mrs. Guzman has a Master’s of Public Policy from the Sanford School of Public Policy at Duke University and a Certificate in International Development from the Duke Center for International Development. In this evaluation, Mrs. Guzman led the quantitative part of evaluation, designed the survey, analyzed the data and prepared the report. In addition, she also visited Colombia to meet and interview stakeholders.