January 31, 2017 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Cultural Practice, LLC and The QED Group, LLC. ENHANCED ENGAGEMENT IN RESEARCH: Mid-Term Evaluation of the Partnerships for Enhanced Engagement in Research (PEER) Program Contract No. AID-OAA-I-15-00016 Order No. AID-OAA-TO-16-00012 COVER PHOTO- A student from the research team at Indonesia International Institute for Life Sciences (i3L) working on their PEER project “Developing a bio-economy in Indonesia: identification of novel microorganisms and microbial enzymes from Indonesian peatland and buffaloes to improve bioconversion of oil palm residues”. Photo taken by Evaluation Science and Technology Advisor Dr. Carolyn Fonseca, 2016 (Jakarta, Indonesia). MID-TERM EVALUATION OF THE PARTNERSHIPS FOR ENHANCED ENGAGEMENT IN RESEARCH (PEER) PROGRAM January 31, 2017 Contract No. AID-OAA-I-15-00016 Order No. AID-OAA-TO-16-00012 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i TABLE OF CONTENTS ENHANCED ENGAGEMENT IN RESEARCH: .........................................................................................................1 Mid-Term Evaluation of the Partnerships for Enhanced Engagement in Research (PEER) Program.............1 Acknowledgments............................................................................................................................................................. ii Acronyms............................................................................................................................................................................ iii Executive Summary ........................................................................................................................................................... v Evaluation Purpose & Evaluation Questions................................................................................................................1 Evaluation Purpose....................................................................................................................................................1 Evaluation Questions ................................................................................................................................................1 Project Background......................................................................................................................................................................2 Evaluation Methods & Limitations............................................................................................................................3 Methods...........................................................................................................................................................................................3 Limitations ......................................................................................................................................................................................6 Findings, Conclusions, and Recommendations...........................................................................................................8 Findings.....................................................................................................................................................................8 Conclusions .............................................................................................................................................................47 Recommendations...................................................................................................................................................57 Evaluation Question 1................................................................................................................................................................57 Evaluation Question 2................................................................................................................................................................58 Evaluation Question 3................................................................................................................................................................58 Evaluation Question 4................................................................................................................................................................59 References.........................................................................................................................................................................60 Annexes.............................................................................................................................................................................61 Annex I: Evaluation Statement Of Work .................................................................................................................62 Annex II: Evaluation Methods and Limitations .......................................................................................................78 Annex Iii: Data Collection Instruments....................................................................................................................86 Annex Iv: Sources Of Information .........................................................................................................................131 Annex V: Policy Change Literature Review............................................................................................................143 Annex VI Descriptive Statistics.............................................................................................................................150 i TABLES Table 1: Data Collection Methods, Sources and Analytical Strategies..........................................................................4 Table 2. Data Collection Activities: Planned vs. Actual ...............................................................................................5 Table 3: Comparison Program Objectives ....................................................................................................................9 Table 4: PEER Program Comparison Table ..................................................................................................................15 Table 5: Perceptions of PEER by Principal Investigators/Co-PI and U.S. Partner Scientists ........................................17 Table 6:. New Technology and Tools Produced Under PEER Per Survey Results by Sex (PIs only). ........................22 Table 7: PEER products for policy recommendation/analysis by gender (PIs/Co-PIs survey responses)....................25 Table 8: Summary list of PEER projects connected to policy change..........................................................................26 Table 9. Summary list of PEER projects with potential (i.e. likelihood to affect policy in the future, however they are too early in project or process) for policy change. ..............................................................................................28 Table 10. US Partner Scientists Survey Responses – PI Partner Selection .................................................................32 Table 11: PEER PI/Co-PI Survey Responses on How they First Met US Partner .......................................................33 Table 12: PI/Co-PI Expectations for PEER Grant (PI/Co-PI Survey Data) ..................................................................34 Table 13: PEER PI/Co-PIs Responses on Who They Interact With (Q42).................................................................37 Table 14: Changes in Relationships Due to PEER .......................................................................................................39 Table 15:Prospects for Continued Collaboration (PEER PI/Co-PIs survey – response frequency by gender and country).......................................................................................................................................................................41 Table 16: Frequencies of responses types of changes requested by US Partners.......................................................41 Table 17:Table 16. Student Engagement in PEER by sex and year (PEER Monitoring Data) .....................................43 Table 18: able 17. PI/Co-PI Perceptions of PEER Benefits .........................................................................................43 Table 19: Survey data on Perceptions of PEER Benefits for PIs/Co-PIs and US Partner Scientists.............................47 Table 20:Contribution of PEER Grant (PI/Co-PIs responses).....................................................................................55 FIGURES Figure 1. Survey response rates per USG Agency for both PEER PI/Co-PIs and US Partner Scientists.......................7 Figure 2: Feed the Future Innovation Lab ...................................................................................................................10 Figure 3: United Kingdom Official Development Assistance (ODA)- funded Newton Fund ........................................12 Figure 4: WOTRO Science for Global Development.....................................................................................................13 Figure 5: PEER PI Response Regarding PEER Products ...............................................................................................21 Figure 6. i3L Lab Facilities (Indonesia). ........................................................................................................................22 Figure 7. US Partner Scientists Survey Responses – PI Partner Selection (sex disaggregated)...................................33 Figure 8: PEER PI/Co-PI Interactions with Other Stakeholders..................................................................................35 Figure 9: US Partner Scientists: Pre-PEER Interactions with Other Stakeholders......................................................36 Figure 10. PI/Co-PI Pre-PEER Interaction with Government Agencies by Sector......................................................38 Figure 11: PI/Co-PI Pre-PEER Interactions By Grant Value (PEER PI/Co-PI Survey Data) ........................................39 Figure 12: Percentage of Male and Female Students Participating in PEER (Survey data)...........................................44 Figure 13: Survey reported student participation in PEER (PI data) ...........................................................................45 Figure 14: Survey reported student participation in PEER (US partner scientist........................................................45 Figure 15. Easton, David (1965). A Systems Analysis of Political Life, New York, S.32............................................145 Figure 16. Policy process model by Meier (1991). ....................................................................................................145 Figure 17. ACF Model - Sabatier and Weible (2014). ...............................................................................................146 Figure 18. The IAD model by Ostrom (2005)...........................................................................................................146 ii ACKNOWLEDGMENTS The evaluation team would like to extend their heartfelt thanks to the PIs and Co-PIs, and their team members in Kenya and Indonesia who gave so generously of their time, thoughts, and enthusiasm to discuss their research and experience with the PEER program. We are similarly grateful to the 178 PIs and Co-PIs around the world and the 111 U.S. partner scientists who responded to our survey. The information they provided was invaluable to learning about the program and how it is preforming. We are thankful to Robert Mbeche and Daisy Chepkoech in Nairobi, Kenya, and Hanifah Siregar and Asri Puji Lestari in Jakarta, Indonesia for the excellent job they did in reaching out to PEER grantees, setting up meetings, translating, and taking notes. They contributed to making the process seamless and to ensuring that the data we collected was accurate and complete. We would also like to thank the USAID Mission staff, host-country government officials, and staff at partner USG agencies who met with us to discuss their perspectives on the program, as well as the managers of comparable programs, who shared the elements of their programs, to allow the team to better contextualize the strengths and challenges of PEER. Finally, we would like to thank the NAS staff who treated our requests with the same alacrity and thoroughness that the PI’s said they have come to expect from NAS. To our colleagues, at the USAID Global Development Lab, we reserve a special debt of gratitude for their partnership in the process and for pushing us to really probe on issues of great import to the Lab in their process to make PEER a model program. They created a learning and sharing environment that contributed enormously to the rigor of the evaluation methods and analysis. EVALUATION TEAM MEMBERS Eric Benschoter, Team Leader Carolyn Fonseca, Evaluation Science and Technology Advisor Deborah A. Caro, Evaluation Health Advisor Caitlin Nordehn, Data Analyst Ingrid Orvedal, Data Analyst iii ACRONYMS ADS Automated Directives System ARF Food & Business Applied Research Fund ASADI African Science Academy Development Initiative BEIS United Kingdom Department of Business, Energy and Department of Industrial Strategy BFCI Baby friendly community initiative BFS Bureau for Food Security CDR Center for Development Research CDS United Kingdom Collaborative on Development Sciences CONYTEC National Science Council of Peru CRSP Collaborative Research Support Program DDL The Development Data Library DEC Development Experience Clearinghouse DoD United States Department of Defense E2A Evidence to Action E2P Evidence to Policy FCR Findings, conclusions and recommendations FGD Focus Group Discussions FIC Fogarty International Center FY Fiscal Year HESN Higher Education Solutions Network KMC Kangaroo Mother Care KU Kenyatta University LAB USAID Global Development Lab MSC Most Significant Change NAS National Academies of Science NASA National Aeronautics and Space Administration NCI National Cancer Institute NICHD National Institute for Child Health and Human Development NIH National Institutes of Health NSF National Science Foundation NWO-WOTRO Netherlands Organization for Scientific Research- Science for Global Development programs ODA United Kingdom Official Development Assistance ONS National Drought Observatory PEER Partnerships for Enhanced Engagement in Research PI Principle Investigator PMP Performance Management Plan SFEWRA Sustainable Fisheries for Enhanced Water Resources in Armenia iv SHERA Sustainable Higher Education Research Alliances SOP Standard Operating procedure SOW Statement of Work TDY Temporary Duty USAID U.S. Agency for International Development USDA United States Department of Agriculture USF University of South Florida USG United States Government USGS United States Geological Survey v EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS This mid-term evaluation of the Partnerships for Enhanced Engagement in Research (PEER)is crucial to inform the next five years of the program. The evaluation will contribute to the Global Development Lab’s (Lab) objective to document program effectiveness to better inform programming decisions through analysis and action-oriented recommendations for improvements in: (1) management and implementation; (2) stakeholder engagement; (3) results; (4) partnerships; and (5) strategies to promote PEER research application. The primary audience for this evaluation will be PEER staff and USAID senior management, National Academy of Sciences and Engineering (NAS) managers, United States Government (USG) Partners, and USAID Missions. The evaluation responded to the following primary level evaluation questions: 1. To what extent is the PEER program being implemented efficiently? 2. To what extent has PEER helped fill evidence gaps which influence policy/program change? 3. To what extent has PEER promoted collaborations between the US scientific community, local research institutions, host country governments, and USAID Missions and Embassy staff, and private sector partners? 4. To what extent has PEER strengthened research capacity in developing countries? The evaluation also responded to many sub-questions broken into two categories addressing: (1) implementation performance and (2) program outcomes, which were incorporated into the data collection instruments and were used to answer the primary evaluation questions. PROJECT BACKGROUND The PEER program, implemented by NAS and managed by the USAID Center for Development Research (CDR) in the Lab, was developed to support research projects jointly conducted by developing country scientists and researchers supported in the U.S. by federal science agencies. With a funding ceiling of $100 million, PEER operates over a period of performance of ten years (2011-20121). Between 2011 and 2016, PEER supported 250 grants in 50 countries worldwide through Lab core and buy-in funding, primarily from USAID Missions, and with some USG partners, like the National Cancer Institute and Office of Naval Research to sponsor specific PEER awards. The target sectors are: biodiversity, health, agriculture, environment, water, disaster mitigation, climate, education, food security, and energy. USG Partner organizations (NASA, NIH NOAA, NSF, USDA, USGS, and Smithsonian Institute) support awards to U.S. partner scientists who serve as mentors to PEER researchers. To-date, PEER program has released seven funding cycles and supported four annual regional meetings for participating PEER Principal Investigators (PIs) in Bangkok, Arusha, Lima, and Amman, with a focus on. capacity building and networking for PIs with NAS and USAID Mission and Lab staff. NAS hosts two financial management trainings per year for administrative staff at institutions with PEER grants. vi EVALUATION QUESTIONS, DESIGN, METHODS AND LIMITATIONS METHODS The PEER midterm evaluation employed a mixed method design integrating quantitative and qualitative data collection and analysis. The evaluation team analyzed program progress and performance on the three key dimensions of the program in line with the evaluation questions: The evaluation team conducted interviews with key staff from USAID, NAS, USG partner institutions, private sector partners and US-based partner researchers. The team made two country visits to Indonesia and Kenya, the two countries with the most number of PEER grants, together making up 27 percent of the PEER portfolio. The evaluation team conducted 91 key informant interviews, surveyed 315 P1s, Co-PIs and US partner scientists out of 426 contacted, conducted four focus groups with 45 student and women PI participants, four site visits and held two workshops to implement the Most Significant Change (MSC) technique with 30 PEER stakeholders in Kenya and Indonesia. LIMITATIONS Data collection activities deviated slightly from the work plan. While the data collection exceeded targets for interviews (97 actual compared to 83 planned) particularly with PEER funded PIs. The evaluation team interviewed fewer US partner scientists than planned (5 of 11 planned). The team compensated by conducting an extensive survey of the entire population of partner scientists, which filled many of the gaps. In addition, the team conducted interviews with nine of 11 planned USAID Missions, three of four planned comparator programs. The large survey response from PIs and US partner scientists lends a very high degree of confidence in data collection, however the evaluation team does recognize a selection bias and therefore some degree of limitation in terms of how representative our sample is and the degree to which we can generalize across the program, since field work was conducted in the two largest PEER programs of the 49 countries, as specified in the evaluation design. FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1 – TO WHAT EXTENT IS THE PEER PROGRAM BEING IMPLEMENTED EFFICIENTLY? FINDINGS AND CONCLUSIONS  In comparison to other programs, the sole control over the grant by the PI is a strong advantage of PEER. Other programs offer features that are more supportive of face to face collaboration between U.S. partner scientists and host country PIs, support to host-country research insitutions, and convenings that bring together researchers with other stakeholders, including polity makers.  PEER Researchers are satisfied with several elements of the program; a) networking; b) capacity building opportunities; c) NAS support; and d) the grant cycle and proposal process. US partner scientists were measurably less satisfied than PEER PIs.  PIs gave PEER implementer NAS high praise for responsiveness, engagement, guidance, and flexibility.  PIs desire improvements in NAS communications in a few key areas: (1) more facetime between PEER PId and US partner scientists; (2) more frequent updates and access to progress reports by USG partner agencies; (3) more frequent and focused engagement with USAID Missions to leverage vii their support and enhance their understanding of PEER; and more flexibility in choosing mentors beyond those funded by USG agencies. No. RECOMMENDATIONS PERTAINING TO Q.1 Responsibility 1.1 Expand the type and number of partnership organizations to the private sector (e.g., General Electric) and foundations (e.g., Bill and Melinda. Gates Foundation) to offer PIs a wider choice of partners among U.S. Partner Scientists) USAID 1.2 PEER funding should include a travel budget for the US Partner Scientists, when needed, to increase face to face interactions between PIs and US. Partner Scientists. USAID and NAS 1.3 Augment the size and flexibility of the grants by increasing the ceiling (to $500k), raise limit of purchases of equipment, and extend the life of the project to five years, or allow for more options for no-cost and costed extensions USAID 1.4 To increase the number of women and you scientists, increase emphasis on mentoring targeted at these groups and address barriers they face during the proposal period and for travel. PEER may consider funding cycles just for women and young scientists. USAID and NAS 1.5 Provide a 3-6 post-award planning period to address issues around permits, infrastructure, equipment, procurement, staffing, and other factors with the potential to delay research. USAID and NAS 1.7 To increase engagement of USAID Missions with PEER researchers:  Prioritize and standardize functions of TDY Supervision trips for NAS and USAID. NAS should focus on supervision of PEER research, while USAID Lab staff focus on improving connectivity between the Lab and Mission staff and to better align PEER research awards with Mission programs and policy dialogue.  Send staff from Missions with low PEER engagement to Missions with high PEER engagement so they can see the benefits first hand. USAID 1.8 PEER should continue an analysis beyond the parameters of the midterm evaluation of the rich data collected by the evaluation team, including comparison analysis by agency; testing of scientific productivity models and determining significant variables of PEER productivity; and to create and publish peer-reviewed journal articles with the data/findings USAID 1.9 Create a special track for technology projects with high risk of failure or high risk of delays (such as those with unique equipment purchasing needs) but high potential for substantial innovation. NAS 1.10 Use PEER to create “Research Assessments” of the countries they target for the benefit of the Mission, but also to ensure a more cohesive match between Mission needs and priorities, country focus areas, and researcher interests. By providing a profile to each Mission of the ways in which research and science can improve their development outcomes, and in which specific areas, would increase PEER participation. NAS viii EVALUATION QUESTION 2 – TO WHAT EXTENT HAS PEER HELPED FILL EVIDENCE GAPS WHICH INFLUENCE POLICY/PROGRAM CHANGE? FINDINGS  34% of PEER PIs/Co-PIs responding to the survey stated they believe their research products are being used to change policy at the national level, with a greater number of these occurring from research conducted in earlier rather than later research cycles. According to the literature, the influence of research on policy change usually takes place over a timeframe spanning 10-12 years.1  Less than a third of PEER PIs/Co-PIs said they shared research products with government agencies, yet 56% believe products are being used to improve decision making.  PEER researchers speak widely to the scientific community as 58% presented at International meetings and 55% at national meetings. In addition, 49% spoke at stakeholder meetings. Less than a third (27%) have published in peer reviewed journals, although another36% have articles under review.  The focus of PEER PIs is often not on policy change because of the length of research funding (projects are too short); funding levels (not enough in the budget to engage decision-makers); or insufficient training to communicate their findings to non-technical audiences. In addition, ambiguity exists around what policy change means, and how different stakeholders define policy. RECOMMENDATIONS PERTAINING TO Q.2 Responsibility 2.1 To increase policy change outcomes, review grants and categorize by: 1) level of connectivity to government and 2) whether the projects have characteristics or expected results with the potential application for policy change? NAS 2.2 Consider two options for redirecting funds towards policy and program-oriented research: (1) increase funding specifically for policy-oriented research; or (2) hold special calls to support policy- or program-directed research, perhaps as a pilot, to explore the significance of certain types of research for informing policy and program design and implementation. USAID/LAB, USAID Missions, and NAS 2.3 Recognize that program or policy change is not always the most effective and efficient way to achieve, or measure research results. Consider conducting joint assessments with Missions to determine how research can best serve the country’s development needs, which can form the basis for calls supported by PEER and the Missions. NAS and USAID/Missions 2.4 Include “number of articles submitted” and “number of articles submitted and rejected” as part of the metrics for Scientific Productivity and PEER Outcomes. Create incentives and opportunities to capture evaluation data post-project periods. NAS 1 See Annex V for a brief literature review of theories on the process for getting research into policy ad practice. ix 2.5 The capacity of PEER funded scientists to communicate and advocate effectively for program or policy change is inconsistent and inadequate. PEER needs to consider focusing resources on building capacity through its grants and other support structures to develop this skillset or assisting them with finding advocacy partners similarly to the way PEER assists PIs to find US science partners. NAS EVALUATION QUESTION 3 – TO WHAT EXTENT HAS PEER PROMOTED COLLABORATION BETWEEN THE US SCIENTIFIC COMMUNITY, LOCAL RESEARCH AND USAID MISSION, EMBASSY STAFF, AND PRIVATE SECTOR PARTNERS? FINDINGS  PIs and US Partner Scientists place a high value placed on their partnership as a driver for improving scientific output, strengthening professional relationships, and advancing careers. One constraint, however, is lack of travel funds for some US Partner Scientists to travel to the PI’s country for more face to face encounters at research sites.  One major constraint is the requirement that PIs partner with scientists with a USG funded grant. This requirement is challenging in some fields where there is no a good fit between the types of research funded by the US counterpart organization and USAID Missions’ local research and programmatic needs. This was particularly true for PEER maternal and child health projects, and some climate change projects.  Growth of professional networks was the number one reason for why most PEER PIs competed for the grant. Both PIs and US partner scientists stated that their professional networks have been strengthened by participation in the program. Evaluation findings demonstrate that successful awardees are already very well connected, and those that are most widely connected in their fields are awarded the largest amount of funding.  The findings indicate PEER is not a program for young and relatively inexperienced researchers, except insofar as they are part of PIs’ research teams. Men and women are also not participating in PEER in equal proportions. Approximately 70% of PEER PI survey respondents are male and 30% female. Compared to global ratios, survey responses indicate that sex ratios in PEER are similar to UNESCO numbers of 27% female researchers globally2. No. RECOMMENDATIONS PERTAINING TO Q. 3 Responsibility 3.1 Facilitate the broader dissemination of research findings by convening PEER grantees, the private sector, government officials, and civil society partners together to network and share findings and policy and program challenges. This could include a roundtable series to discuss the activities and areas for synergy to broaden Mission and host country government engagement. NAS and USAID Missions 2 http://uis.unesco.org/sites/default/files/documents/fs34-women-in-science-2015-en.pdf. The Dutch comparator program WOTOR offers some strategies for expanding the number of women among PEER researchers. x 3.2 Create country-wide and/or field-specific PEER conferences hosted or convened by Missions as outreach events with government entities and as opportunities for scientific sharing in support of communities of practice. NAS and USAID Missions 3.3 Create a virtual space for PIs such as a PEER “YouTube” channel to share with both their PEER colleagues but also with the public. Request every PI create one instructional video per project, with guidance on structure and content to support capacity building and teaching. NAS EVALUATION QUESTION 4 – TO WHAT EXTENT HAS PEER STRENGTHENED CAPACITY IN DEVELOPING COUNTRIES? FINDINGs  PIs and students appear to mutually benefit from student participation in PEER., which contributes to improvements in research, learning, and teaching skills.  One primary skill set that PIs and Co-PIs identified as needing improvement was in writing capacity for journal publication.  Another important skill set that was discussed consistently in interviews with all stakeholders was the capacity to engage at the right time and with the right set of actors to influence program or policy decisions with the scientific evidence generated.  The perception among most PIs and U.S. partner scientists was that having access to skilled students is an important factor in their productivity signals that student participation in PEER research is an essential component of the PEER research model. While, there is evidence that PEER is benefiting from student participation in research, there is limited data collected on factors contributing to building student research and analytical capacity. RECOMMENDATIONS PERTAINING TO Q.4 Responsibility 4.1 Provide training in journal article writing by leveraging US Partners’ expertise at PEER workshops or conferences. NAS 4.2 Make draft peer-reviewed journal articles from students a deliverable through student swaps. Send PEER PI students to the US partner scientist’s Lab (or vice-versa) for a semester (or more) and increase the likelihood of scientific paper productivity. NAS 4.3 Enhance data collection about students to better capture PEER impact and benefit. 1 EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE The PEER Program has been reauthorized for five years of additional funding through 2021. This mid￾term evaluation is crucial to help inform the next five years of PEER, as well as follow-on or ‘spin off’ programming that supports Intermediate Result-2 and Intermediate Result-3 of the Science Objective Results Framework provided in Annex 1 as part of the Scope of Work. The evaluation will contribute to the Global Development Lab’s objective to document program effectiveness to better inform programming decisions. To that end, the evaluation will accomplish the following objectives defined by the Lab: 1. Evaluate the implementation performance of the PEER program, namely operations procedures, operational efficiency, and management structure. 2. Assess how the PEER program is perceived and valued by key stakeholders such as PEER PIs, Universities, USAID Missions, host country government ministries and local NGOs. 3. Provide evidence of the extent to which the PEER program is working and producing real change, specifically whether the program is making progress towards achieving the targets in the results frameworks for both the PEER program and the LAB’s Science Objectives. 4. Provide action-oriented, practical, and specific recommendations to improve the implementation of the PEER program and inform decisions about current and future programming. 5. Identify and detail strategies to promote public and private sector adoption of PEER-supported research. The funding for PEER is an extensive web of buy-in funding from USAID Missions and other Operating Units, as well as core Lab funds and, in some years, other supplemental sources outside of USAID, including US science agencies and others. As an innovator and an incubator bringing together networks of partners, the Lab annually weighs its funding decisions for PEER and other programs against its priorities for new and existing programs, as well as what other resources can be leveraged from partners. This evaluation can inform strategic thinking on what programs to support. The primary audience for this evaluation will be PEER staff and senior management within the Lab, NAS managers, and to some degree USAID Mission funding partners. During this evaluation, nearly all other types of stakeholders expressed great interest in receiving the final evaluation for their own knowledge and decision making. These include: PEER-funded PIs, US government partner agencies, US partner scientists, and other collaborators from the private sector, other donors, and NGOs. EVALUATION QUESTIONS The evaluation addressed the following primary level evaluation questions: 1. To what extent is the PEER program being implemented efficiently? 2 2. To what extent has PEER helped fill evidence gaps which influence policy/program change? 3. To what extent has PEER promoted collaborations between the US scientific community, local research institutions, host country governments, and USAID Missions and Embassy staff, and private sector partners? 4. To what extent has PEER strengthened research capacity in developing countries? In addition to these questions, a large number of sub-questions were identified in the scope of work for this evaluation and broken into two categories addressing: (1) implementation performance and (2) program outcomes. These sub-questions were incorporated into the data collection instruments and were used to answer the primary evaluation questions above. PROJECT BACKGROUND The Partnerships for Enhanced Engagement in Research (PEER) program was initiated in 2011 and implemented by the National Academy of Sciences (NAS) to support research projects jointly conducted by developing country scientists and researchers supported in the U.S. by federal science agencies. Currently PEER is implemented by NAS and managed at USAID by the Center for Development Research (CDR) as part of the Global Development LAB. PEER operates under a funding ceiling of $100 million with a current period of performance of ten years. The program has funded about 250 grants in 50 countries worldwide between 2011 and 2016, with funds and buy-ins primarily from multiple USAID Missions and through partnerships other US Government Agencies. This mid-term evaluation is focused on the first five years of PEER implementation, through the fourth grant funding cycle, with the purpose of informing program management and implementation of the next five years, as well as any additional related or “spin-off” programming. Not only does PEER support research relevant to USAID’s development objectives, it also aims to establish long-lasting research relationships globally, build the capacity of local scientists and engineers to conduct high-caliber research, and enable scientists to become better partners in development. Additionally, the program seeks to leverage and mobilize the US scientific community and the USG investments in research already made --as represented by formal partnerships with US federal science agencies-- to help USAID achieve development objectives in multiple sectors. The target sectors are: biodiversity, health, agriculture, environment, water, disaster mitigation, climate, education, food security, and energy. USG Partner organizations (NASA, NIH NOAA, NSF, USDA, USGS, and Smithsonian Institute) support awards to U.S. partner scientists who serve as mentors to PEER researchers. Since 2011 the PEER program has released 7 individual solicitations referred to by program staff as: PEER Science-Cycle 1, PEER-PIRE, PEER Science-Cycle 2, PEER Science-Cycle 3, PEER Health-Cycle 1, PEER Health-Cycle 2, and PEER-Cycle 4. After the third year of PEER, a USAID made a strategic shift to integrate separate PEER Science and PEER Health work-steams into a single ‘PEER’ program. In addition to annual solicitations the program has also supported four annual meeting for participating PEER PIs in various regions including, Bangkok, Arusha, Lima and Amman. These regional meetings provided capacity building and networking activities as well as the opportunity for PIs to connect with USAID Mission, 3 USAID Washington, and NAS staff. NAS also hosts at least two financial management trainings per year for administrative staff at institutions with PEER grants. In order to fund those projects with the highest likelihood of achieving development impact while building local research capacity, PEER supports an annual solicitation and review process. The Annual solicitation includes various research “focus areas” with priorities specific to technical areas like biodiversity, health, wild-life trafficking, among others. The PEER solicitation is released in early October. Pre-proposal submissions, including a letter of support from the applicant’s research collaborator (U.S. partner scientist) who is supported by one of the USG agency collaborating agencies on PEER, are due in January. USAID leads a review of these pre-proposals in early February based on the relevance of their work to the USAID Missions’ development objectives outlined in their country development and cooperation strategy (CDCS) or regional development and cooperation strategy (RDCS). Proposals deemed relevant by the Mission are invited to submit a full proposal. NAS leads an NSF style technical merit review of full proposals in May/June of each year. July allows time for back and forth with Missions to make funding decisions. Final awardees are announced in August. In addition to ‘core’ funds put in annually by the Global Development Lab, PEER also accepts ‘buy in’ funds from both central and field operating units across the agency. The mechanism has also been used by external partners like the National Cancer Institute and Office of Naval Research to sponsor specific PEER awards. USAID buy-ins are typically incorporated as ‘focus areas’ in an annual RFA. Focus areas reflect research priorities of the funding operating unit and may cover a range of sectors including: biodiversity, energy, wild-life trafficking, water, and maternal health, among others. EVALUATION METHODS & LIMITATIONS METHODS The PEER midterm evaluation employed a mixed method design integrating quantitative and qualitative data collection and analysis. The evaluation team analyzed program progress and performance on the three key dimensions of the PEER Program, also in line with the evaluation questions: 1) critical evidence: research quality and applicability to solving existing and emerging problems; 2) collaboration: professional and institutional linkages among researchers, policy makers, and practitioners; and 3) capacity development: to develop local capacity and increase global knowledge of how to develop context specific innovative solutions to alleviate poverty, mitigate climate change, and reduce morbidity and mortality. The evaluation team conducted a series of interviews beginning in Washington, DC with key staff from USAID, NAS, USG partner institutions, private sector partners and US-based partner researchers. The team conducted two country visits, one to Indonesia as stipulated in the scope of work, and one to Kenya. Kenya was selected because it is the largest PEER program in East Africa and because PEER research projects span several of PEER’s focus areas, including climate change, natural resource management, energy, health, and food security. The two countries offer an interesting contrast: while Indonesia has had several monitoring site visits from USAID Regional Advisors, Kenya has had only one to date. And while the Mission in Kenya has bought into PEER recently to support research on 4 wildlife trafficking, the Kenya Mission has not had as significant and direct funding engagement with PEER. The size and importance of USAID/Kenya’s program and its close relationship with the government of Kenya provides an enabling environment for the application of research to policy and programs. In contrast, the Indonesia Mission has funded at least 27 projects through their Education, Health and Environment Offices, and taken an active role in setting research priorities and reviewing proposals. Over the course of ten days in each country, two members of the evaluation team, supported by a local data collector and logistician, conducted interviews with key informants at the USAID Missions, host country government agencies, PEER-funded research institutions and collaborating organizations. The team conducted focus group discussions with selected students and research assistants. Finally, the team led a workshop to employ the Most Significant Change Technique, discussed in more detail in Annex 2. The Evaluation Team Leader traveled with the Science and Technology Advisor to Kenya from September 6th to 17th. The Health and Science and Technology Technical Advisors from the evaluation team traveled to Indonesia from September17th to 27th. Annex 2 details the application of each data collection method and the type of corresponding analysis used in distilling the findings, conclusions, and recommendations from the raw data in order to address each evaluation question. The data collection and analysis methods have been selected based on the most efficient pathway for getting to the answers of the evaluation questions. The different data collection and analytical methods used are summarized in the table below. Table 1: Data Collection Methods, Sources and Analytical Strategies Data Collection Method Evaluation Question Data Sources Anal;ysis Method Desk Review 1, 2, 3, 4 Progress reports, monitoring data, financial reports, assessments and evaluations, PEER funded-research, comparison programs; PIs and Co-PI reports; Document coding; consolidation of standardized notes; comparative analysis; trend analysis (changes over time) Key Informant Interviews 1, 2, 3, 4 USAID/W and Mission staff; NAS staff; USG partner and partner institution staff; host country government reps; collaborating organizations (NGOs, private sector, research institutions) Content pattern analysis; divergence/ convergence analysis Online Survey 2, 3, 4 PIs, Co-PIs, US-based Research Partners Frequency Distribution; Cross￾tabulation of survey variables; Gender constraints analysis; Simple univariate analysis such as averages, sums, percentages. Group Interviews 2, 4 Students and Research Assistants (Question 4); Female PIs/Co-PIs (Questions 2 and 4) Content pattern analysis; divergence/convergence analysis; Gender constraints analysis Site Visits 2, 3, 4 Selected PEER activity sites in Kenya and Indonesia Comparison analysis – planned vs. actual and/or before/after Most Significant Change Workshop 2, 3, 4 University staff, PIs and Co-PIs involved in PEER grant implementation Collection of stories to help identify most significant planned and unplanned PEER outcomes; Gender constraints analysis. 5 The following table details the actual data collection activities conducted by the PEER mid-term evaluation team in contrast to the planned activities. Issues encountered to explain any deviations are discussed in the section on Limitations, below. Table 2. Data Collection Activities: Planned vs. Actual Stakeholder KII Survey Focus Groups Site Visits MSC Workshop Plan￾ned Actual Plan￾ned Actual3 Plan -ned Actual Plan -ned Actual Plan￾ned Actual USAID/ Washington Officials 8 6 USAID Mission Officials 13 19 NAS Staff 4 4 US Partner Scientists 11 5 188 125/197 USG and Private Partner Institution Officials 10 9 Host Country Government Officials 4 4 Comparison Program Officials 4 3 PIs/Co-PIs 21 40 204 190/229 16 14 2 5 16 30 PI University Officials 6 5 2 8 Students and Research Assistants 16 29 16 Staff from Other Collaborating Organizations (NGOs, private sector, research) 2 2 2 TOTAL 83 97 392 315/ 426 32 43 6 5 40 30 3 Figures for actual survey responses are comprised of (# of valid responses received/# of surveys distributed) 6 LIMITATIONS This section provides a discussion of limitations showing where actual data collection deviated from planned activities, what impact if any resulted from the deviation and what mitigating action were taken to ensure that the evaluation findings, conclusions, and recommendations are valid and evidence based. DATA COLLECTION PLANNED VS. ACTUAL In terms of overall targets for all data collection activities, we exceeded the number of planned activities, as evidenced by Table 2 above. The most significant difference in planned versus actual activities were the number key informant interviews conducted with US partner scientists, with 5 of a planned 11 interviews conducted. Scheduling was a challenge, as the evaluation team’s interview request received six nonresponses. However, the survey response for this cadre was significant, 125 completed out of a total of 197 or 63% of the total population. This provided a credible way to fill gaps in data from the interviews. Across all stakeholders, the evaluation team exceeded targets by conducted 97 interviews out of a planned 83. This included oversampling interviews of PIs (40 conducted of a planned 21) and USAID Mission Officials (19 conducted of a planned 13), though Mission official were sampled in fewer countries than planned. The evaluation team compensated for this by conducting additional interviews after the first draft report to close the gap. The team interviewed Mission staff in seven of the planned 11 countries, including Armenia, Bangladesh, Brazil, Haiti, Indonesia, Kenya, as well as interviews with two regional Mission staff from RDMA (on TDY in Jakarta) and El Salvador for a total of nine out of a planned 11 Missions engaged. GENERALIZATION AND REPRESENTATIVENESS Of 49 countries, 88% of these were captured in the survey. Interview data was oversampled for the two countries with the largest portfolios (Kenya and Indonesia) as these two locations were selected as site visits. The interviews covered 11 countries comprising 22% PEER countries and 50% (104 projects) of the total portfolio (204 projects). External validity is likely high for the survey data. Regarding the interview sampling structure and process, there is likely some selection bias present affecting external validity. Concerning information about PEER PIs: The evaluation is likely very representative, with high confidence for results concerning their views and experiences. The data is likely generalizable to the two largest countries, Kenya and Indonesia, as there was a larger portion of data collection from these two locations. For countries with 1 project, data might provide information about the project but not scalable to the nation. Extrapolation towards a whole country’s PEER experience from 1 project should be avoided as this is not likely representative of the scientific cohort for that nation. Sample data from the PEER PI survey is within a 95% CI, meaning the results (for variables with 178 observations - total number of surveys) are within a ±5% margin of error. Based on the response rates, the margin of error is even smaller (±3.46) given the number of surveys (178) was much higher than the need to meet the 95% CI (144). Concerning US Partner Scientists: The data is somewhat representative and although the evaluation does not have a complete picture of their experience (63% response rate), it has provided some key areas of focus for future changes in the PEER program. Likely the length of the survey, variation in request for participation (each Agency varies in effort), and the lack of incentives 7 contributed to the lower response rates. To obtain a 95% CI the sample size needed was 130 surveys, however only 116 US Partner Scientists responded. The margin of error for this group is larger than that for the PEER PI. Based on the number of surveys for US Partner Scientists, the error size is ±5.85. RESPONSES BY AGENCY The findings from this evaluation are likely more representative of those projects from NSF and NIH, and not representative of projects from agencies like Smithsonian Institute with only a 12.5% response rate (Figure 1). The evaluation team compensated for the uneven survey response distribution through interviews with nearly all the science agency partners to ensure their perspectives were included in the analysis. The survey results also do not include information from the newer partners, such as the private sector. Key interviews were conducted with the new private sector partner National Instruments to inform program management, but these projects (Cycle 5 forward) were not part of the scope of the evaluation. Concerning the Views of USG Agencies, USAID, and NAS: Findings from interview analysis are likely to be representative of USAID PEER DC, USG Agencies and NAS. Participants from these three groups, overall, have been part of the PEER program since the inception and can provide complete information across time. Although we did conduct interviews covering nine USAID/Missions, data about their experience is somewhat limited due to the diverse nature of engagement with PEER, size and scope of Mission objectives and priorities and the fact that staff rotate frequently between Missions such that during August September and October when we conducted many of these interviews, most mission staff are on vacation or in transition to new posts or are working on fiscal year reporting and less available. Accessing their time and accessing the right persons can be very difficult and the data collected reflects these constraints. It was hard to get staff from Missions, even in-country (i.e. Kenya) to participate in an interview. Some of the staff listed as a potential interview were new and/or relocated; this caused some delay in setting-up appointments for interviews. Figure 1. Survey response rates per USG Agency for both PEER PI/Co-PIs and US Partner Scientists. 8 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS FINDINGS EVALUATION QUESTION 1 – TO WHAT EXTENT IS THE PEER PROGRAM BEING IMPLEMENTED EFFICIENTLY? This evaluation approaches program efficiency according to a series of sub questions established by the statement of objectives and includes evidence gathered around the following categories: 1. Comparison to similarly structured programs; 2. Perceptions of PEER stakeholders of program structure, management and benefits; 3. Stakeholder communications; 4. PEER solicitation and review process; 5. Participation of women scientists; and 6. Program buy-in mechanisms. COMPARISON TO OTHER PROGRAMS During data collection, there were a large number of comparison grant programs that PIs cited from their earlier experiences, but consistently stated that PEER is superior in flexibility and process. The evaluation examined three programs with similar structures and objectives as the PEER program to understand different program management and implementation approaches, which could serve as alternative or complementary structures or processes to improve the efficiency of PEER program implementation. The three comparator programs reviewed include the Feed the Future Innovation Labs (Innovation Labs), the UK Newton Fund, and the Netherlands Organisation for Scientific Research￾WOTRO Science for Global Development programs (NWO-WOTRO). Each of these programs engages researchers in the donor country and requires collaboration with a host-country counterpart to address development challenges. Like the PEER program these programs’ objectives aim to build capacity, encourage collaboration between developing country researchers and donor-based researchers, and support scientific research to address development challenges to varying degrees (See Table 3). Each comparator program is described demonstrating similarities between the comparator management structure and processes and PEER’s and highlights unique features of the programs. The conclusions bring to light components of the comparison programs which could serve as alternative or complementary structures or processes to improve program implementation. This review is based on analysis of publicly available documents and interviews with staff affiliated with each program. 9 Table 3: Comparison Program Objectives Comparison Program Objectives Feed the Future Innovation Labs (formerly CRSP model) BIFAD Review 2012:3 1. Bringing good science to the solution of significant development challenges in the agricultural and related sciences 2. Capacity building in host countries, especially human capacity development, linked to research 3. Provide mutual benefits to U.S. and partner countries for research and HICD activities UK Newton Fund Programs http://www.ukcds.org.uk/funding/funding￾landscape/newton-fund 1. People: improving science and innovation expertise (known as ‘capacity building') through student and researcher fellowships, travel schemes and joint centers 2. Research: research collaborations on development topics 3. Translation: innovation partnerships and challenge funds to develop innovative solutions on development topics. NWO-WOTRO Science for Global Development programs www.nwo.nl/wotro 1. Increase the generation, dissemination and utilization of high-quality knowledge 2. Increase the generation of groundbreaking ideas 3. Enhance knowledge synergy for development 4. Strengthen research capacity in developing countries USAID-funded Feed the Future Innovation Labs Model: The Feed the Future Innovation Lab model was born out of the Collaborative Research Support Program (CRSP) model, a USAID-funded mechanism supporting collaborative international agricultural research. The CRSPs were founded in 1978 under Title XII “Famine Prevention and Freedom from Hunger,” as an amendment to the1961 foreign assistance act (Rubin 2008). The model’s dual goals include “Bringing good science to the solution of significant development challenges in the agricultural and related sciences” and emphasize “capacity building in host countries, especially human capacity development, linked to research.” This investment in international agriculture research is intended to mutually support the U.S. and partner countries. (BIFAD Review 2012:3). I In 2012, the CRSPs were rebranded as Feed the Future Innovation Labs for collaborative research, and managed under USAID’s Bureau for Food Security (BFS). Among those that were rebranded, eight continue to operate similarly to the CRSP model, which more closely resembles PEER in terms of its management structure and project design. Since 2012, 15 Feed the Future Innovation Labs were added. The 24 Innovation Labs are managed by a U.S. University or a consortium of partners. The size of each Innovation Lab’s portfolio varies. For example, the Integrated Pest Management Innovation Lab (former CRSP) currently includes eight projects with different focus areas in seven countries whereas the Feed the Future Innovation Lab for Climate-Resilient Wheat led by Washington State University focuses 10 its research only in one region in India. These grants are administered under the Leader with Associates award mechanism. Mission buy-ins and Associate Awards can occur at any time. The ceiling on a single buy-in is stipulated in the centrally managed Leader Award (ADS 303: 78). The Innovation Labs are Leader with Associate Awards managed by U.S. Universities or a consortium, which serves as the management entity, with the Associate Award held by the Mission. The management teams typically include a director, deputy director, communications manager, and financial support team. Their responsibilities include management of institutional sub awardees, a competitive grant selection process, and interfacing with the sub awardee PIs and Co-PIs, and providing supervision, and serving as the main point of contact to an AOR at USAID/BFS. Each Innovation Lab is assigned an AOR in BFS who oversees anywhere from 3-5 Innovation Labs. A level of engagement with the Mission is required for the overall management of the awards, approval of work plans and other inputs. The sub awardees selected by the management entity can include U.S. universities, host country research and university institutions, and international agricultural research institutions. Funding for the sub awardee institutions is allocated by the management entity. The Horticulture Innovation Lab in 2016 RFAs required sub awardees to cost share 25 percent of the total federal funds requested. Sub awardees select and collaborate with host-country collaborators, such as universities, research institutes, government agencies, NGOs, other USAID-funded projects, etc. The Innovation Lab management entity pre-determines the research topic. The research implemented by the sub awardees varies from a focus on “upstream” research (e.g., developing drought-resistant varieties of cowpea) to “downstream” applied research. The length of the program cycle can vary. The Innovation Labs used to f fully funded graduate degree programs for students, and still do on some occasions covering four or five years. More recently, it is more common to support short periods focused on the most productive research years, with a greater emphasis on research output and less of Figure 2: Feed the Future Innovation Lab 11 an emphasis on building the individual capacity of the student or researcher. The eight former-CRSP Innovation Labs also include a human and institutional capacity development focus, providing funding to support research of Master’s and Ph.D. candidates from sub awardee institutions. Short-term training courses are also offered in host countries. It is not uncommon for a PhD to be fully funded by the lab, as a student can be “handed off” from one Innovation Lab to another. Additionally, the Innovation Lab management entities facilitate meetings among project stakeholders and management staff. A single Innovation Lab convenes stakeholders in host-countries annually. This provides a mechanism for PIs to share highlights from their research and network with other PIs/CO￾PIs, collaborators, students, and management staff. The Innovation Lab Council whose members include leadership from the Innovation Lab management entities convene in-person at least once a year where they have a collective opportunity to engage with USAID. The Innovation Lab council also arranges calls throughout the year to discuss cross-Innovation Lab management related issues. A unique feature of the Innovation Labs is their long-term collaborations in host countries. Eight of the 24 Innovation Labs have long-standing connections with host-country institutions and collaborating institutions based on, in some cases, decades of programming. While each award cycle is only 5-years, half of the Innovation Labs have been awarded to the same management entity or similar set of sub awardees at least twice. The BIFAD-commission review of the CRSPs found benefits in this long-term sustained investment arguing that “Investment in research requires long term, sustained support to promote development impacts” and found that “long-term presence has also provided a mechanism for the development of regional networks, ‘spillover’ benefits to non-participant countries from a given CRSP, and the emergence of new institutional partnerships that further enhance the CRSP consortia” (BIFAD Review 2012: 19). The management entity’s institutional memory and long-standing relationships put the Innovation Labs in a unique position to achieve development impacts. Like PEER, the FTF Innovation Labs also place an emphasis on tracking the impact of funded research on policy, in cooperation with their USDA partner. During an interview, they cited an example of how research supported by FTF led to the lifting of an export ban on maize in Tanzania, something that USDA strategically focuses on when supporting international agricultural development. Another example illustrated how the AMS Basis Innovation Lab developed index based livestock insurance. Some innovation labs follow a specific process for setting a research agenda, which requires engaging policy makers in a dialogue with researchers to ensure research will meet policy requirements. Links with the Consultative Group for International Agricultural Research Centers (CGIARS) and US university partners were also cited as important factors in developing linkages with policy makers. 12 Figure 3: United Kingdom Official Development Assistance (ODA)- funded Newton Fund4 United Kingdom British Council: The Newton Fund The Newton Fund’s website,managed by the Department of Business, Energy, and Industrial Stragegy (BEIS) of the U.K government, provides resources for foreign researchers on how to find research partners in the UK. 5 Piirus (https://www.piirus.ac.uk/) is a new online tool used by the Fund to assist researchers with finding other researchers to collaborate with. The Fund website also offers advice on writing research proposals and guidance on the UK visa application process. The British Council, recently established the Newton Institutional Links program6 offering “research and innovation collaboration,” with grants between £30,000 to £300,000, for UK and developing-country partner teams over two years. Among the Council’s various research programs, Newton is most similar in structure to PEER. Research topics covered include weather conditions, access to affordable health care, and food and enegy security.7 The program aims to faciliate new and develop existing collaborations between academic groups, departments, and institutions in partner countries and the UK for research and innovation. Newton requires the research to be responsive to country priorities, which are established prior to the call for proposals through discussions with national stakeholders.8 Newton 4 https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/550747/Newton-Fund￾frequently-asked-questions.pdf 5 This is also marked on the UK Collaborative on Development Sciences website, which outlines funding all UK funding mechanisms for collaborative research and capacity building actitivies supporting developing countries. 6 Additionally, the British Council through the Newton fund supports researcher workshop and travel grants for early career researchers, Ph.D. scholarships, professional development for researchers, development of STEM teaching resources, and technical and vocational training in the research and innovation sector. 7 https://www.britishcouncil.org/education/science/institutional-links 8 https://www.britishcouncil.org/sites/default/files/2016_july_il_-_guidelines_for_applicants_v5.pdf 13 promotes flexibility and retaining a portion of the funds each year respond opportunistically to unexpected events, and to integrate better with other development programs. For example, during an outbreak like the Zika virus, they may release a special call with that focus. The funding covers expenses including “research-related costs; the exchange of researchers, students and industry staff (including SMEs and not-for-profit organisations); costs of organising meetings, seminars, training; and other activities to establish and strengthen collaborative links.”9 Netherlands Organisation for Scientific Research-WOTRO Science for Global Development programs: The Netherlands Organisation for Scientific Research (NWO) – WOTRO program was established in 1964. The Netherlands Organisation for Scientific Research is an independent agency which sits under the Ministry of Education, Culture and Science. The WOTRO Science for Global Development funds research focused on addressing development challenges in food and business research, security and rule of law, sexual and reproductive health, and research for inclusive development in sub-Saharan Africa. Its overall objectives are similar to PEER’s, focusing on building evidence, fostering innovation, supporting collaborative relationships, and strengthening research capacity (see Table 3). Figure 4: WOTRO Science for Global Development The program funds several different kinds of programming. It’s large-scale, long-term research programs are most analogous to PEER. Formerly they conducted these programs with a focus on thematic areas, 9 https://www.britishcouncil.org/education/science/institutional-links 14 such as food security, reproductive health, rule of law and conflicts around climate. They involve collaboration between researchers, and close consultation between the partnering organizations. Some calls require private sector involvment as well. The open calls were typically of a longer duration, up to four years and fully supported an entire PhD research period, but like the FTF Labs, they are trying to be more focused, with shorter grant periods, with the addition of using core funding to add value in support of collaboration. These teams are composed of Dutch researchers and researchers from developing countries with the goal of strengthening linkages between research, policy and practice. Funding for programs goes to the institution rather than an individual.10 Programs are managed by WOTRO. The 18-person staff is primarily responsible for coordinating the proposal process. The office coordinates writing the calls for proposals, stakeholder workshops, posting the calls for proposals, and identifying experts for proposal review, and establishing an advisory committee which ranks the proposals (Gerri Tuiter, pers. comm). The stakeholder workshop is used to raise research challenges to be explored and potential collaborations. 11 The staff are also part-time on other tasks and according to our interviews, they are finding this difficult to sustain, particularly during calls for proposals. One NWO-WOTRO program, similar to PEER, is the Food & Business Applied Research Fund (ARF), which funds applied research focused on food security and the private sector. The grants vary from 50,000 Euros over six months to 300,000 Euros over 36 months. Proposals are submitted by a consortium which includes at least one organizational partner (private or public) from one of the 15 Dutch development cooperation partner countries12 and one research or higher education organization. It requires that one of the organizations is Dutch. It emphasizes co-creation of the research between the consortium members and demonstration of how the research could be applied. It does not require that the partner country organizational partner implement the research. Funding may be used for personnel overhead, audit costs, travel (up to 20% of the budget), research costs, and knowledge sharing (e.g., workshops, stakeholder meetings) for up to 15 percent of the budget. Funds are provided to the institution of the applicant, which seems to work well according to their staff. Funding frustrations tend to stem more from internal delays. The international research organization is required to co-fund, either cash or in-kind, from the international research organization. This co-funding must amount to as least 20 percent of the grant total. 13 A unique feature of the WOTRO program is its requirement for each research team to include gender expertise and female scientist participation. The overall NWO uses funding instruments that are aimed 10 http://www.nwo.nl/en/about-nwo/organisation/nwo-divisions/wotro 11 http://www.nwo.nl/en/about-nwo/organisation/nwo-divisions/wotro/wotro+and+the+sdgs 12 Afghanistan, Bangladesh, Benin, Burundi, Ethiopia, Ghana, Indonesia, Kenya, Mali, Mozambique, Palestinian Territories, Rwanda, South Sudan, Uganda, Yemen 13 http://www.nwo.nl/en/funding/our-funding-instruments/wotro/food--business-research/food--business-global￾challenges/food--business-applied-research-fund-arf.html 15 at engaging women scientists including Aspasia, Athena, and FOm/f. It also contributes to a network of female scientists called LNVH. In the proposal review process, it monitors the proportion of proposals submitted by men and women which is reported on in an annual report. 14 When asked if they have an explict focus on policy impact, the WOTRO representative claimed that they have just started to increase their focus on policy, but that they do not have the capacity for the follow-up required to track it confidently. They generally collect data similiarly to PEER through self reporting of each project, and collecting outputs like publications, which often yielded information about policy impact by coincidence. To improve knowledge sharing, they focus on workshops and platforms designed to make stakeholders aware of each other and they attempt to map the influences they have on each other. Summary and Comparison of the three Comparator Programs and PEER: The table below summarizes key characteristics of the three programs compared to PEER. The FTF Innovation Labs formally institutionalize long-term relationships with USAID and consortia of U.S. Universities, and between US Universities and universities in USAID host countries. Both types of relations often exist de facto in PEER but are not formalized under the program. An advantage of the UK Newton fund related to the link between research and national policy objectives is that research priorities are set in consultation with national stakeholders. The evaluation team was not able to determine, however if these consultations result in a greater level of uptake of Newton research for application in programs and policies. WOTRO offers a model which supports increased involvement of women scientists in both the Netherlands and host countries by requiring that all research teams include women scientists and members with gender expertise. A network of women scientists provides ongoing support during and beyond the life of the research grant. Table 4: PEER Program Comparison Table Program Name Funding Agency Approximate Annual Budget Grant Sizes Program Strengths Program Challenges PEER USAID Global Developm ent Lab $10 million $40,000 to $300,000 Leverages partnerships with USG science agencies and private sector; US Partner scientist funding; balancing portfolio management with Mission buy-ins Feed the Future Innovatio n Lab USAID Bureau for Food Security $50 million Roughly $2 million across 24 LWAs (plus buy￾in) Long-term relationships/networks with USAID Missions and US universities foster links to policy makers; resources to convene stakeholders; consider impact on policy making a key result. Greater emphasis on research outputs means less opportunity to support student degrees; Complex network with a large number of LWAs and institutions – not sure if reaching a “critical mass”; only agriculture and food security focused. 14 http://www.nwo.nl/en/policies/gender+diversity 16 Newton Fund UK ODA managed by Departme nt for Business, Energy and Industrial Strategy (BEIS) $188 million Between $37,500 and $375,000 Flexible programming – funds held for moments of opportunity; Sets funding priorities with national stakeholders in host country prior to making grants Requires a lot of coordination and communication across agencies with varying capacities WOTRO Science for Global Developm ent Netherlan ds Organisati on for Scientific Research (NWO) $20 million Between $53,000 and $320,000 Resources to convene stakeholders; specific focus on engaging women scientists; Management stretched and part-time only; still nascent on tracking policy impact. STAKEHOLDER PERCEPTIONS The survey of PIs and their US partner scientists provide an overarching measure of perceived satisfaction with their experiences participating in the PEER Program, as well as perceptions on the benefits of participating in PEER. There was a high level of satisfaction from PIs/Co-PIs and slightly less so with their US partners on the PEER experience. Eighty-five percent (150 out of 177) (of PIs/Co-PIs and 66 percent (73 out of 1 111) of US partners had a very good or outstanding experience. Ninety-three percent (162 out of 177) of PIs/Co-PIs and 69 percent (77 out of 104) of US partners were satisfied or extremely satisfied with PEER. However, 10 percent of US partner scientists were "extremely unsatisfied." (See Table 4). Concerning the US Scientist's experience, the survey data captures the size and magnitude of level of dissatisfaction and negative view about PEER. In reviewing comments from this group, there are no good responses or explanations for why they had negative experiences. The interviews, however, provided some information regarding areas that may be problematic to this group of scientists. The evaluation team heard examples ranging from unresponsive PIs that broke off contact upon award to high levels of discomfort with the lack of transparency surrounding financial management practices at PEER PI institutions. When looking at the variables for PEER PI responses of satisfaction against sector, career stage and cycle, Chi-Squared / Fischer’s Exact Test were not significant. For PEER PIs when comparing their responses across their career stage, satisfaction level and cycle, the data appears to cluster primarily with scientists having ten years or more of experience, and funded during science cycle 1-3 and cycle 4. This group generally shows a high level of satisfaction. (See Annex VI, Table 2). Similarly, when comparing PEER PIs/Co-PIs in survey responses by sector, career stage and cycle, there is some clustering with scientists with ten years or more of experience in biodiversity, in atmospheric 17 sciences/disaster management, health and in water for the same cycles (science cycles 1-3 and cycle 4) 15 Table 5: Perceptions of PEER by Principal Investigators/Co-PI and U.S. Partner Scientists Principal Investigator/Co-Principal Investigator (Q17 Sex) Female Male Total (Q17 Sex) Female Male Total Satisfaction with PEER Nu m￾ber Per￾cent Num￾ber Per￾cent Overall Experience Num -ber Per￾cent Num -ber Per￾cent Num￾ber Outstanding 18 25.4 53 74.6 71 Extremely Satisfied 32 34.0 62 66.0 94 Very good 27 34.2 52 65.8 79 Satisfied 19 27.9 49 72.1 68 Good 8 36.4 14 63.6 22 Neutral 0 0.0 3 100. 0 3 Poor 1 100.0 0 0.0 1 Dissatisfied 1 50.0 1 50.0 2 very poor 0 0.0 3 100.0 3 extremely Dissatisfied 2 28.6 5 71.4 7 Too new to PEER and.. 0 0.0 1 n/a 1 Total 54 123 177 Total 54 120 174 (US Partner Scientists Q17_Sex) Q17_Sex) Female Male Total Female Male Total Overall Experience (Q36) Num -ber Per￾cent Num -ber Per￾cent Num￾ber Satisfaction with PEER (Q58) Num￾ber Per￾cent Nu m￾ber Per￾cent Num -ber Outstanding 10 29.4 24 70.6 34 Extremely satisfied 7 31.8 15 68.2 22 very good 10 25.6 29 74.4 39 Satisfied 16 32.0 34 68.0 50 good 5 18.5 22 81.5 27 Neutral 2 14.3 12 85.7 14 poor 0 0.0 5 100.0 5 Dissatisfied 0 0.0 7 100.0 7 very poor 1 33.3 2 66.7 3 Extremely dissatisfied 1 9.1 10 90.9 11 Too new to PEER 1 33.3 2 66.7 3 Total 27 n/a 84 n/a 111 Total 26 n/a 78 n/a 104 The most common PI expectations are related to the “opportunity for building new relationships with other scientists” and the most common US Scientist expectation is to “help build the capacity of scientists.” However, other common responses from interview data with US Scientists was the ability to maintain and expand existing relationships and to improve access to and quality of data sources from other countries. 15 These results mostly reflect the fact that the majority of PEER scientists have 10 or more years of experience, so if the majority of the scientists are also satisfied, then, thein then most PIs who are satisfied will also have 10 years or more of experience. 18 NAS consistently received high praise for responsiveness, engagement, guidance, and flexibility both in the survey responses from PIs and their partners, as well as in interviews with other stakeholders who interact with NAS, including USAID Mission staff. Forty-three percent of PI/Co-PI surveyed rated NAS responsiveness as exceptional and about 75 percent provided positive responses with regards to NAS taking action to resolve conflict. One area identified for improvement was in better leveraging NAS core competencies in scientific research, beyond grant management, to potentially improve or increase training and capacity building opportunities. In addition, Fifteen percent of the survey respondents felt that NAS could improve how they connect PIs with the US partner scientists. Right now, it is mostly through a list they provide. Survey results also indicate a fairly positive performance for US Funding Agency Partners in support of the US Scientists they fund as PEER partners. Fifty-nine percent of US Scientists said their agency Program Officer was very supportive, but also felt that it was not their role to problem solve and were not the primary access point to their PEER PI partners. Sixteen percent felt that the program officers needed to improve communication related to programmatic information relevant to the PIs and scientists. Indeed, interviews with representatives of the US partner agencies consistently pointed to information access as a difficulty for them in maintaining good communication. STAKEHOLDER COMMUNICATIONS There are mixed opinions on reporting. Some felt that quarterly reports forced them to reflect more and appreciated the rigor. Some felt it was over burdensome and felt that three months was too early to report changes and would prefer semiannual. Both the survey and interviews consistently pointed to the inability to use PEER funds to support the travel of the US partner scientist as a significant barrier. PIs, their partner scientists, and the US partner agencies generally agreed that lack of travel funds for U.S. Partner Scientists hinders the level and frequency of communication between the partners. PEER CYCLE PROCESS Although the PEER evaluation team collected a fair amount of suggestions from stakeholders on how to make improvements to the solicitation process, there was a consistency in responses that indicate, not surprisingly, that PIs with some experience and familiarity with writing research grants tended to feel that the PEER process was not too difficult and found the process straightforward. The pre-proposal concept submission was cited as saving time and useful for quickly matching research ideas with USAID priorities. By contrast, PIs with little or no experience in grant writing tended to find the process a bit more difficult, instructions unclear and complex, and priorities difficult to discern, or shifting. Several US partner agencies also felt that it was difficult to keep track of the funding priorities from year to year to keep their PIs looped in for participation and matching with PEER PIs. PIs frequently expressed appreciation for the feedback received during the proposal review process, not only from reviewers but “My expectations of PEER is that it would support enhanced engagement, building bridges between us, but we faced many obstacles in actually collaborating more directly” (US Partner Scientist) 19 from their partner scientists. Many felt this was much different than other grants they received. In terms of outreach during the grant cycle, and connecting with the scientific community, the survey data indicates most PEER PIs/Co-PIs learned about the call from other colleagues. In addition, a third of the respondents stated learning about PEER online, suggesting the online PEER presence is effective in reaching out/encouraging scientists to apply. Word of mouth via emails lists from colleagues was mentioned frequently by PIs and US partner scientists as a key source for learning about a PEER call. A larger percentage of the US partner scientists believed their project was connected to the PEER project. US partner scientists stated the PEER project was very closely tied to their USG funded grant (48%), and with another 44% stating they felt the link between the projects was “somewhat” close. Another finding related to the PEER grants cycle is that US partners were usually contacted by one potential PEER PI (41%), but a close second was the group with 2-3 people contacting them (38%). There is a likelihood USG agency type and number of potential PEER PIs (who contacted the US Scientists) are associated (p=0.007). Is there a chance the type of field makes a difference, in other words, are some US scientists in some fields getting contacted more? Based on frequencies, the data suggests US scientists in the “environmental fields” (environment, climate, energy, biodiversity, agriculture) are being contacted more than those in other fields. Supplemental resources outside of the PEER grant also play an important role. Almost half of the US partners provided funding to support PEER activities (45%). Overwhelmingly, supplemental requests from USG agencies were for equipment (95%). However, requests for student support (86%) and travel (72%) had high consensus amongst respondents as areas of supplemental requests. Delays in purchasing and receiving equipment was consistently cited by PIs and their partners. Common specific issues were not around the time of the receipt of funds, but rather in many cases the need for a tax exemption letter or changes in procurement rules either at the PI’s institution or elsewhere that ultimately impacted the price of equipment due to delays, such that the costs no longer aligned with budget allocations. Indeed, budget management in general was consistently cited as an issue, ranging from a general lack of experience in managing budgets, to unbudgeted administrative costs and fees, to currency devaluation, particularly impacting equipment purchasing. PARTICIPATION OF WOMEN USAID staff place a clear emphasis on addressing the gender gap in research, identified by their own study of PEER’s efforts. There is a clear indication from reviewing the calls for proposal and speaking with scientists who responded to them that there is an effort made to specifically promote the participation of women PI applicants. We also consistently heard in interviews with program staff that all things being equal, an award will typically be made to a woman PI over a man PI during the review and selection process. Indeed, PEER is approaching 40% participation of female PIs in 2016, which is an improvement even from the previous two years of data used in USAID’s own analysis from 2014 and 2015 showing roughly 27 percent of applicants are women. The issues faced by women in the field of scientific research varies widely across all countries where PEER operates. PEER data is skewed somewhat based on two of the largest PEER countries in terms of grant numbers, Indonesia and Kenya, having some of the highest participation rates of female PIs. Some PIs in Kenya, both men and women, 20 did not feel that gender was a significant issue to be addressed in term access, nevertheless one PEER woman PI cited a government grant set aside only for female PIs as extremely important as an entry point for women in scientific research. PROGRAM BUY-IN PEER annual budgets have experienced an ebb and flow, ultimately trending towards fewer core dollars each year and relying much more on buy-in funding from USAID Missions and other Operating Units. A quick comparison shows the funding ratio went from 62% core funding and 35% buy-ins for PEER Cycle 1 in FY 2011 to 21% core funding and 79% buy in for FY 2014 Cycle 4 funds. This requires significant staff time for outreach and capturing priorities that will be unique for each annual PEER grant solicitation to line up with a variety of Mission priorities. EVALUATION QUESTION 2 – TO WHAT EXTENT HAS PEER HELPED FILL EVIDENCE GAPS WHICH INFLUENCE POLICY/PROGRAM CHANGE? PRODUCTS AND OUTPUTS Data from multiple sources (survey, interviews, desk-review) show PEER has produced more products related to capacity building (conference presentations, speaking engagements, stakeholder workshops, training curriculum than other types of products. Survey data demonstrates that attendance and presentation of papers at international conferences/meetings are the most numerous PEER products (58%). Presentations and attendance at national conferences are he second most frequent scientific research products (Figure 5). Paper and poster presentations within the scientist’s department was the third most frequent PEER output, with about 44% of the sample stating they produced these. Other common products were “new grant proposals for new funding opportunities” (42%), invited speaker within country (43%), and stakeholder workshops (49%). Not anticipated was the 16 percent of PEER PIs producing blogs/vlogs/videos with scientific instruction with PEER support. Responses ranged broadly about other types of products (Figure 5), such as on new technology (14%), tools (19%), new methods (28%), open access data 19%), and 16% said they had produced blogs/vlogs/videos with scientific instruction. It is unclear whether PEER scientists report a complete inventory of all products to NAS. or provide links to their datasets for sharing. Of those who stated producing new technology, men produced more technology compared to women and no new technology was seen for Brazil or Kenya (Table 5). However, when testing for significance for an association between sex of the researcher and new technology, these variables were not found to be significant16. 16 This does not signify there is not an association, but that the data (small sample) does not find sufficient evidence to suggest a relationship between gender and new technology. 21 Figure 5: PEER PI Response Regarding PEER Products This pattern was also seen in the product “tools” where more men than women produced PEER tools per the PEER PI/Co-PI survey responses. In summary, 22 new technologies were recorded by the survey mostly created by men none created in Kenya or Brazil, two of the countries with the largest portfolios. Other countries with new technology included Cameroon, Bangladesh, Columbia, Ghana, India, Lebanon, Philippines, and many other countries with 1 product per country. The project fields shown with new technologies were Water (6), Biodiversity (5), Agriculture (3) and Climate (3). A total of 33 new tools were recorded in the survey response with 78% of them created by men and most of them in the field of biodiversity (6) and Water (5). 22 Figure 6. i3L Lab Facilities (Indonesia). Table 6:. New Technology and Tools Produced Under PEER Per Survey Results by Sex (PIs only). New technology Tools Yes No Yes No Female 7 (28%) 47 (31%) 7 (21%) 47 (32%) Male 18 (72%) 106 (69%) 26 (79%) 98 (68%) Total 25 (100%) 153 (100%) 33 (100%) 145 (100%) The opinions of the US Partner scientists about their own participation in the creation of tools (16%), new technology (12%), open access data (16%), and programmatic strategies (2%) were similar to those given by the PEER PIs. Where these opinions did differ were concerning the product “policy recommendations/analysis for government agencies (8%). From the desk-review and interviews several examples emerged of technologies with the potential for large impact but some are stalled in their development and other are too young:  Wireless network system for real-time landslide monitoring – Indonesia (2-year delay in equipment purchase)  Pharma check – Indonesia (3-year delay in getting supplies and access to the right equipment)  Tungsten fire-set clay pot for improving water quality – Kenya (still under development early tests show clearly the pot can remove both inorganic and organic matter)  Alternative energy project iL3 Indonesia  Waste to renewable energy: biogas cleanup – Tanzania and Kenya (too early still underdevelopment and with some equipment delays)  Sustainable conversion of oil palm lignocellulosic waste into pentanol using metabolically engineered microbes – Indonesia (issues with access to materials causing delays as well a change in partner)  Mwangaza project on science, technology, engineering, and mathematics and computing education for students in Kenya with vision loss - Kenya (recently completed and application/impact is still unknown) The projects listed above can be classified as “high risk/high reward” projects. They have excellent potential to create large scale impact within a country, yet they are also experimental, and might require new technology not easily obtainable. Currently, these types of projects are more likely to receive core 23 rather than bilateral funding as they may be perceived by the Missions as less likely to contribute directly to programs and policy initiatives, especially in the short to medium terms. PUBLICATIONS PEER PIs/Co-PIs reported to have published 47 (27%) peer-reviewed journal articles, with another 64 (36%) under review, and with 11 (6%) submitted but rejected. Looking at the percent’s given by the US Partner (who may or may have not been an author with the PEER PI/Co-PI) they reported 37%, 27%, and 10% for articles published, under review and rejected, respectively. Looking at the difference between men and women for articles published, the data did not support a significant association between gender and peer reviewed journal articles published. Overall, Indonesia had the most for both men (4) and women (4), Kenya had 2 for each, and Brazil had 2 for men only. The other countries (as a group) had a total of 9 for women and 20 for men. However, when looking at articles under review by gender these seemed to differ significantly, 72% of these were by men and only 28% by women (even though there was not significant association between these two variables). INFLUENTIAL FACTORS ON PEER PRODUCTIVITY Almost half of all PEER PIs did not feel USAID/Washington or the Mission had any impact on their ability to produce PEER products. Factors that were rated highly positive for PEER productivity included length of time of the project (79%), quality of communication with US scientist (87%), size of the budget (89%), support from NAS (86%), level of support from their institution (86%), and frequency of communication with US partner (87%). The US Partner felt the lack of budget for their participation was a negative factor on their ability to produce PEER products (72%), and felt no impact from interacting with NAS (69%), home institution (61%), and USAID (64%). However, they felt the level of technical capacity of PEER scientists was a positive factor in PEER productivity (72%). PEER researchers indicated that equipment and infrastructure are important factors in their research; 79% of PEER PIs indicated in survey responses that infrastructure and equipment are significant factors in producing PEER products. Additionally, 60% of PIs indicated that physical infrastructure is a barrier to innovation. From PI interviews primarily conducted in Kenya and Indonesia, interviewees reported prevalent delays in equipment purchase and procurement processes, which may be contributing to the low number of new tools and technology. Any innovations linked to technology have largely been stalled due to equipment. Delays in purchases not only slow productivity and likely affect the period in which the products emerge, they also can affect teams as the work cannot start. Equipment issues also had a negative impact on their research budgets, when some PIs had to purchase equipment at much higher prices than anticipated originally due to choices offered per their procurement offices, leaving fewer resources for other research costs. Others encountered significant differences between their original cost estimations at the proposal stage and real costs at the purchasing stage, which the number and quality of their products. One PI offered the GIZ example concerning taxes and VAT issues. GIZ has a clause to cover the costs of the taxes regardless of what was estimated in the proposal. Another PI requested more “no-cost” extension options/flexibility to work within the system, as it can be slow and 50% of the budget cannot always be committed in the first year. In addition, some projects require permits which often can take time, causing delays. 24 USE AND PERCEPTIONS OF USE What did you do with the products? Both the PEER PIs and US Partners were asked in the survey about their perception of use of their products as well as what action these groups took with the PEER outputs. For PEER PIs/Co-PIs, 82% of them stated they shared their PEER products with other people in their scientific community, 41% sent them to USAID, and 34% to their government agencies. The US partners shared their products with other people in their scientific community (74%) almost as equally as the PEER scientist, but sent their products to USAID in much lower numbers (17%) Perception of use differed between the PEER PI and the US Partner: More than half of PEER PIs/Co-PIs felt their products were likely being used to improve evidence in their field (63%), whereas only 45% the US partner respondents felt the same. Both groups of respondents stated similar answers about use for changing policy at the national level, but 28% the US partner scientists’ responses were use of PEER products to help change policy at the community village level. US partner scientists (36%) and PEER PIs (46%) felt their products were being used to provide guidance at the community; 56% of the PEER PIs stated they though their products were being used as information for improve decision￾making by their governments, compared to only 24% of US partner scientists. POLICY AND PROGRAMMATIC CHANGE It proved difficult to document systematically the degree to which PEER is contributing, policy change. Most of the evidence is anecdotal as even the survey responses depended on respondents interpreting what constituted a contribution or influence. Examples of PEER research informing policy or program change should be put in the context of the processes through which policy change occurs, the conditions or enabling environment needed to support such change, and engagement of critical actors beyond researchers and policy makers. A brief review of theories of policy change appear in Annex V. Interviews with PIs and Co-PIs indicate that many either do not believe they have a high capacity for engaging and communicating effectively and with the right stakeholders to affect policy or program change, or they do not view it is their role as a researcher. Some PIs, however are well connected and contribute to networks and membership organizations designed to advocate for policy. Others work directly with advocacy organizations government institutions as part of their research platforms. For example, one research team at Kenyatta University in Kenya engaged in the Baby Friendly Community Initiative has taken on a Co-PI that works for a policy advocacy NGO. This has created direct collaboration with district level government officials in their research. Regardless of their capacity for affecting program or policy change through research, the Most Significant Change workshops conducted during this evaluation show a strong preference for stakeholders towards PEER funded research that has a higher-level impact. Groups consistently voted for the stories around policy impact as the most significant. Per the survey data, about 34 percent of PEER scientists are sharing their products with the government. Most of them (82%) are sharing amongst colleagues in their field. 31% PEER PI stated they sent products to NGO/international organization, which may also be capturing products sent to NAS and USAID, but survey data was not specific on that nuance. One respondent of the 17 that provided “other” stated “private sector. If the goal of PEER is to impact policy and influence government 25 decision-making, more effort will need to made to incentivize this behavior. Based on interviews, PIs have stated cost as a limitation. Indonesia institutions offered funds to host an event but felt they needed the help of USAID to ensure government officials would attend. In Kenya, funding needs were cited in relation to convening outreach events to share with decision-makers, as many government officials expect a payment or stipend to participate. However, this also may represent the true focus of most scientists which is to share their findings, advance their field through publication. An alternative explanation could be they are informally sharing with government and thus do not necessarily count these interactions, thus the data might be underestimating the level of scientific information sharing. Of the Missions interviewed, none appear, except for one, to suggest PEER is providing input into programmatic change or influencing program design. Comments from one Mission suggested the activity of affecting mission programs is likely much more difficult than impacting national policy change. There is still limited understanding as to where in the process PEER results are needed or useful. The Indonesia Mission had some comments about a misalignment with their goals and the projects in place; RDMA only takes projects that support existing programs ensuring alignment and connection to the mission goals. A more fundamental question remains as to whether the Mission wants to use PEER inputs for programmatic change and perhaps PEER should determine this during the pre-call prioritization of projects with missions. PEER participants were asked about whether they knew of an example where policy change had been created/changed/affected because of PEER activities. Of the 178 individuals who responded to the PEER PI/Co-PI survey, 23% felt they had produced policy recommendations/analysis for government agencies and 3% programmatic strategies. Results for the US Partner survey (representing 63% of the population of this group) showed 8% and 2% for policy recommendations/analysis for government agencies and programmatic strategies, respectively. The survey data showed more men than women producing policy recommendations/analysis although this was not significant (p=0.701)17. More PIs/Co-PIs (138 responses) stated they had not produced a product related to policy recommendations or analysis for their government (Table 6) compared to the small percent that said yes (22%). These responses do not necessarily signify that there was a policy change, only that the PI provided as a product of PEER a policy recommendation or analysis for their government. Table 7: PEER products for policy recommendation/analysis by gender (PIs/Co-PIs survey responses) Product – Policy Recommendation/ Analysis For Government Female Male Total Yes 11 (28%) 29 (72%) 40 No 43 (31%) 95 (69%) 138 Total 54 124 178 17 STATA results for a Fischer’s Exact test (appropriate for cells with small numbers), did not support a significant relationship between gender and Q22_policyrecomendations. 26 Regarding the number of responses for Q22, policy recommendations/analysis by field, there were 7 responses for Water, 5 for Climate, 8 for Biodiversity and 0 for Health. Furthermore, responses for this variable by country showed 5 responses for Indonesia, 2 for Kenya, and 1 for Brazil for “policy recommendation/analysis for government”. The remainder of the responses fall within all the other countries. The survey data provided limited information about policy change or the effects of PEER on policy. Data from the interviews and desk review provided examples of where PEER research may be affecting policy. Table 7 provides a list and some details about each project. Out of the more than 88 interviews conducted for this evaluation, with larger sampling of PIs and projects in Kenya and Indonesia, the findings suggest PEER is likely contributing to policy change. The size and significance of this effect is not known, but the interview data and desk-reviews highlight projects likely having policy impacts and/or with a potential for it in the future (Table 8). Table 8: Summary list of PEER projects connected to policy change. Country Project Description Policy Change Level Field Source Lebanon Landslide risk mapping for Lebanon Adopting the use of the maps to assess landslide risk areas fully supported by the Lebanese Parliament and likely to be used for concerning camps with Syrian refugees National Disaster Mitigation Desk￾Review/ Interview Kenya DNA Wildlife Trafficking Project by the National Museums of Kenya and Kenya Wildlife Service Changing the legal standards/procedural rules of evidence for court cases in illegal trafficking through improved DNA science and technology of plant and animal species. The government has been involved in this project since the beginning and is a partner through KWS. Government Institution/ National Biodiversity (Natural Resources/ Environment) Interview Feasibility and effectiveness of the baby friendly community initiative (BFCI) in Kenya: A pilot community trial in a rural setting. Is seen globally as an example on how BFCI successfully can be implemented in public hospital settings in Kenya District Health Interview/ MSC Impact of PRONTO training in emergency obstetric and newborn care on 24-hour neonatal The rural community hospital has adopted the approach and there is potential others will once final analysis has been completed this year District (government hospitals/ institution￾level) Health Interview 27 Country Project Description Policy Change Level Field Source mortality Indonesia Development of a Referral System using Kangaroo Mother Care (KMC) for Low Birth Weight Babies Hospital has adopted the KMC approach and the district health office is a partner. This work is informing other hospital developing criteria for discharge of the baby. Two hospitals now have standard operating procedures (SOP) for KMC, and they did not before. The hospital has proposed to the new health insurance agency [new national health policy providing insurance to all Indonesians] to include payment for a KMC gown. District (Hospital/ National) Health Interview Integrated Local Emergency Response Policy Improvement and Capacity Building for Advance-Early Warning System in the face of Near￾Field Tsunami Risk Policy change on tsunami warning and emergency response for the city District Disaster Mitigation Desk￾Review Incorporating Bali's Subak heritage into primary and secondary education: curriculum development, teacher training, and action research (Indonesia, Education, Cycle 1) Indonesian government adopted new curriculum to enhance awareness of agro￾ecological issues and subaks with thousands of Co-PIes of the book going to schools, students, teachers, and libraries National Education (Water/ Agriculture) Desk￾Review 28 Table 9. Summary list of PEER projects with potential (i.e. likelihood to affect policy in the future, however they are too early in project or process) for policy change. Country Description Potential for Policy Change Level Field Source Indonesia Integrated Watershed Management for Enhancing Local Livelihoods and Biodiversity Conservation in Indonesia – CIFOR Revision of land use policies and planning based on biodiversity and conservation Approach to integrate research and capacity building of local communities in to District Environment (Governance) Interview Sediment Transport Evaluation on Bengawan Solo River (downstream and estuary) to Minimize Sedimentation and Flood Combining Effect on Nearby Infrastructure Working with the Ministry of Public Works of East Java to help improve estuary management through improved data with the use of new patented wireless technology District Water Desk￾review Kenya Mwangaza project on science, technology, engineering, and mathematics and computing education for students in Kenya with vision loss - Kenya Ministry of Education has shown a high level on interest in the technology and data created by this project. Discussions are occurring around the use of the data within the Ministry’s new database system to be built National Education Interviews Peru Strengthening Resilience of Andean River￾Basin Headwaters Facing Global Satellite-based monitoring system of water availability and quality integrated into the National Drought Observatory (ONS) National Water Desk￾review 29 Country Description Potential for Policy Change Level Field Source Change used by the government Armenia Sustainable Fisheries for Enhanced Water Resources in Armenia (SFEWRA) Working early on with the government and USAID mission. High level of support and interest in the PI’s models for water recirculation, find alternative uses for water (testing water quality now). Also looking at models related to fisheries and risks with nuclear power plants. Has raised Mission understanding of the issues (not necessarily program design, but maybe in the future) Programmatic National Water/Energy Interviews Ghana PRESSA: Photovoltaic Reliability Evaluation in Sub-Sahara Africa Contributions to country standards and Ghana’s 30-year National Development Plan National Energy Desk￾review Brazil Capacity Building for Participatory Monitoring of Changing Forests in Sustainable Use Areas of the Southwestern Brazilian Amazon Already created a new forest management training curriculum developed under PEER approved by Ministry of Education Community leaders are interested in applying the results of the research including associations of residents and producers in the CMER and governmental environmental agencies at the state and municipal levels. Community State Biodiversity Desk￾review 30 Three examples Illustrate how PEER might be influencing policy at difference levels: 1. Institutional Policy Change – Kenyatta University (KU) Research Policy: From the interview with a director in the Office of Research, Innovation and Outreach, he felt strongly that the presence of PEER projects at his university, as well as other grants, has directly affected the restructuring of his unit, and thus policy change at the university level around research – reflected in their new policy. For KU, PEER has helped faculty see that It is possible to write proposals, get grants, manage resources. To be able to provide more support to faculty to management international grants, his office has re￾structured to created new divisions for research support specifically around faculty capacity. This division is comprised of 4 units, Finance and Development, Administration, Academic, and the new one only 2 years old, Research Innovation and Outreach The PEER experience, as part of the larger growing research portfolio and grant awards at KU, has influenced changes in the new 5-year KU Strategic Plan. This interviewee believes in the next 5 years the university will need to make changes (outlined in the new strategy to come out this year) to build capacity in research (writing proposals, searching for funds, partnership development), research dissemination and uptake (peer-reviewed publications, community engagements), and improve collaboration between scientists. In addition, having PEER and other large awards from other international organizations (Rockefeller, Gates Foundation, USAID) is pushing the university to switch to E-Systems to better access data about overall project progress, reporting for leadership, and increased speed in approvals for purchases. 2. National Policy Change – National Capacity Building Research Initiative by Indonesia: Indonesia has just recently launched a new program in partnership with USAID, aimed specifically at building research capacity for the country and modeled after the PEER program. his new initiative, Sustainable Higher Education Research Alliances (SHERA), has likely come about, in a small or large part due to the presence of PEER in Indonesia (the size of the impact of PEER on the creation of SHERA is unknown). Multiple factors have likely contributed to helping the Ministry of Higher Education and Research move ahead with this new alliance and initiative. First, Indonesia has the largest portfolio in PEER and has had the longest direct support for this program by the Indonesia Mission. Second, the staff at the Mission show a unique level of enthusiasm and effort for PEER outreach. During the interviews with mission staff, it was clear they specifically ensure their trips around the country are opportunities for outreach on PEER. SHERA’s goal is to increase the national scientists’ capacity for research and innovation. The program is still new - only launched in October 2016. 3. Community/District Policy Change – Kangaroo Mother Care and Baby Friendly Community Initiative: Based on both site visit and interviews, this project has changed procedures at the hospital level and has a probability of being implemented in larger scales. Indonesia requires the testing of any approach in their country even if it has proven effective in other regions. The district government is engaged directly with the project and is an equal 31 partner, which likely has provided help in the adoption of this approach. This project has also been given support from the Mission. In addition, the hospital has proposed to the new health insurance agency (new national health policy providing insurance to all Indonesians) to include payment for a KMC gown. Should this occur, the impact of the project will be national. Similarly in Kenya, PEER supported collaboration between a PI at Kenyatta University and a Co-PI at a health policy advocacy and research NGO to study the application of the Baby Friendly Community Initiative implemented in other places, like Vietnam. There was extensive involvement from the beginning from community, district and national government officials because of the interest in policy implications. UNICEF joined as a collaborator as well for the potential to scale such an initiative. Ultimately the research impacted recently completed policy documents that were shown to the evaluation team members during our interviews. This project was also selected at the Most Significant Change workshop as the top story presented because of its policy impact.. EVALUATION QUESTION 3 – TO WHAT EXTENT HAS PEER PROMOTED COLLABORATIONS BETWEEN US SCIENTIFIC COMMUNITY, LOCAL RESEARCH, AND USAID MISSION, EMBASSY STAFF, AND PRIVATE SECTOR PARTNERS? This section presents the combined findings from the survey and key informant and group interviews on different types of connections, collaboration, and other types of engagement generated by the project. This section examines the formation of partnerships, incentives and benefits of collaboration, quality and growth of professional networks, and scientific outcomes in relationship to the project’s underlying assumptions. Formation of Partnerships: The PEER competitive grant making process requires each host-country scientist seeking funding to identify a scientist collaborator with funding from a USG agency, such as the NSF, NIH, NOAA, NASA, USDA, or the USGS. There are a several assumptions about collaboration under PEER that merit examination: Incentives: The objectives of the project are that the relationships created through the project will increase scientific output, strengthen ties among scientists in the US and USAID host countries, build capacity, and strengthen policies and programs. Unlike similar programs that provide joint funding for collaborative research, PEER provides funding to the host country PI for his or her own research. It is incumbent on the PI to identify a collaborating partner from among US scientists who have separate grants for their own research from another USG program. PEER does not provide financial incentives to the US scientists, who do not receive any additional funding beyond their own separate grants for engaging with the host country scientist. The project design is premised on the assumption that the opportunity to work in the country, potential to access new and expanded data sets, and to expand professional networks will provide sufficient incentives to motivate US scientists’ engagement with PEER PIs. Networks: A second premise behind the project design is that PEER PIs, compared to their US peers, have relatively narrow professional networks, confined mostly to their own countries, and that the relationship with US collaborating scientists will expand their networks, especially when these 32 partnerships are between collaborators who had not known each other previously. In addition, PEER grants often provide funding for attendance at international conferences, and occasionally for the PI or other research team members to travel to the US Partner Scientist’s university. Benefits: A third premise is that the relationships among USAID and other USG agencies allow them to leverage their resources for greater impact on scientific output than either could do on their own. A parallel is often posited by these agencies about the ability of host country scientists and their USG partners also to be able to leverage their joint research funding for greater scientific output than either one could do independently. Formation of Partnerships Collaboration between the PI and the US partner scientists is foundational to the PEER project design. A majority of PIs knew their US partner scientist prior to asking them to collaborate on their PEER grant. Slightly less than a third of PIs (30%) and US partner scientists (29%) said they did not know each other prior to PEER. More than two fifths of both PIs and US partner scientists said they had collaborated on previous research efforts. The survey data also show that most of both PIs and the US partner scientists are mid-level to senior scientists, with considerable experience in accessing research funds. PIs the evaluation team spoke to in Indonesia, explained that for the minority of the PIs, who did not know their US partner scientists, NAS provided a list which they consulted to identify a partner scientist with USG funding. In most instances, the relationships formed with unknown partners were as supportive as those formed with known partners. Per the survey results, 39% of all US partner scientists stated “previous experience with the scientist” as the primary reason for their selection of their PEER PI. Men (34%) and Women (37%) had scored similarly on their primary choice in PEER PI (Figure 7). The second highest reason for PI choice select in the survey was “topic of research” (19%). Both groups were asked if they would collaborate with each other again; 96% of PEER PIs/Co-PIs stated yes and 89% of US Partner Scientists also stated yes, they would collaborate again. Thirty-one percent of PIs and 23% of US partner scientists had known their respective collaborators for 5 years or more. Table 10. US Partner Scientists Survey Responses – PI Partner Selection What was the primary reason for choosing your PEER PI Partner? (Q28) Frequecy Percent Previous experience with the scientist 39 34.82 Topic of research 19 16.96 Scientist expertise and qualifications 15 13.39 Location of research 11 9.82 Recommended by a colleague/friend 11 9.82 Other (please specify) 9 8.04 The project augmented my current grant 8 7.14 33 Figure 7. US Partner Scientists Survey Responses – PI Partner Selection (sex disaggregated). Table 11: PEER PI/Co-PI Survey Responses on How they First Met US Partner How did you first meet your US Science Partner? (MARK ALL that apply) Answer Options Response Percent Response Count I worked with this person on prior research projects 43.9% 75 We did not know each other prior to PEER 30.4% 52 I hired him/her 0.6% 1 He/she hired me 0.6% 1 The US Science Partner had hired me 2.3% 4 We consulted together 10.5% 18 I was an invited speaker/scientist/lecturer to the US Partner scientists's home institution 3.5% 6 we met at a conference 17.5% 30 we wrote a paper together 15.2% 26 the US Partner scientists was my graduate student 0.0% 0 I was the US Partner scientists's graduate student 3.5% 6 we were in graduate school together 1.8% 3 the US Partner scientists attended one of my training/workshop/lectures 4.1% 7 We met at my University/Institution during one of my US Partner scientists visits 15.2% 26 Other (please specify) 20.5% 35 answered question 171 skipped question 19 In interviews conducted with PIs, they described a variety of ways in which they either met or selected their US Science partner, if they had not already worked with them. Examples of responses are below: 34  Met at a local, regional or international conference;  Selected name from a list supplied by a USG Science Agency as part of the PEER application process;  Introduced through a mutual colleague such as a Co-PI; There was considerable variation in the depth of collaboration between PIs and US partner scientists on the development of the proposal, with 47% of PIs and 52% US partner scientists saying that they had put equal effort into development of the PI’s PEER proposal. Among the rest of the groups, involvement in the proposal ranged from the US partner scientists having no involvement to completely writing the proposal for the PI (19%). Similarly,10% of PIs stated that they had no input from the US partner scientists to receiving some collaboration in the form of the review of different versions of the proposal. One PI interviewed in Indonesia said that the US partner scientists sent his assistant to Indonesia to work with them on the proposal over two weeks. With respect to the probability that PIs and US partner scientists will collaborate in the future, 96% stated Yes, they would collaborate again with their US Partner, and 89% of US partner scientists stated they would collaborate with their PEER PI again, even after their own grant ended (87%) and after the PEER PI’s grant ended (83%). INCENTIVES The primary motivations for PIs to apply for a PEER grant is the desire to build new relationships with other scientists and to develop new and advanced scientific skills. Other strong incentives include, the objective to grow existing relationships with other scientists and to develop solutions to problems in their communities. Table 12: PI/Co-PI Expectations for PEER Grant (PI/Co-PI Survey Data) What did you EXPECT to gain through PEER? (MARK ALL that apply) Answer Options Response Percent Response Count New/advanced technical and/or scientific skills 79.7% 141 Capacity to acquire more funding for my research 66.1% 117 New material/expertise to teach courses/classes 41.2% 73 More graduate students to support my research 48.0% 85 Develop a new product (journal article, technology, model, data, laboratory process, other) 66.1% 117 Build new relationships with other scientists 84.7% 150 Grow existing relationships with other scientists 75.1% 133 Provide my community with a solution 68.4% 121 Change the information my government uses to make decisions 62.7% 111 Improvement in/or access to equipment/technology 54.8% 97 Other Other (please specify) 4.0% 7 answered question 177 35 The survey results also revealed motivations for collaboration on the part of US partner scientists, which include the opportunity to continue to work with the PI on areas of mutual interest; potential for helping and receiving help from the PI on a shared research topic of interest; an interest in the country or region; the need for a research partner in the region or country of interest; enthusiasm for building collaborative partnerships; and a direct request to the partner from the PI. Additionally, US partner scientists interviewed for the evaluation mentioned other incentives, such as, the desire for greater international exposure of their own work (US partner scientists collaborating with PI in Bangladesh); the desire to help scientists in other countries working on topics of mutual interest and exchange of scientific data (US partner scientists collaborating with a PI in Afghanistan); to connect more directly with work on the ground and more practical application of research (US partner scientists collaborating with a PI in Armenia)’; and to gain access to key stakeholders in other countries, such as government officials (US partner scientists working in Kenya). The desire for building capacity of both host-country and US students was an incentive for both US partner scientists and PIs. The incentives for USG agencies to participate in the programs are mostly focused on expanding their international footprint through their association with USAID. For example, NASA cannot directly fund researchers outside of the United States. PEER has allowed the agency to broaden the community of researchers they can engage with (NASA interviewee). The National Cancer Institute, in contrast, does not rely on PEER for global engagement as they have other mechanisms for engaging globally, but PEER has allowed NCI to interact more directly with USAID and an expanded and more diversified network of international partners (interviewee at NCI). NETWORKS As presented in the previous section, PIs’ greatest incentive for applying for a PEER grant was the desire to expand their professional networks with other scientists. A related, and somewhat surprising finding, is that the majority of PEER PIs already have quite extensive and diversified contacts with a range of stakeholders. Figure 8: PEER PI/Co-PI Interactions with Other Stakeholders 36 Figure 9: US Partner Scientists: Pre-PEER Interactions with Other Stakeholders Host country PIs appeared to have very comparable networks to those of their US collaborating partners. Figures 8 and 9 demonstrate the great range of individuals and groups host-country PIs/Co-PIs and U.S. partner scientists engage with. Men’s and women’s engagement in networks are proportional to their representation as PEER awardees. Women and men U.S. partner scientists also appear to engage proportionally with different networks with their representation research positions. The survey indicates that close to 90% of PEER PIs/Co-PIs said they had collaborated with scientists from another country prior to receiving a PEER grant, and 75% had received an international research grant before competing for a PEER grant. An even larger percentage of US scientists have collaborated previously with scientists from other countries.18 Many respondents had prior interaction with 9 or more of these groups. These findings support the data that respondents have extensive experience in science, grants, and international collaboration (see Descriptive Statistics in Annex VI). Both types of respondents were asked about prior interactions with several groups including US Scientists, USAID, and NAS. Prior to PEER participation. PEER PIs and Co-PIs stated that they have interacted with US partner scientists (81%) as much as they had with scientists from their own department (81%). Prior to PEER, 84% of the US partner scientists stated to have had interactions with “US Scientists” and 80% with “USG Agencies”. 19 Only 12% of the PEER PI/Co-PI sample had prior-to-PEER interacted with NAS, whereas the US Partner Scientist had a larger percentage (37%). The data for interaction with the private sector showed larger 18 At least 10 percent of PEER PIs/CoPIs stated that they did not interact with any group – not necessarily to signify this but perhaps likely missing data (Figure 8. Q42). However, 12 of the 19 are surveys were dropped due to missing data for all variables. These observations have been dropped and thus a very small number (7) remains which had never interacted with any of the groups. 19 For these questions, respondents were able to mark all that applied. 37 percent interaction for the PEER PIs (46%) then for their partnering scientists in the US (34%). As Table 12 shows prior to PEER many of the PIs/Co-PIs had some connection to USAID (36%), the US Embassy (25%), and even USG agencies (21%). Through interviews with PIs/Co-PIs, the evaluation team observed that there was a lot of confusion about which individuals associated with PEER in Washington were from NAS and which were from USAID, especially as representatives from both organizations often make joint trips. PIs expressed the desire to have more interaction with USAID and wanted the Mission to play a greater role as a convener of PEER researchers, government officials, USAID-funded programs, and the private sector, so that they could share their findings more broadly than within their professional networks. The survey findings strongly echo this desire across other countries and among US partner scientists as well. In the survey, both PIs and US partner scientists recommend that NAS and USAID invest more resources and effort in convening gatherings of PIs and US partner scientists to allow more face to face interactions, and interactions with other stakeholders, such as the private sector, national and regional research entities, and US and host country government officials. In interviews with PIs in Indonesia and Kenya, the evaluation team learned that many PEER PIs heard about PEER through USAID emails they receive as a consequence of prior collaboration with or funding from the Mission. Another group heard about the grants program from colleagues who were on the email list. The USAID Mission in Indonesia also has an active outreach program for PEER. The PEER Coordinator in the Mission visits universities all over the country to give talks about the program. The Program also enjoys high level visibility among the highest levels of the US Embassy. PEER projects are often part of Ambassadorial and other USG visitors’ trips within the country. In contrast, the evaluation team found that there was little interaction between PEER PIs and the USAID Mission staff in Kenya, apart from a few high-profile research projects, such as those engaged in research on wildlife trafficking. Other Missions discussed a varying degree of engagement, from relatively frequent contact with PIs because there is a small number in their country to outreach through professional networks like working groups or steering committee participation that is relevant to research institutions. The host-country PIs also have a very high level of engagement with their governments. This was confirmed through the interviews with PIs in different countries. The evaluators found that a majority of PIs and Co-PIs interviewed engaged with a government agency, either at the national or local level, on topics related to their research. For instance, three health projects. Baby- friendly/breastfeeding initiatives in both Kenya and Indonesia are working with the MOH and district hospitals to produce the evidence that will contribute to more effective implementation of national policies. Table 13: PEER PI/Co-PIs Responses on Who They Interact With (Q42) Prior to PEER, who did you interact with? Response Percent Response Count Female Male Scientists from my Department 81% 139 42 97 US Scientists 81% 138 39 99 Scientists from other Universities in my country 76% 130 41 89 my country's Government Agencies 74% 127 36 91 Scientists from other Departments 71% 121 39 82 Rector/Dean of my university 60% 103 31 72 Non-Profit organizations 59% 100 28 72 my country's National Science academies 58% 99 29 70 38 Private Sector 46% 78 25 53 USAID 36% 61 25 36 US Embassy Staff 25% 43 12 31 US Science Government Agencies 21% 36 5 31 National Academies of Sciences (NAS) 12% 20 9 11 A higher percentage of US partner scientists have had interactions with USAID than host country PIs, 49% of US partner scientists as compared to 36% of the PEER PIs. In comparison, US partner scientists look to their partnership with PIs as a means of establishing relationships with host-country governments. In Kenya, the US partner scientists’ relationship with the Ministry of Health, which was facilitated by the local PI, allowed the US partner scientists to arrange for staff from the MOH to visit Cambodia to see how the Baby Friendly Initiative was working there. Prior work with governments for PEER PIs might be an important factor when looking at policy change. When disaggregating the data by sector, PIs with projects in biodiversity, water, health and environment had higher responses to pre-PEER engagement with government agencies. This is also a reflection of the larger portfolio in these fields. Some of the examples of PEER projects with likelihood of policy effect where from the health and biodiversity sectors, as well as one from water. Figure 10. PI/Co-PI Pre-PEER Interaction with Government Agencies by Sector (PEER PI/Co-PI Survey Responses) “If PEER did not exist…if this work was not ongoing, I think we would not have anything if we did not have the BFCI and it came out at a time when UNICEF was re-working we were in the process of exploring what other communities’ options could we use we were not being successful…so having the ministry seeing other countries and how they do it, that gave us an eye opener, implementation site was evident can happen in Kenya and not just Cambodia. If that had not happened. Would have not been a case clear…because in our model exploration we came across the CARE one but this one was much easier to take up. The intervention would not have been taken up in the country if not for the BFCI. (US partner scientist) 39 There is some indication that the PEER grants increased interaction with key stakeholders. For example, before the grant, only 12% of PEER PIs reported interactions with MAS. After receiving their PEER grants, 71% of PIs reported making new connections with NAS. For USAID, the percentage of PIs/Co￾PIs with pre-PEER interactions with USAID was 36% and post-grant new connections made with USAID was 66%. When disaggregating the data by size of PEER award, more of the awards went to individuals with a larger number of previous interactions. Those with larger budgets for their PEER research projects, with an award range between $100-199k, also had many prior interactions with multiple agents (for list of agents for interactions see Figure 8 (Q42) above. Figure 11: PI/Co-PI Pre-PEER Interactions By Grant Value (PEER PI/Co-PI Survey Data) BENEFITS AND CHALLENGES A majority of PIs stated that PEER strengthened or greatly strengthened relationships within and outside their institutions. A much smaller proportion said that grant led to the creation of new relationships with scientists within their institutions (9%) or with scientists outside their institutions (19%). PEER participation had a lesser impact on US partner scientists as a group. Only 25% said that PEER strengthened’ relationships with scientists within their institutions, although a majority said it strengthened their relationships with scientists outside of their institutions. There were very few, who stated that PEER created new relationships either inside (2%) or outside (5%) their institutions. Table 14: Changes in Relationships Due to PEER PEER PI/Co-PI Survey Data US Partner Scientist Survey Data Changes in Relationships due to PEER (170 responses): 1- Scientists inside my institution – 45% “strengthened”, 36% “greatly Changes in Relationships due to PEER: 1- Scientists inside my institution – 25% “greatly strengthened”, 65% “no change”, 2% “created NEW relationships” 40 strengthened”, 0% stated weakened, 9% “created NEW relationships” 2- Scientists outside my institution – 19% “created NEW relations, and the rest fell in the strengthened categories (about 70%+) 3- 28% created NEW relationships with Scientists from other countries 4- 37% said “created NEW relationship” with USAID 5- 48% said “created NEW relationship” with NAS 2- Scientists outside my institution – 5% “created NEW relations 3- 80% said no change with USG funding agency 4- 11% said “created NEW relationship” with USAID 5- 82% said “no change” with NAS This finding does not negate the value that both PIs and US partner scientists put on the partnership. Both groups were asked if they would collaborate with each other again, and both stated similar responses. An agricultural project in Kenya illustrates how PEER contributes to enlarging networks. Through a PEER sponsored meeting in Arusha, Tanzania, the PI met someone from the World Vegetable Center who alerted him that they had a post-harvest expert. He made a connection with the expert and they are now they are collaborating on research and training on indigenous vegetables. Both PIs and US partner scientists stated in interviews that one of the greatest challenges is the lack of face to face interactions over a sustained period of time, if at all. The lack of funding for US partner scientists to travel to the PI’s country and research site was a common complaint, by PIs in Kenya and Indonesia. One US partner scientist lamented that he couldn’t bring his partner to his lab for a couple of months from Afghanistan. Those who had not worked together before said that it was especially difficult to work at a distance with someone they did not know. In many countries, PEER has contributed to increased collaboration among national scientists and research institutions. In Brazil, PIs said that the greatest advantage of PEER was that resulted in more collaboration among Brazilian institutions, which had not worked together as closely before. The USAID representative in Brazil also indicated the importance of PEER as a convener or sciences and scientific evidence in support of their objectives. Similarly, the first regional PEER workshop in Lima allowed researchers to talk about methods for improved monitoring of biodiversity. As a result, they are now all using a common monitoring framework. In Kenya, PEER has expanded PIs’ and students’ networks beyond the university to engage with the Ministry of Health. It also brought MOH officials into the university as co-authors and reviewers of student proposals. In Armenia, most university research is not applied and tends to lack a platform for providing evidence or innovation to support local solutions. The PEER grant allowed PIs to connect with local fish farms and best practice sharing with people in other countries. It also contributed to the Mission objectives of enhancing safety related to the aging nuclear power facility by improving water conservation and thus contributing less to potential seismic events caused by changes in groundwater flow. Several interviewees said that they would like NAS and USAID to increase opportunities for networking among PIs and US partner scientists, both in their countries and internationally. Given the three-year 41 term of the grant, conferences and meetings are more effective ways to make others aware of new research than publications, because of the length of time it takes for an article to get published. Table 15:Prospects for Continued Collaboration (PEER PI/Co-PIs survey – response frequency by gender and country) Would you collaborate again with your US Partner scientists/Team? (Q42) Female Male NO YES NO YES Indonesia 1 11 0 14 Kenya 0 7 0 12 Brazil 0 1 1 5 Other 4 26 0 72 Overall, most PEER PI/Co-PIs would be willing to continue collaboration with their US partner scientists; particularly for the countries with the largest portfolios, they had only 2 “no” responses. The rest of the countries, with smaller PEER portfolios also support future collaboration, but of the few who stated they would not, these fell mostly in the “Other” country category (which includes all other countries with PEER activities), and largely female. Fischer exact test results did not find a significant relationship between country and gender (p=0.426). More than 80% of PEER PIs and Co-PIs serve as mentors to others, while only about a third say that they have a mentor. In part this may be a consequence of the fact that most PEER grantees are well established in their fields before having received their grants. During the Most Significant Change sessions in Kenya and Indonesia, participant PIs chose the stories about research programs that had the greatest policy application and engaged the widest network of stakeholders. US Partner Scientists were asked in the survey “if you could change three things in PEER what would those be and why?”. The largest change requested was increased funding (27%) for travel to meet PI, to exchange students/team members, and to collaborate with PEER PI (Table 15. frequencies per type of response category). The second largest request for change was concerning flexibility of funds and length of project (11%). Other changes requested by US Scientists included improvement in communication, transparency, matchmaking process, opportunities to collaborate/meetings, oversight of PEER PI projects, and flexibility of partnerships. Table 16: Frequencies of responses types of changes requested by US Partners. Collapsed Categories (US Partner requests for changes in PEER) Count % increased funding (for travel, for students, from USG, for PEER PI) 32 27.1 flexibility of funds and length of projects (increase) 13 11.0 improve and increase communication (with USG, PEER PI, Missions) 11 9.3 Other 11 9.3 increase opportunities for collaboration 8 6.8 improved oversight and accountability of PEER PI 6 5.1 42 Collapsed Categories (US Partner requests for changes in PEER) Count % capacity building for PEER PI and US partner 6 5.1 increase type and number of awards 5 4.2 changes rules on partnering/support 4 3.4 improve transparency and process for review 4 3.4 improve matchmaking 4 3.4 improve information 4 3.4 help decrease admin burden on PEER PI 3 2.5 improve flexibility of partnerships, size of projects, and length 3 2.5 increase USAID Mission support 2 1.7 improve process for travel (students and PEER PI) 2 1.7 TOTAL 118 100 EVALUATION QUESTION 4 – TO WHAT EXTENT HAS PEER STRENGTHENED CAPACITY IN DEVELOPING COUNTRIES? This evaluation approaches capacity building by considering the following sub categories around which data collection tools were formulated: 1. Individual Research Staff Capacity: PIs and Students 2. Student Capacity 3. New curricula and training opportunities 4. Impact on institutional capacity, including government use of scientific products. Findings in this section will be centered on these categories. Individual Capacity PIS AND CO-PIS:According to the PEER Results Framework, which ties directly into the Global Development Lab’s Science Results Framework, two out of the three intermediate results include some focus on building the capacity of local researchers as part of their sub results. Under those two IRs, Sub￾IR 2.1 is “local research teams strengthened to produce high-quality research” and Sub-IR 3.3 is “human scientific capacity in targeted areas strengthened.” The PEER Program has required reporting on several indictors to track progress along these lines, with some selected and presented below to provide a frame of reference for the evaluation data collected. PEER staff have identified data quality issues with the self-reported information collected from PIs, such as the high likelihood of double-counting within and across project years. For these reasons, the evaluation team did not have a high degree of confidence in using the existing monitoring data for any significant analysis, however it does help to illustrate the focus of the program on capacity building. It is also important to note that these indicators have been revised in a new PMP drafted in 2016, but some indicators still track fairly well with earlier versions. However, they are generally output indictors that do not provide a high degree of quality information on the outcomes or results of PEER funded activities. 43 Table 17:Table 16. Student Engagement in PEER by sex and year (PEER Monitoring Data) Forms of Engagement Year 1 Year 2 Year 3 Total Number of Students Engaged in PEER projects Female 962 981 327 Male 922 1,422 177 2,521 # of project-related events organized, such as workshops, conferences, exchange visits 434 304 138 876 # number of participants engaged in PEER S&T training activities and events (workshops, conferences) Female 6,321 3,449 2,883 12,653 Male 7,836 4,757 4,523 17,116 The evaluation team collected data through surveys and interviews intending to capture not only indications of the capacity built of individual researchers participating in PEER, but also the impact of PEER on their career and status within their institutions. Some key elements captured through perception survey questions are presented in the table below. PIs and Co-PIs largely agree that PEER has helped them to access additional funding, increased their status among peers and broaden their research horizon. US partner scientists indicated some similar results, though not to a high degree or as consistently, with many indicating “neither agree nor disagree” to these questions. One exception was on increasing scope and scale of research, to which 63 percent of the US scientist agreed that PEER has helped them to accomplish this. Table 18: able 17. PI/Co-PI Perceptions of PEER Benefits Survey question Response PI/Co￾PI US Partner Scientist PEER helped leverage more funding Strongly Agree 38% 8% Agree 38% 18% Neither agree nor disagree 40% Disagree 22% Strongly Disagree 11% PEER increased recognition with colleagues and institution Strongly Agree 37% Agree 46% PEER helped advance my career Strongly Agree 63% 6% Agree 32% 26% Neither agree nor disagree 43% Disagree 16% PEER helped increase scope/scale of my research Strongly Agree 11% Agree 52% Neither agree nor disagree 23% One primary skill set that PIs and Co-PIs identified as needing improvement regarding journal publication is writing capacity. Fifty-six percent specifically sited training in peer-reviewed journal paper writing could have helped them to better achieve their goals and 44 percent identified improved writing capacity as a need when asked what would most help them to publish articles. PEER has provided some training 44 on writing skills, primarily aimed at grant proposal development to improve success rates of applicants. When US partner scientists were asked what additional support they could provide if given funds, 76 percent stated that they could provide help with peer-review journal paper writing and 63 percent stated they could provide advanced technical training. Related to evaluation question two, another important skill set that was discussed consistently in interviews with all stakeholders was the capacity to engage at the right time and with the right set of actors to impact program or policy decisions with scientific evidence generated. PEER funded PIs seem to vary widely in terms of the networks they maintain that can assist them with influencing policy as well as the skill and capability to conduct outreach, promote, and collaborate in a sustained fashion to generate the level of engagement and buy-in over time that is generally needed to affect change at this level. STUDENT CAPACITY:Among the PEER PIs and U.S. partners who completed the survey PI’s reported directly supporting just over 1,800 students with PEER funding and the U.S. partner scientists reported directly supporting just over1,100 students with PEER funding. Overall, the PIs reported supporting nearly an equal number of male and female students (n=1836). Similar ratios were reported among U.S. partner scientists (n=1183) (See Error! Reference source not found.). Overall, PIs reported supporting m ore female students (n=970); however, slightly more male than female Ph.D. and post-doctoral students were supported (See Error! Reference source not found.). PI and U.S. partners’ relationship with students, facilitated by the PEER project, was perceived by the PIs and US partner scientists to be mutually beneficial to PEER research productivity and PEER students. In the survey, 80 percent of PIs and 33 percent of U.S. partner scientists reported building student capacity through resources the PI provided themselves. In interviews PIs also said that students benefited from PEER by gaining access to data sets for their research projects and access to experiences, products and presentations. Figure 12: Percentage of Male and Female Students Participating in PEER (Survey data) PIs also reported through interviews that the PEER program opened doors for students’ career development, allowing them to attend conferences or trainings abroad. This, one PI said, gave students new “confidence.” PIs also mentioned in the interviews offering training opportunities to build students skills including training on scientific writing and on data collection methods. U.S. partner scientists in interviews said PEER benefited students by facilitating students’ access to conferences. Access to equipment and laboratories, was also cited by PIs in the interviews, as a clear benefit of 45 student’s participation in the PEER research. Engagement of students in PEER research, as noted in quarterly reports, has led to completion of students’ theses on PEER research topics. Figure 13: Survey reported student participation in PEER (PI data) The PEER students who participated in Focus Group Discussions (FGD) in Indonesia and Kenya shared similar views as the PIs about what they have personally gained through participating in a PEER research project. In career development and gaining access to new equipment students in Kenya mentioned access to supplemental funding for their degree programs, networking opportunities, and gaining new knowledge as positive aspects of their participation in PEER. Students in the FGD in Indonesia cited parallel benefits as well as expanding their networks, travel, and contributing to research to address a development-related issue. In both groups, the students reported feeling fairly supported by their PEER advisors. Figure 14: Survey reported student participation in PEER (US partner scientist 46 PIs also viewed new course offerings, developed under PEER, as benefiting students; however, this benefit was not reported by students during the FGDs. For example, through the cycle 4 project Capacity Building for Participatory Monitoring of Changing Forests in Sustainable Use Areas of the Southwestern Brazilian Amazon in Brazil developed a Master’s course in Forest Science, which was approved by Brazil’s Ministry of education. In Lebanon cycle 2 Health Assessment of Earth Dams in Lebanon: Towards Sustainable Development project’s new curriculum was developed for university and high school students on earth dams as direct output of the PEER project. In the case of the cycle 2 Strengthening Resilience of Andean River-Basin Headwaters Facing Global Change project in Peru, the PIs reported in their quarterly report that participation in PEER led to additional funding from the National Science Council of Peru (CONYTEC) to develop a Ph.D. program in physics. During the FGD, the Kenyan students were asked to share their aspirations for the future. Everyone said they sought to use their technical skills to build capacity of other scientists, become a prestigious researcher in an academic or private institution, continue to conduct research in his or her field of study, or earn an advanced degree. PIs through interviews said their students are interested in pursuing advanced degrees and also saw value in opportunities for either studying in the U.S. or participating in an exchange program. In some cases, the PEER project has helped students realize these aspirations. For example, a PI and Ph.D. student from Ghana traveled to Arizona State University for a month under the cycle 3 PRESSA: Photovoltaic Reliability Evaluation in Sub-Saharan Africa project to work with the U.S. partner scientists. A student in Armenia who participated in an exchange visit under the cycle 3 Volcanic hazard assessment of the Ararat Valley project to the PEER U.S. partner scientist’s institution University of South Florida (USF) built a rapport with the U.S. partner scientist, applied for a Ph.D. program, was accepted, and now the U.S. partner scientists is his supervisor at USF. CURRICULA AND TRAINING PEER PIs/Co-PIs were asked about whether the support PEER has provided allowed them to create a variety of products. Concerning capacity building products (i.e. workshops, trainings, new course material, theses, and other), responses ranged from 6 percent to the highest at 41percent. For new courses, only 19% of respondents stated PEER had provided them support to create new curriculum. PEER appears to provide more support for the creation of new trainings, new theses, new workshops, and new stakeholder workshops and trainings (49 percent). This was consistent with data collected in the interviews with PIs in which a small percentage did cite examples of new courses developed, but nearly all discussed the opportunities for training, workshops and doctoral student development opportunities that PEER has afforded them and their students. There does not appear to be a be relationship between gender and PEER support to create new courses, new training, new training material, new thesis (either for students or for the PI), stakeholder workshops, and new workshops. There is not enough data to determine if there is a difference between the results of men and women for theses variables. Concerning whether there is a connection between the country of the PEER activity and the creation/or not of new training material, there does appear to be a significant relationship (p-values > 0.05) for “new courses”. However, this does not hold true for the other categories. 47 When looking at the data for the US Partner Scientists, and determining whether gender is related to any of products related to CP supported by PEER, there does not appear to be sufficient evidence to suggest there is no significant relationship. However, it is probable that gender is significantly associated with the production of “new material” (p=0.054) . Table 19: Survey data on Perceptions of PEER Benefits for PIs/Co-PIs and US Partner Scientists Benefits Identified PI/Co-PI US Partner Scientist PEER help me produce: New courses 19% 11% New training 41% 37% New training material 32% 22% Stakeholder workshop/training 49% 21% New workshop 42% 30% New MS/PhD concluded for student 41% 14% New MS/PhD Thesis concluded for self 6% - INSTITUTIONAL CAPACITY Infrastructure was often cited as a need. PEER has provided equipment to many projects, but the need likely goes beyond what PEER can provide. Creation of new Indonesia Universities and US Universities with a $20 million funding for these partnerships called SHERA (Strategic Higher Education Research) - it is somewhat clear what the causal relationship between this program being created and the presence of PEER but USAID Indonesia seems to think there is at least a correlation - this partnership is focused on building institutions capacity for the country by having longer partnerships and with larger amounts. The example of Kenyatta University cited in the Policy section speaks to the influence of PEER on institutional development. PEER improved faculty members’ capacity in proposal writing, managing grants, and research skills. To respond to increased faculty demand for grants management, one university administrator interviewed said that he has also increased the university’s capacity to manage multiple grants and to help faculty manage their grants more effectively. To respond optimally, he has re-structured his office to created new divisions for research support into 4 units, Finance and Development, Administration, Academic, and the new one only 2 years old, Research Innovation and Outreach, all with a focus on faculty support. CONCLUSIONS EVALUATION QUESTION 1 – TO WHAT EXTENT IS THE PEER PROGRAM BEING IMPLEMENTED EFFICIENTLY? COMPARISON TO OTHER PROGRAMS Each of these programs’ objectives and structure are similar to PEER fostering international collaborative research to address developing country challenges. Yet, there are several elements in these programs’ 48 management structure and processes which could be considered either as alternatives or complementary features to of the PEER program to improve PI and Co-PI productivity and capacity building. These elements fall into three main categories: (1) Donor-partner and Host-country partner project co-ownership thoughout the project cycle; (2) Promoting and funding face-to-face international researcher collaboration; and (3) Faciliating researcher and stakeholder convenings. Donor-partner and Host-country partner project co-ownership thoughout the project cycle  Funding allocated to both donor-country and host-country institutions  Co-creation of research proposals between donor-country and host-country researcher  Co-implementation of the project All three comparator programs are designed to foster co-ownership of the project through cost sharing, co-creation of the research project, and sharing in the implementing of that project. For example, the FTF Innovation Labs cite their strong relationships with Missions built over many years and now managed contractually through the Leader with Associate mechanims, as a huge strenght from which Missions are seeing the benefits from investing in research. It has also helped with scaling up new technologies. PEER does engage opportunistically with Missions for buy-in support, but they do not have as strong of a history of interaction an contracting with Missions. In addition, the PEER program, by design, does not require the U.S. partner scientists’ equal participation in the project compared to the host-country PI. While there is collaboration, the type of collaboration varies considerbly across the PEER portfolio. This is in part due to incentives the individual PEER PI has to invest and “co-own” the research with the U.S. partner scientist. Alternative ways to strucure the matching of PIs, co￾development of proposals, finanical resposibility, and implementation could foster more interaction and collaboration between the PI and U.S. partner scientists. This could allow the PI to leverage U.S. partner scientist resources which could enhance the productivity of the research as well as spill over effects like expanding the PI’s networks. The kind of engagement with partners fostered through the FTF Innovation Labs has also seemingly influenced engagment with policy makers and the ability of the FTF Labs to track impact on policy change. This includes both on the US side with USDA and university partners, as well as policy makers in countries where research is funded through specific mechanisms designed to bring them to the table. PEER may want to consider how it can better foster this kind of engagment in support of its policy and program change goals. Additionally, promoting face-to-face international researcher collaboration can change the nature and mutual benefits of the collaboration between the PI and U.S. PI. Through international exchange the U.S. partner scientist and PI gain access to each others’ research sites and equipment and fosters more opportunities for knowledge sharing. All of these potential outcomes through face-to-face exchange could increase prouctivity. These types of exchanges do occur under PEER; however, U.S. partner scientists are not provided with any budget to pay for their travel. As is done under the Horticulture Innovaiton Labs, including budget line item for donor-country partners to travel to the host-country research institution or site could encourage these face-to-face international exchanges. 49 Another way to encourage increased collaboration, networking and knowledge sharing is through researcher and stakeholder convenings. The Feed the Future Innovation Labs facilitate annual meetings for project stakeholder including manager, PIs, students, and collaborators. Through these meetings PIs network with other scientists in their country or region expanding their network. Although WOTRO is just working to increase their capacity to engage policy makers, they have committed resources to convening stakeholders around policy in order share knowledge and to map the level of influence each has on the others. STAKEHOLDER PERCEPTIONS Overall, the data suggests PEER PIs expected to grow their network, both in size (new relationships) and in depth (existing relationships) as well as to increase technical skills. For the US partners, they primarily hoped to contribute to building the capacity and new relationships. Both groups seemed to be interested in increasing networks and scientific capacity. When looking at peoples’ responses about experience, about 80% of the PEER PIs/Co-PIs and about 60% of the group for US Partner Scientists stated that PEER was a positive experience. However, when looking at “satisfaction” (often defined as the combination of expectation and experience), the US scientists seemed far more unsatisfied (17% combine unsatisfied and very unsatisfied) then the PEER PIs (5% combined). For the US scientists, sex did not appear to be associated with satisfaction. Similarly, the data did not find enough evidence to support a significant association between type of agency and satisfaction level. PEER CYCLE PROCESS Whether PEER should or not engage in these activities is discussion that should take place with each Mission. In interviews with Mission staff, they expressed a general preference for PEER projects that have the potential to contribute to existing projects, programs, or strategies, which may limit the level a Mission will invest in an outside program, as higher risk projects with higher failure rates may have broader implications for related activities. If Mission staff is more engaged in strategically planning for how PEER can support them and in selecting the specific PEER activities that are funded, as they have stated, regardless of whether they buy-in or not, then they may be willing to support higher risk endeavors. However, the implications of higher risks must be clearly communicated to upper management as well as Mission staff. It could mean the funds placed on a project (with high risk high reward) will yield zero benefit. Missions are risk-averse and in general want to see impact on development within very short time frames (1-3 years max). Each discussion with Missions during proposal stage should make jointly with the buy-in unit decisions about the level of risk that is acceptable, if any is at all. Issues around delays due to equipment purchasing can be addressed by PEER through changes in the way projects with purchases (of unique or complex equipment/infrastructure needs), should have: 1- A specific person on staff at NAS and USAID to manage their processes and keep in close contact with them through their procurement process 2- Determine early on, shortly after the PI has been awarded, where the issues might arise, including budget changes due to price changes at the time of purchase 50 3- Put in place a contact with the Mission, if needed, to help reduce the VAT costs and other tax fees that might hinder the timeline of the project USAID staff in both Washington and the Missions need to be aware of what is required of them to facilitate timely implementation of research as some things are beyond the reach of NAS (e.g., customs issues affecting importation to. equipment). In addition, it is not always clear who is best positioned to purchase equipment, the PI, U.S. Scientists, or some other stakeholder. Both PEER scientists and USAID Mission staff both voiced the opinion that it be helpful to anticipate equipment purchase issues at the outset to iIdentify them, make Lab and Missions aware of pending actions of tax exempt letter or other solutions. When facing issues related to equipment, as well as the application of PEER research to programs and policy, PEER researchers would appear to benefit from greater engagement of USAID PEER Washington staff with USAID Mission staff partners when they do site visits. Instead of "monitoring" PEER projects as NAS staff do, they should spend more time with Missions, to better understand their priorities and to iron out challenging issues. EVALUATION QUESTION 2 – TO WHAT EXTENT HAS PEER HELPED FILL EVIDENCE GAPS WHICH INFLUENCE POLICY/PROGRAM CHANGE? PRODUCTS PEER PIs appear to be sharing among peers their findings, attending meetings (which is likely where they are sharing), and believe their information is improving evidence in their field. This is a very common archetype for a scientist as their incentives are to present their research to peers for input consensus and expansion of their field as well as to produce publications for career advancement. The focus of a scientists often is not on policy change, in the case of PEER PIs because of time (projects are too short) or funding (not enough in the budget to engage decision-makers) or insufficient training to share their findings to non-technical audiences. These factors plus equipment delays are likely hindering the size of PEER impacts. However, PEER funded scientists and other stakeholders highly value the potential for and actual impact on policy from PEER research, even more so than capacity building outcomes, as evidenced in interviews and in the Most Significant Change selection process. Data suggested both PEER PI and US Partner feel training in peer-reviewed journal article writing is wanted and needed – and thus this may explain the low number in published articles if capacity is limited. However, there does appear to be a significant amount of journal writing as 36% of PEER scientists stated they had articles under review. This metric of productivity has not been captured by PEER, and it may be more relevant of scientific productivity as publications take time; PEER scientists themselves stated they felt they would have time to submit an article but had uncertainty about it being published before the project was completed. When looking at the distribution of products for the PEER portfolio, based on multiple data sources, a large portion falls within the field of Environment (which captures Climate, Biodiversity, Water, Disaster Mitigation, Energy and Biotechnology). The rest of seem to cluster in Health. This is due in part to: 1) a larger portion of PEER projects in two countries (these represent 27% of the entire PEER portfolio), and 51 2) the evaluation team had site visits at these locations with more in-depth conversations about PEER projects. Surprisingly, 19% of the PEER PIs reported creating open access data and 16% said they had produced blogs/vlogs/videos with scientific instruction. It is unclear if this online presence includes links to datasets for sharing. This may be a possible area for growth – particularly– as more and more scientist are e-connected and scientific information is created with multiple types of media. Environment projects often could be described in one of three categories: technology, inventory, or resource management. In some cases, projects were a combination of both and their products reflected that mix. For health, these tended to be more applied, which usually were created to help inform an existing standard/process/approach or to provide guidance for a health issue. Health projects almost always have combinations of community engagement, government involvement, and controlled testing. However, most products appear to be related to the capacity building aspect of PEER, and less about research productivity. PEER scientists often produce at the end of the project, and impact is likely after project completion. Thus, data collection for products and impacts is mismatched with actual levels, and it is possible looking at productivity 1 year post completion might reveal larger effects than currently known. Publications take longer (weeks to month depending on the field), policy impacts are usually after the evidence has been vetted by peers/or provided at opportune moment’s (policy window), and programmatic uptake usually comes after the field accepts the findings. Most projects are a combination of sectors and contextual factors. The oversimplification of these projects, although useful for high level analysis, can omit the interaction effects that are likely causing the outcomes/impacts. For example, as previously cited in the Findings section of “Benefits and Challenges” related to evaluation question 3 on collaboration, PI participation at a workshop in Lima has led to the use of a common monitoring framework on biodiversity in the region. It would be a major advance if all local governments in the Amazon region adopted a common framework like this. The logical next step would be to advocate for the use of these indicators for this purpose. This kind of policy level impact resulting in agreements on international standards can be a reasonable expectation of outcome based on research, which may come after PEER research introduces the evidence and concepts. Research scientists may not necessarily be the ones engaging in advocacy, but they can be the catalyst for this kind of change. POLICY CHANGE The causal relationship between PEER projects and policy change impacts is not always clear, nor could the data provide evidence for this. The data did provide evidence of connections between the presence of PEER and policy change. The projects showing impact or connected to policy change were those who often had been in earlier cycles. The projects showing policy change or potential for policy change from earlier cycles suggest this type of impact is not happening within the lifespan of the project or is less likely to be visible then. As would be expected, many of the PEER projects that have demonstrated application of findings to application are from early cycles, some of which have already complete. It is not surprising that completed projects would have more to offer to policy makers and program implementers than those in early stages. Also, the early cycles included more health projects, many of which were designed as operations research with clear programmatic or policy related outcomes incorporated into the design. 52 As a result, there are also health projects in earlier stages that also are designed to speak to policy and practice. In some other areas of research, the type of research may be more formative and have less clear applications until they farther along. The low number also may be in part to sampling bias as many were selected due to pre-existing conditions. Beyond research design there are several factors that may play a role in uptake of findings, including the researcher’s social network, effectiveness of his/her communication skills, especially to audiences outside of the scientific community, and the degree to which the enabling environment is supportive of evidence-based policy making. Also, the projects that appear to be affecting policy change had from the start direct engagement with their government entities, or had PIs connected to government. Some PIs believe that policy impacts are not achievable within the PEER project length or budget. To affect policy change scientists stated they would need funds to convene policy-makers, USAID missions to bring to the table the key parties, and longer time periods for visible effects. USAID Mission staff agreed consistently in interviews that convening and engaging policy makers and Mission staff in some way, more consistently and frequently was important, but desired that PEER staff take a lead role in initiating this engagement through events that would bring policy makers and program implementers together with PEER scientists. In addition, management staff from both NAS and USAID/Washington PEER team have differing opinions about how to collect or whether it is possible to measure policy change. Furthermore, the incentive to produce policy change at the national level may not be present for some PIs. Although there is a high level of interest in creating solutions for their communities, they are also bound by the metrics for advancement within their field – publications. Conversations with PIs in Kenya suggested the funding amount did not permit the type of outreach and engagement needed to affect policy. To do so the project would require additional funds. They suggested condensing these for all projects in Kenya and working with the mission to host a country level meeting bringing together government officials. The expenses in Kenya included funds to pay government personnel to attend the event, which if not provided would not attend, but not always understood by US partners. By also bringing together the multiple projects in PEER under one venue, the agencies can see the benefit of the research activity. Indonesia also stated similar views to Kenya. Yet, for this group of PIs, they suggested USAID be the ‘convener’ that a ministry official was more likely to respond to a request by USAID for an event that to a faculty member at a university. By leveraging their ability to bring key government officials, the Mission/USAID could provide the right environment for the PIs to present their work. They also stated they would like more capacity to communicate with non-technical decision-makers, stating that this was key in helping them understand the value and use of their research. This skill was key in ensuring uptake of their work and in likely affecting policy. The views expressed by researchers in Kenya confirm a finding in Indonesia and Kenya that when host￾country PIs have a very high level of engagement with their governments, there is more likelihood that their research will be of more immediate interest to policy makers. For instance, three health projects. Baby- friendly/breastfeeding initiatives in both Kenya and Indonesia are working with the MOH and district hospitals to produce the evidence that will contribute to more effective implementation of national policies. It worth noting that both research projects are focused on building upon a well￾established research base generated in other countries. The purpose of the research in Kenya and 53 Indonesia is to demonstrate that the evidence-based practices they are testing are also applicable in their respective country contexts. The survey data shows some contradiction about what participants perceive the use of PEER products to be compared to the actual products produced for specifically related to policy. When asked about product sharing, less than a third said they shared with government agencies, yet 56% said they believe their products are being used as information for improved decision-making. There is a possibility they are communicating their products informally to government officials, and thus did not count it when answering the survey question. It is possible their perception reflects their “hope” for use of the products. The ambiguity exists around what policy change means, and how it is defined by the respondent, management, and the cost to create it. Interviews with the management entities themselves revealed different views about the viability and process to measure policy change. Long stall in equipment delays leads to shorter timeframe for impacts and less likelihood of seeing the types of things PEER measures. Limited flexibility, especially for projects with large equipment purchases or unique types of technology, will likely decrease the outputs over the lifespan of the grant. US Partners stated a low interaction with USAID which could be why their responses about sharing their products with USAID were also low. However, having the US partner in the US has the advantage of being able to engage, potentially directly with NAS and USAID PEER in DC. This could be a mechanism for PEER PIs to share their work with USAID and increase the types of benefits for the US PI by exposing them to the USAID network which could yield other opportunities. Also, USAID gets to be exposed to new technology and science without. The role of USAID for the US Partner is nebulous and there are advantages for both groups in improving this interaction. USAID staff in Indonesia described another way that they have raised the profile of PEER within the country that has the potential to raise buy-in from other Missions abound the world. PEER projects are often featured prominently on the list of places visited by the Ambassador and by other visiting government officials. According to Mission staff, PEER projects demonstrate a very positive type of investment in Indonesia that is highly prized by the government and the population as a whole. Whenever the Ambassador makes a trip to different parts of the country, he always asks if there is a PEER project he can visit. Although this is not a direct policy impact, these visits and others by high level officials have the potential to open policy dialogue around health, the environment, energy, and food security, which are all focus areas of PEER research. EVALUATION QUESTION 3 – TO WHAT EXTENT HAS PEER PROMOTED COLLABORATIONS BETWEEN US SCIENTIFIC COMMUNITY, LOCAL RESEARCH, AND USAID MISSION, EMBASSY STAFF, AND PRIVATE SECTOR PARTNERS? FORMATION OF PARTNERSHIPS The value of the partnership between PI and US partner scientists that is the foundational relationship in PEER appears to be borne out by the findings. Large majorities of PEER PIs and US partner scientists 54 concur that the relationship has been of value both for the scientific output, strengthening professional relationships, and advancing careers. Additionally, many participants in the US and abroad stated that PEER has also contributed to developing solutions to problems in health, the environment or agriculture. PEER has also strengthened other types of relationships among PIs, US partner scientists, national and local governments, USG agencies, USAID, and NAS. Students have benefited from these relationships as well (see next section on capacity building). INCENTIVES The incentive structure for the PIs is clearer than for the US partner scientist members of the partnerships. The PIs benefit by getting funding for their research, can travel to a few international conferences and benefit from exchange of technical knowledge and methodologies with scientists in the US and beyond. US partner scientists benefit in some instances from access to new datasets, different visions, and perspectives on share research topics. The also benefit from greater access to policy makers. Students from the US and from the host countries have had some opportunities to participate in student exchange programs at the universities of the partner scientists. There are also limitations that have the potential to act as disincentives. One major constraint is the requirement that PIs collaborated with scientists with a USG funded grant. This is challenging in some fields where there is not a good fit between the types of research funded by the US counterpart organization and local research and programmatic needs. Although almost 50% of PIs partnered with someone they had known previously or with someone connected to someone they knew, the matching of PIs and US partner scientists did not always reflect the best options for the PI. In interviews in Indonesia and Kenya, several PIs said that they had to forgo their preferred candidate because he or she did not have a grant from a participating USG agency. This was particularly true for maternal and child health projects, and some climate change projects. Preferred options for US partner scientists often had grants to support research related to the PI’s research, but they were from other sources, such as foundations. NETWORKING Growth of professional networks was the number one reason for why most PEER PIs competed for the grant. Both PIs and US partner scientists stated that their professional networks have been strengthened by participation in the program. Nevertheless, the findings also demonstrate that successful awardees are already very well connected, and those that are most widely connected in their fields are awarded the largest amount of funding. Clearly the PIs are already connected to US scientists, so the question then remains do applicants with no connections have less chance of winning an award? The evaluators were not able to interview PIs who had competed but were not successful. The findings indicate that, for the most part, PEER is not a program for young and relatively inexperienced researchers, except insofar as they are part of PIs’ research teams. Men and women are also “…because collaboration is one of the drivers in research output and scientific impact, programmes fostering international collaboration for female researchers might help to level the playing field” Lariviere et al. (2013) 55 not participating in PEER in equal proportions. Survey data had 70% male respondents and 30% female. Research shows globally 73% are male and 27% female20. When looking at publications, this number is even lower; recent work published in Nature suggests less than 6% of countries on the Web of Science have gender parity concerning papers published. Consequently, it is hard to argue that PEER is meeting USAID gender and youth equity policy objectives. Interestingly, female PIs do not seem to differ significantly from their male counterparts in terms of the size of their professional networks.21 It is possible that PEER is not fostering more collaboration with US scientists beyond collaborative relationship but MAYBE this award increased resources to make those relationships more active, and to engage with them over a sustained period of time during the PEER grant. It also legitimizes and elevates awareness of them as internationally recognized scientists within their home institutions and their countries more broadly. BENEFITS AND CHALLENGES The survey results demonstrate that there are clear benefits to PIs from receiving a PEER grant. In many countries, PEER has contributed to increased collaboration among national scientists and research institutions. Given that more than 50% of PEER partners knew each other prior to PEER, it is not surprising that the program has not had a greater effect on creating new relationships. The real contribution of PEER to the PIs appears to be their enhanced capacity to compete for new funding and to advance in their careers. As an outcome of PEER collaboration, a Kenyan PI and US partner scientists have now submitted a joint grant for additional funding. PEER does not appear to have a large effect on teaching, in terms of PEER grantees being able to attract more students. It appears to have a strong impact on their recognition as leading scientists. Table 20:Contribution of PEER Grant (PI/Co-PIs responses) Contribution Strongly Disagree Disagree Neither agree or disagree Agree Strongly Agree Rating average Response count Leveraged new funding 3 5 33 67 66 3.32 174 Career advancement 3 1 4 57 110 3.59 175 Institutional recognition as a leading scientist 1 6 24 78 65 3.33 174 Increased number of students in classes 6 11 74 56 25 3.01 172 recognition as a leading scientist y colleagues 3 3 23 82 64 3.31 175 EVALUATION QUESTION 4 – TO WHAT EXTENT HAS PEER STRENGTHENED CAPACITY IN DEVELOPING COUNTRIES? 20 Lariviere, V. et al. (2013). Bibliometrics: Global gender disparities in science. Nature Vol 504, 211-213 pages, December 12, 2013; Women in Science UNESCO Institute for Statistics UIS Fact Sheet, Nov. 2015, No.34. 21 More analysis will be done comparing differences between men and women. 56 PIs and students appear to mutually benefit from student participation in PEER. The perception among the majority of PIs and U.S. partner scientists that having access to skilled students is an important factor in their productivity signals that student participation is PEER research is an essential component of the PEER research model. The FGDs and interviews with PIs also indicate myriad ways that students are benefiting from participation in the project through exchange visits, by expanding their networks, or gaining research experience which could contribute to the advancement in their own careers. While, there is evidence that PEER is benefiting students there is limited data collected on factors contributing to building student capacity. The current M&E framework includes an indicator on # of students trained defined as “Students trained denotes students participating in courses or degree programs designed by PEER PIs plus research assistants, tracked under Sub IR.2.1, indicator 4 of the PEER Results Framework, (undergrads, grad students and post-doctoral fellows) assisting with activities related to the PEER grant.” This indicator does not distinguish between students participating in one course verses those who may access greater benefits through longer term participation in the research project. Other outputs, like number of students completing theses or dissertations leveraging PEER research resources are not collected systematically. Gaining access to new skills, being part of a research team, expanding one’s networks, or leveraging PEER resources to complete a thesis or dissertation can all contribute to the advancement of one’s career, particularly in low resource environments. For example, the data shows that this expansion of student’s networks has facilitated students’ access to advanced degree programs. However, PEER is neither designed to explicitly respond to perceived needs of undergraduate and graduate students in developing country contexts to help them achieve their professional goals nor is it tracking data on the factors positively contributing to their professional development. These benefits are dependent on the resources the PI or U.S. partner scientists invests in any particular student. Greater understanding of what student’s needs are or barriers to accessing benefits would allow the PEER program to design and monitor factors which contribute to PEER student capacity. 57 RECOMMENDATIONS EVALUATION QUESTION 1 No. RECOMMENDATIONS PERTAINING TO Q.1 Responsibility 1.1 Expand the type and number of partnership organizations to the private sector (e.g., General Electric) and foundations (e.g., Bill and Melinda. Gates Foundation) to offer PIs a wider choice of partners among U.S. Partner Scientists) USAID 1.2 PEER funding should include a travel budget for the US Partner Scientists, when needed, to increase face to face interactions between PIs and US. Partner Scientists. USAID and NAS 1.3 Augment the size and flexibility of the grants by increasing the ceiling (to $500k), raise limit of purchases of equipment, and extend the life of the project to five years, or allow for more options for no-cost and costed extensions USAID 1.4 To increase the number of women and you scientists, increase emphasis on mentoring targeted at these groups and address barriers they face during the proposal period and for travel. PEER may consider funding cycles just for women and young scientists. USAID and NAS 1.5 Provide a 3-6 post-award planning period to address issues around permits, infrastructure, equipment, procurement, staffing, and other factors with the potential to delay research. USAID and NAS 1.7 To increase engagement of USAID Missions with PEER researchers:  Prioritize and standardize functions of TDY Supervision trips for NAS and USAID. NAS should focus on supervision of PEER research, while USAID Lab staff focus on improving connectivity between the Lab and Mission staff and to better align PEER research awards with Mission programs and policy dialogue.  Send staff from Missions with low PEER engagement to Missions with high PEER engagement so they can see the benefits first hand. USAID 1.8 PEER should continue an analysis beyond the parameters of the midterm evaluation of the rich data collected by the evaluation team, including comparison analysis by agency; testing of scientific productivity models and determining significant variables of PEER productivity; and to create and publish peer-reviewed journal articles with the data/findings USAID 1.9 Create a special track for technology projects with high risk of failure or high risk of delays (such as those with unique equipment purchasing needs) but high potential for substantial innovation. NAS 1.10 Use PEER to create “Research Assessments” of the countries they target for the benefit of the Mission, but also to ensure a more cohesive match between Mission needs and priorities, country focus areas, and researcher interests. By providing a profile to each Mission of the ways in which research and science can improve their development outcomes, and in which specific areas, would increase PEER participation. NAS 58 EVALUATION QUESTION 2 RECOMMENDATIONS PERTAINING TO Q.2 Responsibility 2.1 To increase policy change outcomes, review grants and categorize by: 1) level of connectivity to government and 2) whether the projects have characteristics or expected results with the potential application for policy change? NAS 2.2 Consider two options for redirecting funds towards policy and program-oriented research: (1) increase funding specifically for policy-oriented research; or (2) hold special calls to support policy- or program-directed research, perhaps as a pilot, to explore the significance of certain types of research for informing policy and program design and implementation. USAID/LAB, USAID Missions, and NAS 2.3 Recognize that program or policy change is not always the most effective and efficient way to achieve, or measure research results. Consider conducting joint assessments with Missions to determine how research can best serve the country’s development needs, which can form the basis for calls supported by PEER and the Missions. NAS and USAID/Missions 2.4 Include “number of articles submitted” and “number of articles submitted and rejected” as part of the metrics for Scientific Productivity and PEER Outcomes. Create incentives and opportunities to capture evaluation data post-project periods. NAS 2.5 The capacity of PEER funded scientists to communicate and advocate effectively for program or policy change is inconsistent and inadequate. PEER needs to consider focusing resources on building capacity through its grants and other support structures to develop this skillset or assisting them with finding advocacy partners similarly to the way PEER assists PIs to find US science partners. NAS EVALUATION QUESTION 3 No. RECOMMENDATIONS PERTAINING TO Q. 3 Responsibility 3.1 Facilitate the broader dissemination of research findings by convening PEER grantees, the private sector, government officials, and civil society partners together to network and share findings and policy and program challenges. This could include a roundtable series to discuss the activities and areas for synergy to broaden Mission and host country government engagement. NAS and USAID Missions 3.2 Create country-wide and/or field-specific PEER conferences hosted or convened by Missions as outreach events with government entities and as opportunities for scientific sharing in support of communities of practice. NAS and USAID Missions 3.3 Create a virtual space for PIs such as a PEER “YouTube” channel to share with both their PEER colleagues but also with the public. Request every PI create one instructional video per project, with guidance on structure and content to support capacity building and teaching. NAS 59 EVALUATION QUESTION 4 RECOMMENDATIONS PERTAINING TO Q.4 Responsibility 4.1 Provide training in journal article writing by leveraging US Partners’ expertise at PEER workshops or conferences. NAS 4.2 Make draft peer-reviewed journal articles from students a deliverable through student swaps. Send PEER PI students to the US partner scientist’s Lab (or vice-versa) for a semester (or more) and increase the likelihood of scientific paper productivity. NAS 4.3 Enhance data collection about students to better capture PEER impact and benefit. NAS and USAID 60 REFERENCES Cerna, Lucie (2013) "Synthesizing the Implementation Literature: The Ambiguity-Conflict Model of Policy Implementation," OECD Report. Paul Cairney and Tanya Heikkila in (eds) (2014). A comparison of Theories of the Policy Process” Kingdon, J. (1984). Agendas, Alternatives, and Public Policies, New York: Haper Collins Matland, Richard (April 1995) "Synthesizing the Implementation Literature: The Ambiguity-Conflict Model of Policy Implementation," Journal of Public Administration Research & Theory. Sabatier, P. and Weible, C. (2014) Theory of Policy Process, Third Edition Chicago: Westview PRESS. Sandfort, Jodi and Stephanie Moulton (2014) Effective implementation in practice: Integrating public policy and management, Jossey Bass and Wiley and Company Publications. Thaler, R. and Sunstain, C (2008) Nudge, London: Penguin Books. . 61 ANNEXES 62 ANNEX I: EVALUATION STATEMENT OF WORK Background Overview of program to be evaluated The program, Partnerships for Enhanced Engagement in Research (PEER), was initiated in July of 2011 through an unsolicited proposal from National Academies of Science to support research projects conducted by developing country scientists in partnership with researchers supported by US federal science agencies. The PEER program, implemented by the National Academies, is managed at USAID by the Research Partnerships for Development (Research) team within the Center for Development Research (CDR) of the Global Development Lab (Lab)22. Table 1. Project Summary Partnerships for Enhanced Engagement in Research Start-End Dates July 2011- July 2021 Obligations as of September 30 2015 $55 million Ceiling $100 million Prime Implementers National Academies of Science Since its launch, PEER has supported 205 projects in 45 countries around the world. NSF was the founding US science agency partner in PEER’s first cycle, followed by the addition of NIH under the PEER Health program and six additional agencies in PEER Cycle 4 including; Smithsonian Institute, NASA, USGS, and U.S. Department of Agriculture’s ARS, NIFA, and USFS. NOAA came on board as a participating PEER partner in Cycle 5. The program currently has nine participating federal science agencies. This count does not include NIH Institutes/Centers that have also had a prominent role in the PEER program including the National Cancer Institute (NCI), Fogarty International Center (FIC), and the National Institute for Child Health and Human Development (NICHD) or the DoD’s Office of Naval Research which bought into the program in Cycle 3. In Cycle 5 two private sector partners were also added. PEER has three strategic objectives (3Cs): I. Critical Evidence - Advance quality research that will fill evidence gaps needed for improving programs or policies in developing countries II. Collaboration - Advance and build existing relationships between local research 22 The LAB is the newest USAID office, established in 2014. See https://www.usaid.gov/GlobalDevLab/about 63 institutions, USG partners, host country government, and USAID Missions and Embassies III. Capacity – Build research capacity in developing countries thereby enabling local solutions to context specific challenges Implemented by the National Academies, PEER seeks to achieve these objectives by directly supporting collaborative research projects lead by local PIs in USAID priority countries. Not only does PEER support research relevant to USAID’s development objectives, it also aims to establish long-lasting research relationships globally, build the capacity of local scientists and engineers to conduct high-caliber research, and enable scientists to become better partners in development. Additionally, the program seeks to leverage and mobilize the US scientific community and the USG investments in research already made - as represented by formal partnerships with US federal science agencies- to help USAID achieve development objectives in multiple sectors. The target sectors are: biodiversity, health, agriculture, environment, water, disaster mitigation, climate, education, food security, and energy. Since 2011 the PEER program has released 7 individual solicitations referred to by program staff as: PEER Science-Cycle 1, PEER-PIRE, PEER Science-Cycle 2, PEER Science-Cycle 3, PEER Health-Cycle 1, PEER Health-Cycle 2, and PEER-Cycle 4. After the third year of PEER, a strategic shift was made by USAID to integrate separate PEER Science and PEER Health work-steams into a single ‘PEER’ program. In additional to annual solicitations the program has also supported 3 annual meeting for participating PEER PIs in various regions including: Bangkok, Arusha, and Lima. These regional meetings include capacity building and networking activities as well as the opportunity for PIs to connect with USAID Mission, Washington, and NAS staff. The next PEER Forum will be held in Amman, Jordan in March 2016. In addition to the PEER Forums, NAS also hosts at least two financial management trainings per year for administrative staff at institutions with PEER grants. In order to fund those projects with the highest likelihood of achieving development impact while building local research capacity, PEER supports an annual solicitation and review process. The timeline for this process was standardized in PEER- Cycle 4. The Annual solicitation includes various research “focus areas” with priorities specific to a technical area like biodiversity, health, wild-life trafficking, among others. The PEER solicitation is released in early October. Pre-proposal submissions, including a letter of support from the applicant’s research collaborator who is supported by one of the USG agency partners on PEER, are due in January. USAID leads a review of these pre-proposals in early February that is based on the relevance of their work to the USAID Mission’s development objectives outlined in their CDCS or RCDS. Proposals deemed relevant by the Mission are invited to submit a full proposal. NAS leads an NSF style technical merit review of full proposals in May/June of each year. July allows time for back and forth with Missions to make funding decisions. Final awardees are announced in August. In addition to ‘core’ funds put in annually by the Global Development Lab, PEER also accepts ‘buy in’ funds from both central and field operating units across the agency. The mechanism has also been used by external partners like the National Cancer Institute and Office of Naval Research to sponsor specific PEER awards. USAID Buy-ins are typically incorporated as ‘focus areas’ in an 64 annual RFA. Focus areas reflect research priorities of the funding operating unit and may cover a range of sectors including: biodiversity, energy, wild-life trafficking, water, and maternal health, among others. In September of this year, PEER was approved for a no-cost extension of 5 additional years, allowing the Cooperative Agreement to run for a total of 10 year. As of July 2016, the PEER program will reach its 5 year ‘mid-term’ mark, approximately 50 out of 200 projects have been completed. This mid-term evaluation is crucial to help inform the next five years of PEER, as well as follow-on or ‘spin off’ programming that supports Intermediate Result-2 and Intermediate Result-3 of the Science Objective Results Framework (see Figure 2). Additional information can be found on the National Academies PEER website at: www.nas.edu/peer III. (b)- Results Framework In consultation with NAS, USAID/PEER developed a theory of change (Figure 1) to align PEER activities with USAID strategic objectives. The PEER logic model outlines a theory of change though which financial support, capacity building activities, and research collaborations, PEER PIs complete high quality research projects. The evidence resulting from PEER research (hopefully) leads to policy/program change and improved development outcomes. The PEER programmatic framework also contributes to the Global Development Lab’s Science Objective Results Framework (Figure 2), specifically indicators IR2 “Locally focused R&D for development increased” and IR3 “Science Ecosystems strengthened”. The highest development objective outlined under the Science Objective RF is “research results that influence policy or program change”. Figure 1- PEER Theory of Change 65 Figure 2 - Science Objective Results Framework III. (c)- Monitoring data collected Since the PEER program has been running for over 4 years, a significant amount of monitoring data has been collected annually on the program to assess performance against objectives (See attached Appendix 1). The Contractor will have access to all the monitoring data collected for the program. SCOPE The task is to conduct a Mid-term Performance Evaluation of the first (5) five years of the ten (10) year PEER project. The evaluation should address all evaluation questions included in attached Appendix 2.This task order will be managed by LAB/CDR. PERFORMANCE OBJECTIVES A. This task order will contribute to USAID’s and LAB/CDR’s broader objective of rigorously and credibly documenting programs’ effectiveness to inform decisions about current and future programming. Specific objectives for this work include: 1) Evaluate the implementation performance of the PEER program (the ‘how’) 66 namely operational procedures, operational efficiency, and management structure. 2) Assess how the PEER program is perceived and valued by key stakeholders such as PEER PIs, Universities, USAID Missions, host country government ministries, and local NGOs. 3) Provide evidence of the extent to which the PEER program is working and producing real change (the ‘what’), specifically whether the program is making progress towards achieving the targets in the PEER and Science Objective RFs (see Figure 1& 2 above). 4) Provide action-oriented, practical and specific recommendations to improve the implementation of the PEER program and inform decisions about current and future programming. 5) Identify and detail strategies to promote public and private sectors adoption of PEER- supported R&D B. Data Collection and Analysis In addressing the task objectives, the contractors should answer the evaluation questions included in attached Appendix 2. In terms of method, the offeror will need to collect qualitative information and combine it with project monitoring documents and quantitative data. This includes gathering financial data that permits computation of unit costs and analysis of cost structure. Specific data gathering activities must include: Required Activities a) Desk Review of PEER program documents, baseline and monitoring data b) Review of programs similar to PEER; analysis of PEER strengths and weaknesses compared to these programs c) Interviews with key actors: i. Project participants: PEER PIs and co-PIs, USGs partners ii. Partner organizations: universities, local science academies and USAID Missions iii. Key audiences for PIs’ research such as host country government ministries, industry organizations, private companies d) Site visits (minimum of 2: Indonesia, + another country) 1) Desk Review Required Sources: The Contractor must collect and review relevant documents, including but not limited to: a) Baseline survey data of PEER PIs (collected by NAS); b) PEER Annual and Quarterly Reports from PEER PIs (collected by NAS); c) Monitoring data collected and tabulated from PEER Annual reports; 67 d) Quarterly reports submitted by NAS to USAID; e) PEER site visit reports and photos, and video; f) Official ‘AOR’ files: including initial unsolicited NAS proposal, Cooperative Agreement and Modifications; g) Surveys, communication documents, one-pagers and other ‘one-off’ documents developed for specific solicitations under the PEER program; and h) Program documents and evaluations of programs similar to PEER23 2) Interviews with key actors: a) NAS PEER staff (current and former) (Washington DC based) b) USAID PEER staff (current and former) (Washington DC based) c) Implementers of programs similar to PEER d) Project participants: i. PEER PIs and co-PIs (located in LMICs) ii. USG Partners of PEER Awards and USG Partner Agency points of contact iii. Trainees: undergraduates, graduate students and post-docs, supported by PEER awards e) Partners: i. Deans, Department chairs, University officials at PEER supported universities ii. Leaders of local Research Institutions iii. Program and technical staff at USAID Missions and Participating USAID OUs (located in Washington DC and LMICs) f) Audiences for PIs’ research24: i. Host country government ministries, ii. Industry organizations, iii. Private companies in relevant sectors iv. Local NGOs 3) Site visits The Contractor must conduct site visits to at least two (2) developing countries, one should be Indonesia. The offeror must identify and explain the selection of the second 23 Comparable programs that may be considered are: Higher Education Solutions Network (HESN) https://www.usaid.gov/hesn, African Science Academy Development Initiative (ASADI) http://www.nationalacademies.org/asadi/, Belmont Collaborative Research Actions, http://www.belmontforum.org/collaborative-research-actions 24 The Evaluation team should consult with PEER PIs and USAID Mission staff to identify the key stakeholders most relevant to interview in each country 68 country chosen. During these site visits, the Contractor will conduct in-person interviews and focus groups with PEER awardees and other stakeholders, including but not limited to, the following: a) PIs and co-PIs b) Trainees: undergraduates, graduate students and post-docs, supported by PEER awards c) Deans, Department chairs, University officials at PEER supported universities d) USAID mission staff e) Host country government ministries, local and international NGOs and other private sector partners. (The Contractor will consult with PEER PIs and USAID Mission staff in these countries to identify the key actors to be interviewed) Illustrative Activities and Data Collection Methods In addition to the required activities in the previous section, the offeror should propose additional activities and specific data collection methodologies to fully answer the evaluation questions in Appendix 2. These may include: a) Online surveys, b) Focus groups, c) Case studies, d) Mapping, or e) Social network analysis Activities by Evaluation Question: Activities are illustrative unless listed as a required activity above. Evaluation Question (additional detail in Appendix 2) Illustrative Activities To what extent is the PEER program being implemented efficiently?  Desk review of PEER program documents  Review of program documents and evaluations of programs similar to PEER; analysis of PEER strengths and weaknesses compared to these programs  Interviews in Washington DC with: • NAS PEER staff (current and former) • USAID PEER staff (current and former) To what extent has • Implementers of programs similar to PEER PEER helped fill evidence gaps which influence policy/program change?  Identify PEER projects with potential policy or program impacts. Interview key informants from government ministries, industry organizations, and USAID missions to verify impact and provide analysis on the nature of the policy or program change.  Write case studies of 3 PEER projects with policy impacts identifying the channel for policy impact, facilitating factors, barriers, etc. 69 To what extent has PEER promoted collaborations between the US scientific community, local research institutions, host country government, and USAID Missions and Embassy staff, and private sector partners? Online surveys and/or phone interviews to:  PIs and co-PIs  USGs PEER partners  Deans, Department chairs, University officials at PEER supported universities  Officials from local science academies and other local research organizations  USAID Missions and Embassy staff  Host country government officials  Companies in sectors in which PEER researchers are working To what extent has PEER strengthened research capacity in developing countries? Focus groups with:  PIs and co-PIs  Trainees: undergraduates, graduate students and post-docs, supported by PEER awards  Deans, Department chairs, University officials at PEER supported universities  Science academies and industry organizations Mapping exercise. For 2 countries, map key actors and institutions in the research ecosystem, identifying where PEER has been most and least successful in building capacity, and highlighting where PEER is best placed to provide additional capacity building. Note: contractors should use standardized data collection and recording instruments and maintain records from the evaluation. For example, if focus groups are conducted, the contractor must maintain and deliver all transcripts to USAID upon request. DELIVERABLES 1. Task Order work plan 2. Draft data collection instruments and interview guides 3. Final data collection instruments and interview guides 4. Draft mid-term Evaluation report 5. Final mid-term Evaluation report 6. Presentation to USAID & NAS PEER team and other stakeholders (slide deck) Contractor must identify instruments to be developed and used in their proposal. The mid-term evaluation report should meet the criteria for evaluation report quality in appendix 3. KEY PERSONNEL: The personnel team must include at minimum the following positions: 70 Position Name: Senior Evaluation Advisor (Team lead) – 1 Position Role and Responsibilities:  Oversee design and execution of evaluation activities throughout the life of the task order, including design of evaluation methodologies, data collection methods and protocols, and facilitation of interviews and focus groups, as well as data quality verification  Oversees the processing and analysis of data and leads effort to draft evaluation report Minimum Qualifications:  Holds at minimum a Master’s degree, PhD (preferred), in international development or related field such as economics, sociology, or evaluation.  Has at least 8 years of professional experience in international development including, at least, 5 years of work designing and executing evaluations of development programs with a strong preference for experience with science, technology, and innovation programs. Position Name: Data Analyst – 3 Positions Role and Responsibilities  Support the collection, coding, processing and analysis of quantitative and qualitative data Minimum Qualifications:  Holds a bachelor’s or master’s degree in economics, statistics, sociology, anthropology or related field  2+ years analyzing data for international development evaluations or programs The contractor may choose to propose additional team members, such as a subject matter expert or additional data analyst, in order to meet the objectives of this task order. In addition the offeror must work in close collaboration with the PEER Program Manager and Research team Monitoring and Evaluation Specialist. USAID staff may join the Contractor and participate in site visits and interviews.All contractors must provide a signed statement attesting to the lack of conflict of interest, or describing an existing conflict of interest relative to the Project being evaluated. Signed statement should be included in the offeror’s proposal for all teams members listed in the proposal. USAID DEVELOPMENT DATA USAID has adopted a new policy that requires all data collected through its projects to be available to the public. This will be a critical component of the work done to assure data gaps are addressed transparently and accountably. All activities related to data should be driven by the following principles: 71 (1) All data should be open; (2) Opportunities for remote-sensing and real-time data collection should be maximized; (3) Data collection methodology should be transparent, rigorous and cost￾effective; and (4)The highest quality of data available and rigorous analysis methodologies should drive planning and policy-recommendations. The Development Data Library (DDL) is the Agency’s repository of USAID- funded, machine readable data created or collected by the Agency and its implementing partners. Datasets and supporting documentation created or collected under USAID￾funded awards must be submitted for inclusion in the DDL (ADS 579.3.2.2). Datasets must be submitted by USAID Operating Units and implementing partners through the DDL Website at www.usaid.gov/data in accordance with the instructions provided at the site. Appendix 1: PEER Indicators for the Science Objective Results Framework 4 PEER Strategic Objective Objective from Science Results Framework Indicator Critical Evidence OBJECTIVE: Use of scientific research for improved development outcomes increased Number of Lab-funded research results that influence program or policy changes (made by the public sector, private sector, or development actors) will double to 30 per year Capacity IR2: Locally-focused R&D for development increased Number of PEER-funded researchers who receive follow-on funding Collaboration IR2: Locally-focused R&D for development increased $ value of PEER partner leverage Capacity/ Critical Evidence sIR2.1: Local research teams strengthened to produce high-quality research Number of PEER research products (including publications, patents, technical presentations) Collaboration sIR2.1: Local research teams strengthened to produce high-quality research Number of PEER research collaborations/awards Capacity sIR2.1: Local research teams strengthened to produce high-quality research Number of PEER-funded researchers Capacity sIR2.1: Local research teams strengthened to produce high-quality research Number of research assistants supported by PEER-funded research Capacity sIR2.3: Global R&D for development networks strengthened to facilitate knowledge sharing and technical collaboration Number of PEER PIs who join professional networks/associations Programmatic sIR2.3: Global R&D for development networks strengthened to facilitate knowledge sharing and technical collaboration Number of USAID or NAS lead science/technical Convenings/forums held 72 PEER Strategic Objective Objective from Science Results Framework Indicator Collaboration / Critical Evidence IR3: Science ecosystems strengthened Number of meetings/workshops between PEER PIs and government/Mission reps Critical Evidence sIR3.1: R&D policies and regulations in targeted areas increased Number of PEER awardees that receive follow-on funding through Evidence to Action (E2A) grants, or policy and communications supplemental funding Critical Evidence sIR3.1: R&D policies and regulations in targeted areas increased Number of Evidence to Policy (E2P) gatherings/trainings/workshops (and their outcomes) Programmatic sIR3.2: Scientific associations and institutions in targeted areas strengthened Number of NAS lead financial and administrative trainings Capacity sIR3.3: Human scientific capacity in targeted areas strengthened Number of new/improved classes/courses developed by PEER PIs Capacity sIR3.3: Human scientific capacity in targeted areas strengthened Number of students trained (in these courses - above) Appendix 2: Evaluation Questions EVALUATION QUESTIONS The Evaluation will answer key questions about the implementation of the PEER program (the ‘how’) and the extent to which the PEER program is working and producing real change (the ‘what’). IMPLEMENTION PERFORMANCE QUESTIONS (the ‘how’): The mid-term evaluation will assess the PEER program performance in regards to its operational efficiency and management structure. Are activities cost-efficient? Is the program implemented in the most efficient way compared to alternatives? The goal of this part of the performance evaluation will be to improve PEER operating procedures for future cycles of grant recipients Stakeholders: current and former PEER staff (USAID and NAS), USAID Mission staff, USG partner Agencies, PEER PIs and co-PIs Performance evaluation questions may include but are not limited to: 1. Is the program implemented in the most efficient way compared to alternatives? 2. What is the relationship and perception of partners (PEER applicants, PEER PIs, OUs, USG partner agencies, etc) in regards to PEER management and implementation? 3. To what extent are PEER Results Framework indicators appropriate for capturing progress towards achieving the programs’ objectives? 4. To what degree have the PEER projects met the needs of the participating OUs? 5. How successful have outreach efforts by the PEER team to the LMIC scientific community been? 6. To what extent do USG Partner Agencies feel a benefit by participating in the PEER program? 7. How could USG Partner Agencies further their participation in the PEER program? 8. How well does PEER/Washington (USAID & NAS) communicate with the field? What are pain￾points in terms of communication? 9. Are the USAID and NAS PEER management structures clear and effective? 10. What are the strengths and weaknesses of the current process for publicizing the PEER grant solicitation? (Gender parity is one of the indicators to look at). 11. Does the review process yield funding of the most rigorous, highest impact research projects? Is the technical merit review conducted in a timely and professional manner? 12. To what extent have PEER PIs and potential PIs benefited from trainings in proposal writing? What is the quality of proposals to PEER from the participants and how does it compare to PEER applicants who have not participated in this type of training? What countries and/or PEER sectors would most benefit from this type of training and which organizations are best placed to provide the training? 13. Is the annual frequency of PEER solicitations sufficient? 14. What can be done to: a) increase the number of women who apply to PEER and b) increase the likelihood they will be selected as grantees? 15. How well is the buy-in mechanism managed? 16. Are the activities that are implemented beyond the PEER grants (such as Evidence to Action follow-on grants) contributing to the effectiveness of the overall program? 74 OUTCOME-RELATED EVALUATION QUESTIONS (the ‘what’): The evaluation should answer the following questions about the extent to which the PEER program is working and producing real change (the ‘what’). In answering these questions, the evaluation should address to what extent the PEER program’s Theory of Change is appropriate for achieving its highest order results and strategic objectives. Questions are organized by the program’s 3 strategic objectives: critical evidence, collaboration, and capacity-building. 1. To what extent has PEER helped generate critical evidence in the sectors it targets5 ? To what extent has PEER-generated critical evidence influenced public policies or development programs? Actors to interview: USAID Missions, Host country government ministries, local NGOs, PEER PIs and co-PIs. Sub-questions include, but are not limited to: a. Scientific evidence: i. What new scientific evidence or discoveries have resulted from PEER projects? ii. Are there any technical innovations that have resulted or have the potential to result from PEER research projects? iii. Have there been any unintended consequences (good or bad) related to new evidence derived from a PEER study? b. Influence on public policies and development programs: i. To what extent have host governments at the national, regional, and local level adopted solutions supported by PEER? ii. To what extent have non-governmental actors, such as NGOs or private companies, adopted solutions supported by PEER? iii. What were the characteristics of PEER research projects that were successful in influencing policy or program change? iv. What patterns and common characteristics emerged among PEER projects that had the least success in influencing policy/program change? v. What were the key factors driving decision-makers to adopt solutions generated through PEER research? vi. Describe if and how PEER research projects been incorporated into USAID mission programming? vii. What do USAID Missions, host country governments, and other local stakeholder see as the best way to promote adoption of results from PEER studies? viii. In what ways were the PEER researcher’s local connections and local knowledge important (or not important)? 75 5 Target sectors are: biodiversity, health, agriculture, environment, water, disaster mitigation, climate, education, food security, energy. c. PEER knowledge sharing activities: i. How have PI-led workshops encouraged sharing scientific findings with stakeholders? ii. What would researchers have done to disseminate their work in the absence of the PEER workshops or funding? iii. To what extent have PI-led workshops contributed to adoption by local, regional or national stakeholders? iv. How would participants improve dissemination efforts? 2. To what extent has PEER promoted collaborations between the US scientific community, local research institutions, host country government, and USAID Missions and Embassy staff, and private sector partners? Key actors to interview: USG partner, PEER PIs and co-PI, USAID Missions, Host country government Ministries, Local NGOs, private sector partners. Sub-questions include, but are not be limited to the following: a. Partnerships between USG-supported researchers and developing country researcher: i. To what degree has the USG partner’s expertise and resources contributed (or not contributed) to the success of the PEER award? ii. To what extent has the PEER collaboration led to further collaborations with the USG partner or other partners beyond the scope of the PEER award? iii. Particularly for multi-institutional awards, what is the context and nature of PEER PI collaborations with co-PIs and/or other institutions? iv. What role has PEER played in improving (or not improving) USG inter-agency engagement around research related to global development challenges? b. Local collaboration and connections: i. To what degree has the local nature of PEER research been important? ii. To what extent has PEER helped to facilitate regional, south-south, collaboration between researchers or within local science academies? iii. What additional opportunities exist for PEER to leverage local connections and knowledge? iv. To what extent have PEER projects addressed gender related gap or issues? c. Collaboration with host country government and USAID Missions: i. To what extent has PEER promoted exchange between PIs and the host country government or USAID Mission/ US Embassy? 76 ii. Have exchanges between the PEER PI and the host country government, Mission/Embassy resulted in changes in direction of the PEER project? In what ways? 3. To what extent has PEER built research capacity in developing countries? Key actors to interview: PEER PIs and co-PIs, USG Partners, students supported by PEER, University officials. Sub-questions include, but are not limited to the following: a. To what extent, and in what ways has the PEER program helped promote the growth of research careers of investigators - young, mid-career, and advanced - in developing countries? (for example have PEER researchers seen promotion within their institutions, country, or professional network?) b. How can the PEER program better support women and gender minorities? c. To what extent, and in what ways, have activities supported under PEER contributed to the development of new curricula or training programs? For example, has the award allowed for purchase of new pieces of equipment, development of new courses for students, or expansion of the curriculum of existing courses? Enabled the training of students in the PEER PI’s institution? d. Have there been ancillary or unintended benefits of the PEER award at LMIC institutions? Of what type? e. What do local stakeholders see as the best way for donors to support research capacity at LMIC institutions? f. To what extent has PEER helped moved governments in prioritizing research – both in supporting local LMIC researchers and/or using research for making policy decisions? Offerors are encouraged to propose additional questions as needed for assessing the program and its performance. 77 Appendix 3: Criteria to Ensure the Quality of the Evaluation Report From https://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf 78 ANNEX II: EVALUATION METHODS AND LIMITATIONS Desk Review During the initial kick-off meeting with the USAID PEER Program staff, the evaluation team discussed the types of documents that would be important for our review of the PEER program. These included: 1. Reports on program progress and activities; 2. Program monitoring data and process included in a Performance Management Plan (PMP); 3. Financial reporting; 4. Prior assessments or evaluations conducted; 5. Examples of research produced under PEER grants; and 6. Documentation of other similar grant programs for comparison; These documents were shared with the team during the first several weeks of implementation of the evaluation and were augmented with materials as requested after additional consultations with USAID and NAS staff. In addition, the evaluation team was granted access to the Foundant grant administration and monitoring system to conduct custom queries. We received or collected roughly 350 documents for review. The project publications cover Cycles 1 through 4. We sampled these documents in tandem with our strategy to select additional countries for other types of data collection in order to get a comprehensive look at approximately 22% of the overall program portfolio spanning Cycles 1 to 4. We selected 11 (22%) out of 50 countries in which PEER Programs have taken place for a deeper look at individual grant level documents. We included the five countries from each of the geographical regions with the largest number of projects: Kenya (Africa), Indonesia (Asia), Brazil (Latin America), Lebanon (MENA), and Armenia (Central Asia). Georgia and Armenia share one regional grant spanning both countries that will be included. Five other countries were randomly selected from a list of all countries with PEER activities using the random sort function in Excel, and include: Peru, Tanzania, Ghana, Haiti, and Bangladesh. Afghanistan was added to ensure an adequate number of country portfolios were reviewed that have significant Mission buy-in. The 11 selected countries consist of 104 projects, which is just over 50% of the 204 PEER projects. For these 104 projects, we will analyze the proposals submitted by PIs, the first quarterly report, and the most recent annual report. The baseline figures from the M&E database will also be reviewed. Country Number of PEER Projects Buy-in partner (Y/N) Afghanistan 2 Y Armenia 3 N Bangladesh 7 N Brazil 13 Y Ghana 4 N Haiti 1 N Indonesia 38 Y Kenya 19 Y Lebanon 8 N Peru 5 Y Tanzania 0F25 4 N Total 104 25 There are 4 projects just in Tanzania and 4 projects that are in both Kenya and Tanzania. The bi-national projects are counted in Kenya. 79 In addition to PEER program documents, we also selected documents related to three other similar programs by which we are assessing PEER’s operational efficiency and management structure by comparison. We examined one additional USAID program funded through the Bureau for Food Security (BFS), the Feed the Future Innovation Labs, along with some background on its predecessor the Collaborative Research Support Program Model (CRSP). We also looked at non-US project models funded in the United Kingdom under the UK Collaborative on Development Sciences (UK CDS), which in a fashion similar to PEER, funds research in developing countries through a collaborative network of 14 British government agencies and research institutions. Finally, we examined the Dutch funded Netherlands Organization for Scientific Research (NWO), which is another collaborative research funding mechanism, operating through the engagement of a decentralized network of research institutes through its Science for Global Development programs, focused on North/South partnerships in research. We attempted to include one additional program, but did not collect sufficient data for comparison. We employed a comparison analysis between PEER implementation data and that of the other programs, highlighting effectiveness and appropriateness of the different models. Key Informant Interviews The evaluation team developed interview guides approved by USAID for the following sets of key informants: 1. USAID officials (both in Washington and at Mission) 2. NAS staff 3. US-based research partners 4. USG and private partner institution officials 5. Host country government officials 6. Comparison program agency officials 7. PIs and Co-PIs 8. University officials supporting PEER PI 9. Staff from collaborating organizations in the field, including NGOs, private companies, other research institutions among others. Participants for Washington, DC-based KIIs were chosen purposively from the stakeholders listed below. The number of KIIs column indicates estimates gathered during the desk review. We limited KIIs with US-based Research Partners to a sample of one individual from each of the 11 countries identified in the desk review for a total of 11. Planned KIIs with comparison agencies included one KII for each of the four comparison programs identified. The Team Lead and one Data Analyst conducted the majority of DC-based interviews, however the Technical Advisors did join or replace the Team Lead during some interviews. These were conducted before, during and after the field work, with additional interviews with Missions completed after submission of the first draft evaluation report. DC-Based Stakeholder Groups Planned KIIs USG and Private Partner Institution Officials: NIH, NSF, US Forest Service, USGS, NASA, NOAA, Smithsonian Institution, NIFA, USDA, ARS, General Electric, National Instruments 10 USAID/Washington Officials 8 80 NAS Staff 4 US-based Research Partners 11 Comparison program agency officials 4 During site visits to Kenya and Indonesia we selected institutions and universities based on location and began with initial contacts with PIs and Co-PIs across all institutions/universities in that location, using stratified random sampling by sector. However, in Kenya we ended up interviewing a larger proportion of PIs and Co-PIs (20 of 24). We identified KII participants in the institutional/university administrative staff and collaborating organizations stakeholder groups through snowball sampling by asking PIs for contact information of others that influence their research in any way. Lastly, USAID Mission staff was identified through the contact list provided by USAID/Washington PEER staff, and supplemental names were provided when contacts did not respond or could not participate. Kenya/ Indonesia-Based Stakeholder Groups Number of KIIs per Country Total Number of KIIs USAID Mission Officials 3 in Indonesia, 1 in Kenya 4 Host Country Government Officials 2 4 PI Institutional/University Staff 3 6 PIs and Co-PIs 6 12 Staff from collaborating organizations in the field, including NGOs, private companies, other research institutions among others. 1 2 Given time and funding constraints, we only planned to conduct KIIs with Mission staff from the 11 countries identified during the document review phase. Ultimately we were able to interview nine of the planned 11. Besides Kenya and Indonesia, KIIs were conducted via phone or Skype with Mission staff from the originally selected countries of Brazil, Armenia, Peru, Haiti, and Bangladesh and based on opportunity, the team conducted interviews with two regional Mission staff from RDMA (on TDY in Jakarta) and El Salvador. We also planned to conduct phone interviews with nine (9) additional PIs outside of Kenya and Indonesia, one for each additional country selected. They were selected considering factors including the potential for their project to lead to policy or programmatic change, promotion of collaboration with multiple stakeholders, and the project’s capacity building achievements. Stakeholder Groups in Sample of 9 Countries Number of KIIs USAID Mission Representative 9 PIs 9 Key informants offer a unique perspective on all or some particular part of the program and often hold knowledge that may not be available elsewhere. Our guides were designed to structure the conversation, but allow some flexibility to 81 probe particular areas of knowledge in more detail as discovery warrants during the interview process. Notes and important quotes from these interviews were uploaded into an analytical software package called Dedoose, allowing the evaluation team to analyze and group responses according to coding based on evaluation questions and subquestions. Significant recurring responses or trends that emerged help bolster findings and support our conclusions. Online Survey We designed a survey instrument and deployed it over email using the web-based Survey Monkey tool. The survey captured data across each of the 4 evaluation questions. It was sent to all PEER funded PIs and Co-PIs for whom we had email addresses, as well as their US-based partners, totaling 426, out of which 229 were PIs and Co-PIs and 197 were US partner scientists. We reached out to the entire stakeholder population rather than employing a sampling strategy. The response rate for PIs and Co-PIs was 83% (190) and 63% for US partner scientists (125). This provided sufficient data for descriptive statistics, summary statistics, and comparison within/across variables related to gender, geographic location, theme, cycle, phase, and other similar dimensions. The survey instrument was reviewed by USAID and pre-tested by five selected foreign scientists, USAID staff, and external survey experts. We tested for: (1) question clarity and order; (2) completion time; (3) platform technical issues; and (4) distribution and bandwidth issues. Their feedback helped shape the final survey template edited in Survey Monkey for distribution. We allowed five weeks to gather responses, with weekly check-in and reminder points built in to increase response rates. Focus Group Discussions In Kenya and Indonesia, we convened two focus groups in each country: one with students engaged through PEER and one with female PIs funded by PEER.. The purpose of the student focus groups was to probe students on: (a) any direct participation they may have had on PEER funded research; and/or (b) any indirect benefits they may have received in terms of new coursework, access to new equipment or technology or training and new skills acquired. These focus groups only conducted in the field work and were designed to provide data to answer evaluation question four, related to capacity building through PEER. The purpose of female PI focus groups were to probe female researchers on their PEER experiences and any barriers they perceive based on sex in the field of scientific research. The data collection instrument was designed to guide discussion around their PEER products and capacity built through participation in PEER. These group interviews provided data to answer evaluation question two, related to filling evidence gaps and identifying ways PEER has influenced program and policy changes as well as question four relate to capacity building. Country Focus Group Discussions Participants Indonesia FGD #1 8 Students/Research Assistants Indonesia FGD #2 6 Female PIs Kenya FGD #3 21 Students/ Research Assistants Kenya FGD #4 8 Female PIs Site Visits In both Kenya and Indonesia, we planned to conduct a minimum of one site visit by identifying one project from each country listed on the Projects Master Program and Policy Shortlist spreadsheet compiled by USAID PEER staff. In reality we 82 were shown several labs at sites either before or after PIs interviews in both countries. In Kenya we conducted four site visits and in Indonesia two. The purpose of each visit was to gather observations about what is actually occurring at sites and to document utilization of research evidence or equipment as well as any research collaboration on site. Site visits are described in the table below: Country Site Visit Location Description Indonesia I3L Advanced molecular lab working to identify local species of bacteria and to keep a genetics bank. Indonesia Regional General Hospital of Koja Toured maternity ward as part of Kangaroo care study. Kenya Kenya Wildlife Service We visited the onsite DNA lab, consisting of four rooms. There they collected biological samples from the field and conducted DNA analysis. There were several pieces of new equipment in use. Kenya Kenya National Museum We toured DNA sample storage and processing facilities as well a warehouse of bird and insect samples collected over a century. Kenya KEFRI We visited three labs on the campus, all of which dealt with processing soil samples. Kenya University of Nairobi Dept. of Animal Husbandry Toured labs for processing plant and animal samples, including new storage equipment. Most Significant Change In both Kenya and Indonesia, the teams implemented an adaptation of the Most Significant Change methodology (MSC) developed by Rick Davies and Jess Dart1F26 to document the changes that have taken place at PEER sponsored local research organizations. the evaluation team convened PEER stakeholders, including researchers, students, university/research institute administrators and government partners for a half day workshop in which they: ● Identified key domains of change (e.g., teaching methods, research methods, application of research, policy change or changes in people-level outcomes); ● Wrote individual stories that best exemplify changes in the domains identified; ● Met in groups, similar to a focus group discussion, to share stories and to select the stories that are most representative of changes brought about by PEER funding and relationships; and ● Presented their selected stories in plenary group by group. The entire group then selected the stories that best represented the most significant change in different domains of change. The evaluation team worked with a local consultant in each country, who participated in the data collection and provided logistical support. The evaluators facilitated the process and took notes on the discussions about the stories and the selection decisions made by the group. They noted why certain stories are not selected in lieu of others. The stories served as the basis for developing case studies about institutional and individual change stimulated by the Program. Stories that demonstrated changes in policy or practice as a result of PEER funded research were used as case 26 Davies, Rick and Jess Dart 2005 The Most Significant Change Technique: A Guide to its Use. Australia. and UK: RJ Davies and J Dart self-published. 83 study examples in the evaluation. The most significant changes identified through the MSC process were compared to other findings from the evaluation elicited through different methods, such as the survey, interviews, analysis of the monitoring data, and from the document review. The process was guided by an instructive presentation. One version of the presentation was developed for the facilitators and used to instruct the local data collector in country prior to the workshop and to allow the entire team to tailor the specific activity according the number of participants. A second version was used during the workshop to inform and instruct participants on the process. In addition to the limitations detailed in the main body of the report, the section below provides additional details on of some of the limitations and effects/types of bias these might be having on the findings. Additional Limitations Response Bias Respondents in both the survey and the interviews may be providing answers they believe are what USAID wishes to hear to ensure their funder thinks the project is performing well (social desirability bias). It was not un common for the interviewees to be nervous about the interview and on several occasions the team had to re-state the purpose of the evaluation (not as an audit of their performance but as an assessment of the program). This could be causing inflation in the responses with a higher probability of skewness towards positive responses, more so for the PEER PI sample who depend directly from USAID funding. For the US partner survey, length and incentive to participate likely affected response rates. This reduces the power of the sample and its ability to be representative of the population of US partner scientistss in PEER. This group of respondents representing 63% of all US partners may be biased towards those having extreme experience (either very positive or very negative as these tend to have more incentive to share/participate in the survey). Concerning the PEER PI sample, 83% responded with largely PIs and thus this is likely very representative of the PEER scientist views. Another type of response bias is due to instrumentation error. For the US Partner, some questions had errors in their categories offered as answers as these had been drawn from the prime survey to the PEER PIs/Co-PIs. This error was caught post survey data collection. Longer pre-testing phase would have helped to reduce the size and number of these errors. Instrument calibration, specifically the survey, benefits the most and has the chance to reduce most sources of bias, when thoroughly vetted prior to implementation. Finally, the presence of USAID at interviews and/or for data collection processes is likely to (depending on the sample, cultural normal, type of project) skew data towards more favorable and positive responses. Particularly for scientists in developing countries, and for those who clearly understand the source of their funding, they are not likely to provide very negative perspectives as to the true nature of the project status or experience. Around negative issues or barriers to project implementation or opinions about the management entities themselves, it is less likely to be shared in the presence of USAID. The size and magnitude of this bias for this evaluation is likely negligible, but it is not clearly known. Most PIs were not offered the option to not include USAID in their interviews nor would they likely decline their presence once at the interview appointment. Generalization and Representativeness Of 49 countries, 88% of these were captured in the survey. Interview data was oversampled for the two countries with the largest portfolios (Kenya and Indonesia) as these two locations were selected as site visits. The interviews covered 84 11 countries comprising 22% PEER countries and 50% (104 projects) of the total portfolio (204 projects). Table X provide a list of countries captured in this evaluation. External validity is likely high for the survey data. Regarding the interview sampling structure and process, there is likely some selection bias present affecting external validity. Concerning information about the PEER PI – the evaluation is likely very representative, with high confidence (80% CI) for results concerning their views and experiences. The data is likely generalizable to the two largest countries, Kenya and Indonesia as there was a larger portion of data collection from these two locations. For countries with 1 project, data might provide information about the project but not scalable to the nation. Extrapolation towards a whole country’s PEER experience from 1 project should be avoided as this is not likely representative of the scientific cohort for that nation. Concerning US Partner Scientists – they data is somewhat representative and although the evaluation does not have a complete picture of their experience (63% response rate), it has provided some key areas of focus for future changes in the PEER program. Likely the length of the survey, variation in request for participation (each Agency varies in effort), and the lack of incentives contributed to the lower response rates. The findings from this evaluation are likely more representative of those projects from NSF and NIH, and not at all representative of projects from agencies like Smithsonian Institute with only a 12.5% response rate shown in the table below. The results also do not include information from the newer partners, such as the private sector. Key interviews were conducted with new private sector partners to inform program management, but these projects (Cycle 5 forward) were not part of the scope of the evaluation. Table 1-21: Number of projects per USG Agency and by survey response. *note: this includes PIs and Co-PIs, as well as a few other surveys completed by team members; multiple projects may be causing the high percent value (blue) although 83% of respondents were PIs therefore data is still heavily skewed towards the PI responses). U.S. partner's funding agency Number of Projects Survey Responses % represented in the Survey NSF 175 74 42.3 NASA 14 10 71.4 NIH 9 12 133.3* SI 8 1 12.5 USDA/ARS 4 1 25.0 USGS 4 2 50.0 USDA/FS 4 2 50.0 USDA/NIFA 2 3 150.0* Other 10 * Concerning the Views of USG Agencies, USAID, and NAS – interview data was captured for these groups but likely not representative of all mission experiences. Findings from this analysis are likely to be representative of USAID PEER DC, USG Agencies and NAS. Participants from these three groups, overall, have been part of the PEER program since the 85 inception and can provide complete information across time. For the Missions, data about their experience is somewhat limited due to the sample and time of data collection. During August September and October, most mission staff are either 1) out of the office on vacation or in transition to new posts, and/ or 2) working on fiscal year reporting. Accessing their time and accessing the right persons can be very difficult and the data collected reflects these constraints. It was hard to get staff from Missions, even in-country (i.e. Kenya) to participate in an interview. Some of the staff listed as a potential interview was new and/or relocated; this caused some delay in setting-up appointments for interviews. Unit of Analysis – although there is merit in evaluating at the individual level (perception, experience, other) for both the US partner and PEER PI, there is also likely key information from a “project level” analysis not present in this assessment. To understand collaboration and partnership outcomes, a complex scenario/activity, a different approach would have been required with a project-level perspective. For this to have been done, the design of the evaluation needed to be different; data collection would have been done by projects and for complete partnerships (PEER PI and US Partner). Due to budget, time and other logistical constraints, this approach was likely not viable to this evaluation. However, the results do give key information about how individuals in PEER, especially Scientists, are experiencing the program. Survey Language –With so many languages and different levels of English comprehension among PIs and Co-PIs funded by PEER, the survey data is likely to contain some level of response error, although we have not isolated examples. The potential size of this error is higher for questions were terms might have not been easily distinguishable such as “technology” versus “tool”. For this survey close to 25 people were invited to test the survey, representing all regions of PEER activity comprised of US and foreign scientists, USAID staff, and external survey experts. Of this group 5 provided feedback. The limited feedback was taken into account, and extensive knowledge and experience of the evaluation team in survey design and implementation (specifically for foreign and US academics reducing error significantly) was also a mitigating factor. 86 ANNEX III: DATA COLLECTION INSTRUMENTS Interview Guide A: USAID Washington and Mission Staff (Some questions color coded to designate as relevant solely for PEER Management Staff or Operating Units buying-in to PEER) INTRODUCTION: ● I/we, [insert Name] are evaluating the USAID funded Partnerships for Enhanced Engagement in Research (PEER) Program. a. I am on the PEER External Evaluation Team conducting a Midterm Performance Evaluation. This evaluation is being conducted by a research firm called Cultural Practice. b. Our evaluation is conducting interviews with key stakeholders, such as yourself. As a PEER [insert TYPE of stakeholder USAID By-in partner, USAID/DC Unit participating partner, other] ● Information you provide will be extremely helpful in providing PEER feedback on your experience participating in PEER and suggestions on how to improve the program. ● This interview will take approximately 60 to 90 minutes. Participant ID: _________ Interviewer: EB SM DC CN CF Name: Position: Location: Date: Time of Interview: Contact Information: ● Information you provide will be extremely helpful in providing PEER feedback on your experience participating in PEER and suggestions on how to improve the program. ● This interview will take approximately 60 to 90 minutes. INTERVIEW QUESTIONS Background Questions [This set of questions are intended to be more brief and less probing. Sets context for answering evaluation questions. 5 to 10 minutes.] What is your current position and role within USAID? 1. How long have you been with USAID and in your current position? 2. What relationship do you have/have you had with the PEER program? [Probe: how are they involved with PEER, are they the activity manager for the PEER by-in? Do they work with a Mission/OU that has PEER by-in?] 3. OU Only: How did you learn about PEER and what were your expectations? [Probe: who told them about PEER, what made them participate/by-in, what were their incentives and needs?] 4. Ou Only: How would you describe the PEER project and its objectives? 5. Are you familiar with the new Results Framework and PMP created for PEER? 87 6. If yes, how well do you feel that it captures the objectives and results of the project? What changes would you suggest, if any? [Probe: past vs. present system; complementarity to your RF (outside of Lab)] a. PEER Management: What input, if any, did you have in the RF/PMP process? 7. Evaluation Question 1: Efficiency of Program Implementation 8. Have you been involved with any other projects/programs similar to PEER, if yes, which ones? 9. What are the biggest implementation challenges of projects of this nature? What implementation challenges have you faced specifically with PEER? 10. What are the operational or management strengths you would identify? How do you feel operational or management efficiencies could be improved? 11. What strengths/issues would you identify with the current grant funding cycle and process? What would improve the funding cycle/process? a. Gender parity b. Research quality c. Buy-in mechanism d. Timeline e. Process/matching f. Goals/targets for PEER 12. What do you think other similar projects could learn from PEER? What could PEER learn from other similar projects? 13. Are you familiar with any activities that began under PEER and continued beyond the life of the grants? If yes, what activities? How do you feel these contribute to program effectiveness? 14. Evaluation Question 2: PEER Program Results 15. OU Only: In what ways has the PEER Program contributed to your unit’s overall goals and objectives? 16. OU Only: What information/products/outputs from PEER have you used? Where have you used these scientific products? How have these findings help your unit? 17. OU Only: What information/products/outputs have you shared and with who (NGO, other unit, other)? 18. Please share any particular strengths and challenges with regards to communication or information sharing of PEER products/activities [Probe: both with Washington and field-based stakeholders.] (Also relates back to Evaluation Q1) 19. Are you familiar with any outcomes associated with PEER knowledge sharing activities? [Probe: examples such as PI-led workshops or presentations.] If yes, which appear to be most effective in increasing awareness about their products? 20. What are some of best examples of product sharing? 21. What are some examples of ways in which PIs share products/outcomes that could be improved? 22. What examples can you share about PEER project(s) that have affected programmatic or policy changes? If none, why do you think not? Do you anticipate any changes in the future? 23. What are some unintended consequences/outcomes of PEER (positive/negative)? 88 24. Concerning the quality of the PEER products/research outputs, how have these met/not met your needs? a. Timing in provision of information b. Format of information c. Confidence in the information to inform decision-making d. Evaluation Question e. Collaboration 25. Please share examples of how PEER has promoted collaboration between scientists in LMIC, including at the level of government, private sector, and/or education and research institutions. [Probe: regional, south-south collaborations and exchanges between PIs, USAID and host country governments 26. Are you aware of any of these collaborations that have been expanded upon or that have lasted beyond the life of the PEER grant? If yes, please describe. 27. How has PEER affected cooperation between your operating unit and others broadly within the US? [Probe: specifically around the use of research to solve development challenges. 28. In your opinion/experience, what factors help change research capacity in a developing country? [Probe: how do they define change in research capacity?] Evaluation Question 4: Research Capacity 29. Are you aware of any examples of how PEER has improved curricula or created new training programs? If yes, please provide examples? 30. How can PEER better support women and minority researchers? 31. What recommendation would you have to increase/improve the outcomes and impacts of PEER for the next five years? Time permitting/Optional questions 32. OU Only: What more would you like to know about PEER? [Intended to help identify other relevant questions for this evaluation, this group might be able to help us. 33. How might a PI/PEER project better share their findings? What format/process would be most helpful/useful to you. 34. For you, what was the best part of PEER? Most challenging? 35. Do you have any other comments you wish to share with us? OTHER NOTES 89 Interview Guide B: NAS STAFF INTRODUCTION: ● I/we, [insert Name] are evaluating the USAID funded Partnerships for Enhanced Engagement in Research (PEER) Program. c. I am on the PEER External Evaluation Team conducting a Midterm Performance Evaluation. This evaluation is being conducted by a research firm called Cultural Practice. d. Our evaluation is conducting interviews with key stakeholders, such as yourself. As a PEER [insert TYPE of stakeholder USAID By-in partner, USAID/DC Unit participating partner, other] ● Information you provide will be extremely helpful in providing PEER feedback on your experience participating in PEER and suggestions on how to improve the program. ● This interview will take approximately 60 to 90 minutes. Participant ID: _________ Interviewer: EB SM DC CN CF Name: Position: Location: Date: Time of Interview: Contact Information: ● Information you provide will be extremely helpful in providing PEER feedback on your experience participating in PEER and suggestions on how to improve the program. ● This interview will take approximately 60 to 90 minutes. OTHER NOTES 90 Interview Guide C: USG Partner Scientists INTRODUCTION: ● I/we, [insert Name] are evaluating the USAID funded Partnerships for Enhanced Engagement in Research (PEER) Program. 1. I am on the PEER External Evaluation Team conducting a Midterm Performance Evaluation. This evaluation is being conducted by a research firm called Cultural Practice. 2. Our evaluation is conducting interviews with key stakeholders, such as yourself. As a PEER [insert TYPE of stakeholder USAID By-in partner, USAID/DC Unit participating partner, other] ● Information you provide will be extremely helpful in providing PEER feedback on your experience participating in PEER and suggestions on how to improve the program. ● This interview will take approximately 60 minutes. Participant ID: _________ Interviewer: EB SM DC CN CF Name: Position: Location: Date: Time of Interview: Contact Information: ● Information you provide will be extremely helpful in providing PEER feedback on your experience participating in PEER and suggestions on how to improve the program. ● This interview will take approximately 60 to 90 minutes. Background Questions [This set of questions are intended to be more brief and less probing. Sets context for answering evaluation questions. 5 to 10 minutes.] 1. What is your current position and role role within your institution? [Probe: rank (tenured, assistant, associate, research, other) 2. How long have you been at your institution and in your current position? 3. How did you learn about PEER and what were your expectations? What did you think PEER was about or what it would accomplish? [Probe: who told them about PEER, what made them participate, what were their needs?] 4. What were your incentives for participating in PEER? 5. What kind of experience do you have working abroad or collaborating on international projects? a. Locations b. Types of collaborations/collaborators (which types of groups/entities) 6. Prior to your participation in PEER, in what ways, if any, have you worked with your partner PEER PI? How did your collaboration begin? Evaluation Question 1: Efficiency of Program Implementation 91 7. Do you have any experience working on programs similar to PEER? If yes, how does your experience with PEER compare? 8. What do you think that PEER could learn from these other similar programs? 9. Can you discuss any interactions you might have had through PEER with USAID or NAS? [record types of interactions, positive/negative]. Evaluation Question 2: PEER Program Results 10. What is in your opinion, the most valuable, useful, product/outcome of PEER, of your project? a. What types of products are coming out of PEER? b. What outcome should be happening that is not? 11. Do you have any examples of how your PEER project(s) has affected programmatic or policy changes either with the funding agencies or other stakeholders within the country or region? a. If Yes, how? b. If No, why do you think not? Do you anticipate any changes in the future? 12. How might PEER improve its program to better target policy change? 13. What are some unintended consequences/outcomes of PEER (positive/negative) [Also probe impacts specifically at developing country institutions – related to Eval. Q4- capacity]? 14. Concerning the quality of the PEER products/research outputs, how have these met/not met your needs/standards? 15. What information/products/outputs have you shared and with who (Probe: within your institution, outside organizations, with other colleagues)? 16. What are some examples of new scientific discoveries or technological innovations that can be attributed to PEER? How have they been used/applied? 17. Are you familiar with any outcomes associated with PEER knowledge sharing activities like PI-led workshops or presentations? If yes, please describe them. (Also relates to Q3) Evaluation Question 3: Collaboration (ask first – frame in terms of results of partnership) 18. What do you think are factors that contribute to a successful partnership with a collaborator from another country? How has PEER contributed to these factors? 19. How do you communicate with your host country PI? How could PEER help you so as to have better/more effective communication with your partner? [Probe methods; frequency; barriers] 20. Were you involved with any efforts to connect scientists in developing countries or regions through PEER? If yes, who was the target? How effective were your efforts? 21. Please share examples of how you have promoted collaboration between scientists, including at the level of government, private sector, and/or education and research institutions. (Probe: regional, south-south collaborations and exchanges between PIs, USAID and host country governments) 22. From this partnership/project, how does this collaboration help you in future collaborations? Expand collaborations with others in your institution? 23. How have you benefited from this collaboration? Evaluation Question 4: Capacity 24. Can you share any examples of how PEER has impacted the careers of PIs in developing countries? How has 92 PEER affected your capacity? 25. How can PEER better support women and gender minorities? 26. Are you aware of any examples of how PEER has improved curricula or created new training programs? If yes, please provide examples? 27. What recommendation would you have to increase/improve the outcomes and impacts of PEER over the next 5 years? Time Permitting/Optional Questions 28. What more would you like to know about PEER? [Intended to help identify other relevant questions for this evaluation, this group might be able to help us.] 29. How might a PI/PEER project better share their findings? What format/process would be most helpful/useful to you? 30. Do you have any other comments you wish to share with us? OTHER NOTES 93 Participant ID: _________ Interview Guide D: USG Agency PEER Partner POCs (NIH, NSF, USGS, NASA, NOAA, NIFA, USDA, Smithsonian, ONR) INTRODUCTION: ● I/we, [insert Name] are evaluating the USAID funded Partnerships for Enhanced Engagement in Research (PEER) Program. 1. I am on the PEER External Evaluation Team conducting a Midterm Performance Evaluation. This evaluation is being conducted by a research firm called Cultural Practice. 2. Our evaluation is conducting interviews with key stakeholders, such as yourself. As a PEER [insert TYPE of stakeholder USAID Buy-in partner, USAID/DC Unit participating partner, other] ● Information you provide will be extremely helpful in providing PEER feedback on your experience participating in PEER and suggestions on how to improve the program. ● This interview will take approximately 60 minutes. Participant ID: _________ Interviewer: EB SM DC CN CF Name: Position: Location: Date: Time of Interview: Contact Information: INTERVIEW QUESTIONS Background Questions [This set of questions are intended to be more brief and less probing. Sets context for answering evaluation questions. 5 to 10 minutes.] 1. What is your current position and role with your organization? 2. How long have you been in your current position? 3. What relationship do you have/have you had with the PEER project? 4. Do you know how many USG PEER Partners your agency supports? What cycles and countries? 5. How did you learn about PEER and what were your expectations? [Probe: who told them about PEER, what made them participate/buy-in, what were their incentives and needs?] 6. How would you describe the PEER project and its objectives? Evaluation Question 1: Efficiency of Program Implementation 7. Have you been involved with any other projects/programs similar to PEER, if yes, which ones? 8. How does PEER compare in goals and approach to other similar programs that you are familiar with? 9. What are the biggest implementation challenges of projects of this nature? What implementation challenges have you faced specifically with PEER? 94 10.What do you think other similar projects could learn from PEER? What could PEER learn from other similar projects? 11.PEER has gone through some structural changes in their calls, including how they solicit and evaluate proposals and convene peer reviewers. How has this affected your agency’s engagement with PEER? 12.Please describe any particular strengths and challenges with regards to your communication with other PEER stakeholders? Probe specific stakeholders, including: NAS, USAID, other USG agencies, researchers, PIs. 13.Are you familiar with any activities or relationships that began under PEER and continued beyond the life of the grants? If yes, what activities and/or relationships? How do you feel these contribute to program effectiveness? Evaluation Question 2: Influence Program/Policy Change 14. What is in your opinion, the most valuable, useful, product/outcome of PEER? a. What types of products are coming out of PEER? b. What outcome should be happening that is not? 15. What are some examples of new scientific discoveries or technological innovations that can be attributed to PEER? What were the drivers? How have they been used/applied? 16. Do you have any examples of how PEER has affected programmatic or policy changes with US-based stakeholders or in other countries or regions? a. If Yes, how? b. If No, why do you think not? Do you anticipate any changes in the future? c. How might PEER improve its program to better target policy change? 17. What are some unintended consequences/outcomes of PEER (positive/negative) [Probe impacts specifically at developing country institutions – related to Eval. Q4- capacity]? 18.What information/products/outputs have you shared and with who (within/outside your institutions, with other colleagues, other)? 19.Are you familiar with any outcomes associated with PEER knowledge sharing activities like PI-led workshops or presentations? If yes, please describe them. (Also relates to Evaluation Q3) 20.Has PEER led to any off-shots within your agency? Has your participation in PEER led to your Agency making new funding allocations for things related to PEER or because of PEER? What were the drivers? Evaluation Question 3: Collaboration 21.How has PEER affected cooperation between your operating unit and others broadly within the USG? [Probe: specifically around the use of research to solve development challenges.] 22. How has the PEER project led to additional partnerships/collaborative efforts in other areas of your Agency not involved with PEER? Probe: relationships lasting beyond PEER grants? 23. How do these collaborations and partnerships affected by PEER contribute to the goals and objectives of your agency or specific units within your agency? 24. In what ways does PEER complement, or pair with other activities of your agency? How has PEER impacted/contributed to these other programs within your agency? What other contributions has PEER made to your agency? Evaluation Question 4: Capacity 25.In your opinion/experience, what factors help change research capacity in a developing country? [Probe: how do they define change in research capacity?] 95 26.Are you aware of any examples of how PEER has improved curricula or created new training programs? If yes, please provide examples? 27.How can PEER better support women and minority researchers? 28. What recommendation would you have to increase/improve the outcomes and impacts of PEER over the next 5 years? Time Permitting/Optional Questions 29. What more would you like to know about PEER? [Intended to help identify other relevant questions for this evaluation, this group might be able to help us.] 30.How might a PI/PEER project better share their findings? What format/process would be most helpful/useful to you? 31. For you, what was the best part of PEER? Most challenging? 32.Do you have any other comments you wish to share with us? ● OTHER NOTES 96 Interview Guide E: Host Country Government Officials INTRODUCTION: ● I/we, [insert Name] are evaluating the USAID funded Partnerships for Enhanced Engagement in Research (PEER) Program. ● I am on the PEER External Evaluation Team conducting a Midterm Performance Evaluation. This evaluation is being conducted by a research firm called Cultural Practice. ● Our evaluation is conducting interviews with key stakeholders, such as yourself. As a PEER [insert TYPE of stakeholder USAID By-in partner, USAID/DC Unit participating partner, other] ● Information you provide will be extremely helpful in providing PEER feedback on your experience participating in PEER and suggestions on how to improve the program. This interview will take approximately 60 to 90 minutes INFORMED CONSENT STATEMENT: Your participation in this evaluation is entirely voluntary and it is your choice whether to participate or not. There are no known risks or discomforts associated with participating in this interview Would you like to continue with your participation? Do you mind if we record and transcribe the interview? [Interviewer CHECK the BOX for consent] YES NO Interviewer: EB SM DC CN CF Name: Position: Location: Date: Time of Interview: INTERVIEW QUESTIONS Background Questions [This set of questions are intended to be more brief and less probing. Sets context for answering evaluation questions. 5 to 10 minutes.] 1. What is your current position and role with your organization? 2. How long have you been in your current position? 3. What relationship do you have/have you had with the PEER project? 4. How many PEER projects have you been involved with or were supported in some way by your organization? In what ways? 5. How did you learn about PEER and what were your expectations? [Probe: who told them about PEER, what were their incentives and needs?] 6. How would you describe the PEER project and its objectives? Evaluation Question 1: Efficiency of Program Implementation 7. Have you been involved with any other projects/programs similar to PEER, if yes, which ones? Could you share any contact information for staff involved in similar projects? 8. In your opinion, what are PEER’s greatest strengths? 9. What are the biggest challenges you have faced specifically with PEER? How were/can these be overcome to improve the program? 97 10.Are you familiar with the PEER grant funding cycle and process? If yes, What strengths/issues would you identify? What would improve the funding cycle/process? a. Gender parity b. Research quality c. Buy-in mechanism d. Timeline e. Process/matching f. Goals/targets for PEER 11. If Yes response to question 10: PEER has gone through some structural changes in their calls. How has this affected your projects? 12.What do you think other similar projects could learn from PEER? What could PEER learn from other similar projects? 13.Please describe any particular strengths and challenges with regards to your communication with other PEER stakeholders? 14.Are you familiar with any activities that began under PEER and continued beyond the life of the grants? If yes, what activities? How do you feel these contribute to program effectiveness? Evaluation Question 2: Influence Program/Policy Change 15. What is in your opinion, the most valuable, useful, product/outcome of PEER? a. What types of products are coming out of PEER? b. What outcome should be happening that is not? 16. What are some examples of new scientific discoveries or technological innovations that can be attributed to PEER? What were the drivers? How have they been used/applied? 17. Do you have any examples of how your PEER project(s) has affected programmatic or policy changes within the country or region? a. If Yes, how? b. If No, why do you think not? Do you anticipate any changes in the future? c. How might PEER improve its program to better target policy change? 18. What are some unintended consequences/outcomes of PEER (positive/negative) [Probe impacts specifically at developing country institutions – related to Eval. Q4- capacity]? 19. What information/products/outputs have you shared and with who (within/outside your institutions, with other colleagues, other)? 20. Are you familiar with any outcomes associated with PEER knowledge sharing activities like PI-led workshops or presentations? If yes, please describe them. (Also relates to Evaluation Q3) 21. Has PEER led to any off-shots within your agency? Has your participation in PEER led to your Agency making new funding allocations for things related to PEER or because of PEER? What were the drivers? Evaluation Question 3: Collaboration 22. How has PEER affected cooperation between your operating unit and others broadly within your government? [Probe: specifically around the use of research to solve development challenges.] 23. How has the PEER project led to additional partnerships/collaborative efforts in other areas of your Agency not involved with PEER? 24. Are you aware of any of these collaborations that have been expanded upon or that have lasted beyond the life of the PEER grant? If yes, please describe. Evaluation Question 4: Capacity 25. In your opinion/experience, what factors help change research capacity in the country? [Probe: how do they define change in research capacity?] 98 26. Are you aware of any examples of how PEER has improved curricula or created new training programs? If yes, please provide examples? 27. How can PEER better support women and minority researchers? Time Permitting/Optional Questions 28. What more would you like to know about PEER? [Intended to help identify other relevant questions for this evaluation, this group might be able to help us.] 29. How might a PI/PEER project better share their findings? What format/process would be most helpful/useful to you? 30. For you, what was the best part of PEER? Most challenging? 31. If PEER were to continue for another five-year cycle, what recommendation would you have to increase/improve the outcomes and impacts of PEER? Or increase your organization’s participation in PEER? 32. Do you have any other comments you wish to share with us? ● OTHER NOTES 99 Interview Guide F: Comparable Programs INTRODUCTION: ● I/we, [insert Name] are evaluating the USAID funded Partnerships for Enhanced Engagement in Research (PEER) Program. ● I am on the PEER External Evaluation Team conducting a Midterm Performance Evaluation. This evaluation is being conducted by a research firm called Cultural Practice. ● Our evaluation is conducting interviews with key stakeholders, such as yourself. As a PEER [insert TYPE of stakeholder USAID By-in partner, USAID/DC Unit participating partner, other] ● Information you provide will be extremely helpful in providing PEER feedback on your experience participating in PEER and suggestions on how to improve the program. This interview will take approximately 30-60 minutes INFORMED CONSENT STATEMENT: Your participation in this evaluation is entirely voluntary and it is your choice whether to participate or not. There are no known risks or discomforts associated with participating in this interview Would you like to continue with your participation? Do you mind if we record and transcribe the interview? [Interviewer CHECK the BOX for consent] YES NO Interviewer: EB SM DC CN CF Name: Position: Location: Date: Time of Interview: INTERVIEW QUESTIONS Background Questions [This set of questions are intended to be more brief and less probing. Sets context for answering evaluation questions. 5 minutes.] 1. What is your current position and role within your program? 2. How long have you been working on your program? Evaluation Question 1: Efficiency of Program Implementation We are primarily interested in learning from you about ________ program. 3. How would you describe the main objectives of your program? 4. Who are the main stakeholders in your program? 5. How long is each program cycle? 6. How many staff (full or part-time) support the management team? 7. In the course of a project cycle what kinds of tasks is the management team responsible for? ● Proposal/grant review? Coordinating with PIs? Monitoring and evaluation? Reporting? Trips to locations? Etc. 8. Could you describe in your own terms the funding mechanism and management of that process? a. Benefits of current system? b. Challenges of current system? i. Getting funds to PIs ii. Accountability 9. What do you think other similar projects could learn from your program? 10. What could your project learn from other projects? b. Are you involved in any other similar international research programs? i. How do they compare to the _______ program? 10 0 Evaluation Question 2: Influence Program/Policy Change? 11. In your opinion, what factors contribute to PI productivity in developing countries? Which one seems to make the most difference over the last 5 years? 12. What examples can you share about your program’s project(s) that have affected programmatic or policy changes? If none, why do you think not? Do you anticipate any changes in the future? Evaluation Question 3: Collaboration 13. In your experience, what is the most effective model for U.S. and developing country researchers/PIs to collaborate on a project? 14. What kinds of support does your program provide to U.S. and developing country PIs? a. Can you provide an example? 15. What kinds of support to you provide to project stakeholders to build their networks? What has been most effective? Evaluation Question 4: Capacity 16. What kinds of support does your program provide to different stakeholders (e.g., fund research, short-term training, degree training)? ● Developing country PIs? ● PIs’ institutions? ● Students? 17. In your experience, what are the greatest challenges PIs in developing countries face to advance in their career? What if anything has your program done to overcome those challenges? 18. Which of your stakeholders do you feel benefits the most from your program? Why? 19. Are you familiar with the PEER program? How did you hear about it? 20. Do you have any other comments you wish to share with us? ● OTHER NOTES 10 1 INTRODUCTION: ● I/we, [insert Name] are evaluating the USAID funded Partnerships for Enhanced Engagement in Research (PEER) Program. ● I am on the PEER External Evaluation Team conducting a Midterm Performance Evaluation. I/we work for a research firm called Cultural Practice/QED. ● Our evaluation is conducting interviews with key stakeholders, such as yourself. As a PEER [insert TYPE of stakeholder USAID By-in partner, USAID/DC Unit participating partner, other] ● Information you provide will be extremely helpful in providing PEER feedback on your experience participating in PEER and suggestions on how to improve the program. This interview will take approximately 60 to 90 minutes INFORMED CONSENT STATEMENT: Your participation in this evaluation is entirely voluntary and it is your choice whether to participate or not. There are no known risks or discomforts associated with participating in this interview Would you like to continue with your participation? Do you mind if we record and transcribe the interview? [Interviewer CHECK the BOX for consent] YES NO Interviewer: EB SM DC CN CF Name: Position: Location: Date: Time of Interview: INTERVIEW QUESTIONS Background Questions 1. What is your current position and role within your institution? a. Ask about their rank (tenured, assistant, associate, research, other) 2. How long have you been at your institution? (as a PEER PI/Co-PI?) 3. How did you learn about PEER and what were your expectations? [Probe: who told them about PEER, what mechanism (via email, online, other)?] 4. What were your expectations about what PEER could do for your career and/or research? [Probe: What did you think PEER was about or what it would accomplish?] 5. Can you share with us the incentives to apply to PEER/participate with PEER? Evaluation Question 1: Efficiency of Program Implementation 6. Have you been involved with any other projects/programs similar to PEER, if yes, which ones? 10 2 7. How difficult or easy did you find the PEER proposal process (writing the proposal, matching with US PI, other)? [Probe: gather positive and negative feedback; barriers during the process? After winning? Issues receiving funds?] 8. What are the operational or management strengths with PEER you would identify? How do you feel operational or management efficiencies could be improved? 9. What are the biggest implementation challenges of projects of this nature? What implementation challenges have you faced specifically with PEER? 10. In your experience with PEER thus far, can you discuss how each of the management entities, if at all, have interacted with you [Probe: what types of interactions, positive/negative; how do you benefit from their leadership/assistance?] a. NAS b. USAID c. USAID Mission d. Other Buy-in funder 11. Have there been any changes in the Program (PEER) that have affected your work/outcomes? 12. Concerning the sharing of your PEER products (papers, technology, results, etc.), what role have you played in this? How have you shared these and to whom (assumes they shared products)? 13. Please describe any particular strengths and challenges with regards to your communication with field-based stakeholders/USAID Missions/PIs? Evaluation Question 2: Influence Program/Policy Change? 14. What is in your opinion, the most valuable, useful, product/outcome of PEER, of your project? [Probe: types of products produced by PEER] 15. What is in your view the anticipated impact of your PEER project, of PEER in general? 16. Where appropriate, what might be an example of how your PEER project(s) has affected programmatic or policy changes? c. If Yes, how? d. If No, why do you think not? Do you anticipate any changes in the future? e. How might PEER improve its program to better target policy change? 17. What are some unintended consequences/outcomes of PEER (positive/negative)? What outcomes should be happening next that are not? Evaluation Question 3: Collaboration 18. How do you communicate with your US-based research partner? How could PEER help you so as to have better/more effective communication with your partner? [Probe methods; frequency; barriers] 19. What do you think are factors that contribute to a successful partnership with a collaborator from another country? 10 3 a. What might be some barriers to this collaboration? b. What could PEER do to improve this? 20. In your experience, what must be present, crucial, for a collaboration with a scientist from another country, from a developing nation? 21. Please share examples of how PEER has promoted collaboration between scientists and others, including at the level of government, private sector, and/or education and research institutions. [Probe: regional, south-south collaborations and exchanges between PIs, USAID and host country governments] 22. From this partnership/project, how does this collaboration help you in future collaborations? Expand collaborations with others in your institution? 23. How have you benefited from this collaboration? Evaluation Question 4: Capacity 24. What types of capacity building efforts have you helped with/participated/provided your host country PI/Co-PI/students? 25. What type of capacity building activities do you think a PEER project needs? 26. How can PEER better support women and gender minorities? Time Permitting/Optional Questions 27. Concerning the quality of the PEER products/research outputs, how have these met/not met your needs/standards? a. Timing in provision of information b. Format of information c. Confidence in the information to inform decision-making 28. What more would you like to know about PEER? [Intended to help identify other relevant questions for this evaluation, this group might be able to help us.] 29. How might a PI/PEER project better share their findings? What format/process would be most helpful/useful to you? 30. If PEER were to continue for another five-year cycle, what recommendation would you have to increase/improve the outcomes and impacts of PEER? 31. Do you have any other comments you wish to share with us? ● OTHER NOTES 10 4 Interview Guide H: University Officials supporting PEER Funded PI INTRODUCTION: ● I/we, [insert Name] are evaluating the USAID funded Partnerships for Enhanced Engagement in Research (PEER) Program. ● I am on the PEER External Evaluation Team conducting a Midterm Performance Evaluation. This evaluation is being conducted by a research firm called Cultural Practice. ● Our evaluation is conducting interviews with key stakeholders, such as yourself. As a PEER [insert TYPE of stakeholder USAID By-in partner, USAID/DC Unit participating partner, other] ● Information you provide will be extremely helpful in providing PEER feedback on your experience participating in PEER and suggestions on how to improve the program. This interview will take approximately 60 to 90 minutes. INFORMED CONSENT STATEMENT: Your participation in this evaluation is entirely voluntary and it is your choice whether to participate or not. There are no known risks or discomforts associated with participating in this interview Would you like to continue with your participation? Do you mind if we record and transcribe the interview? [Interviewer CHECK the BOX for consent] YES NO Interviewer: EB SM DC CN CF Name: Position: Location: Date: Time of Interview: INTERVIEW QUESTIONS Background Questions [This set of questions are intended to be more brief and less probing. Sets context for answering evaluation questions. 5 to 10 minutes.] 1. What is your current position and role with your university or institute? 2. How long have you been in your current position? 3. What relationship do you have/have you had with the PEER project? 4. How many PEER projects have you worked on? How many total supported by your university or institute? What level of financial support, if any, has the university or institute provided in support of PEER funded research? 5. How did you learn about PEER and what were your expectations? [Probe: who told them about PEER, what made them participate/by-in, what were their incentives and needs?] 6. How would you describe the PEER project and its objectives? 10 5 Evaluation Question 1: Efficiency of Program Implementation 7. Have you been involved with any other projects/programs similar to PEER, if yes, which ones? 8. What do you think other similar projects could learn from PEER? What could PEER learn from other similar projects? 9. What are the biggest implementation challenges of projects of this nature? What implementation challenges have you faced specifically with PEER? 10. What are the operational or management strengths you would identify? How do you feel operational or management efficiencies could be improved? 11. Are you familiar with the current PEER grant funding cycle and process? If yes, what strengths/issues would you identify? What would improve the funding cycle/process? g. Gender parity h. Research quality i. Buy-in mechanism j. Timeline k. Process/matching l. Goals/targets for PEER 12. PEER has gone through some structural changes in their calls. How has this affected your projects? 13. Please describe any particular strengths and challenges with regards to your communication with other PEER stakeholders? Probe: national/local government officials, researchers, USG, NGOs, private sector? 14. Are you familiar with any activities that began under PEER and continued beyond the life of the grants? If yes, what activities? How do you feel these contribute to program effectiveness? Evaluation Question 2: Influence Program/Policy Change 15. What is in your opinion, the most valuable, useful, product/outcome of PEER? f. What types of products are coming out of PEER? g. What outcome should be happening that is not? 16. What are some examples of new scientific discoveries or technological innovations that can be attributed to PEER? What were the drivers? How have they been used/applied? 17. Do you have any examples of how your PEER project(s) has affected programmatic or policy changes within the country or region? d. If Yes, how? e. If No, why do you think not? Do you anticipate any changes in the future? f. How might PEER improve its program to better target policy change? 18. What are some unintended consequences/outcomes of PEER (positive/negative) [Probe impacts specifically at developing country institutions – related to Eval. Q4- capacity]? 19. What information/products/outputs have you shared and with who (within/outside your institution, with other colleagues, other)? 10 6 20. Are you familiar with any outcomes associated with PEER knowledge sharing activities like PI-led workshops or presentations? If yes, please describe them. (Also relates to Evaluation Q3) 21. Has PEER led to any off-shots within your institute? Has your participation in PEER led to your institute making new funding allocations for things related to PEER or because of PEER? What were the drivers? Evaluation Question 3: Collaboration 22. How has PEER affected cooperation between your institute and others nationally or regionally? [Probe: specifically around the use of research to solve development challenges.] 23. How has the PEER project led to additional partnerships/collaborative efforts in other areas of your institute not involved with PEER? 24. Are you aware of any of these collaborations that have been expanded upon or that have lasted beyond the life of the PEER grant? If yes, please describe. Evaluation Question 4: Capacity 25. In your opinion/experience, what factors help change research capacity in your country? [Probe: how do they define change in research capacity?] 26. Are you aware of any examples of how PEER has improved curricula or created new training programs? If yes, please provide examples? 27. How can PEER better support women and minority researchers? Time Permitting/Optional Questions 28. What more would you like to know about PEER? [Intended to help identify other relevant questions for this evaluation, this group might be able to help us.] 29. How might a PI/PEER project better share their findings? What format/process would be most helpful/useful to you? 30. For you, what was the best part of PEER? Most challenging? 31. If PEER were to continue for another five-year cycle, what recommendation would you have to increase/improve the outcomes and impacts of PEER? Or increase your organization’s participation in PEER? 32. Do you have any other comments you wish to share with us? OTHER NOTES 10 7 GROUP INTERVIEWS Group Interview Guide A: PEER Students INTRODUCTION: ● I/we, [insert Name] are evaluating the USAID funded Partnerships for Enhanced Engagement in Research (PEER) Program. ● I am on the PEER External Evaluation Team conducting a Midterm Performance Evaluation. I/we work for a research firm called Cultural Practice/QED. ● Our evaluation is conducting FOCUS GROUPS with key stakeholders, such as yourself. ● Information you provide will be extremely helpful in providing PEER feedback on your experience participating in PEER and suggestions on how to improve the program. This focus group will take approximately 2 hours INFORMED CONSENT STATEMENT: Your participation in this evaluation is entirely voluntary and it is your choice whether to participate or not. There are no known risks or discomforts associated with participating in this interview Would you like to continue with your participation? Do you mind if we record and transcribe the interview? [Interviewer CHECK the BOX for consent] YES NO Interviewer: EB SM DC CN CF Name: Position: Location: Date: Time of Interview: IMPLEMENTATION ● Introduction Facilitator(s): (5 minutes) o Welcome and thank you for sharing your time today. o My name is [INSERT NAME] and I am one of the Senior Technical Leads for this evaluation. o My team members are [INSERT team names and members introduce themselves and their roles] o We are part of the Evaluation team for the PEER Program. o Some of you might be very familiar with this program and others might not, and that is ok. o Our goal to today is to have a conversation with you about your experience as students, and as participants in PEER o There are no right or wrong comments. o The information you provide will be collapsed and responses will be stripped of your name ▪ hence your advisor will not know what you might be saying about him or her ☺ ● Introduction of the Attendees: (10 minutes) o Share your name and your school/major o Ice breaker ● Review Agenda: (10 minutes) o Review process 10 8 ▪ First we will start with introductions – which we already completed ▪ Then I will ask the group a series of questions and we will have a discussion and makes some notes together ▪ For each question: ● We will record as a group the answers ● We will then write out and summarize as a group ▪ There will be one break, at the half-way mark of the Focus Group where we can have snacks and drinks ▪ I will also open the session for questions you might for each other ▪ Finally, we will close the focus group, some final comments/videos and or thoughts o Set expectations ▪ “OK, as a group, first we are going to take a few minutes to set the rules for our group” ● You will each give me some rules we might want to follow ● Logistics o Paper and pens o Sticky notes o Recorder/video/voice o Note taker o Facilitator QUESTIONS QUESTION 1. As a young scholar of your country, what inspires you to be in science and technology? Probe: - How did you select your course work? - What do you like about your course work/lab experience? QUESTION 2. Do you know what the PEER Program is? If so, how have you been affected by PEER? (direct/indirect) - New Courses/Classes, workshops, trainings? - Access to new method, equipment, information - Involved in research? How? QUESTION 3. What kinds of opportunities do you expect to have after graduation? In what ways has your work at the university opened up new opportunities? How does this relate to PEER research, if at all? QUESTION 4. You are the next generation of your country, the up-and coming scientists, to go on and discover the new cures for cancer, to engineer the new buildings on Mars…what type of support do you need now, would you need in the future to ensure you are successful in a career in science/technology/engineering? QUESTION 5. For you, what are the barriers within your country, within your institutions to do quality research, to be innovative and inventive, to make the new discoveries? QUESTION 6. Open Time VLOG BOOTH – volunteer “Tell us why you are a scientist and how you plan to change your country with your creativity/innovative power/skills?” VIDEOS/VLOGS for PEER from Students ● OTHER NOTES Some examples: 1. All comments are valid 2. Everyone should be respectful and open to other’s comments 3. The space we are in is a safe and respectful space 10 9 Group Interview Guide B: PEER Female PI/Co-PIs INTRODUCTION: ● I/we, [insert Name] are evaluating the USAID funded Partnerships for Enhanced Engagement in Research (PEER) Program. ● I am on the PEER External Evaluation Team conducting a Midterm Performance Evaluation. I/we work for a research firm called Cultural Practice/QED. ● Our evaluation is conducting FOCUS GROUPS/Gathering with key stakeholders, such as yourself. ● Information you provide will be extremely helpful in providing PEER feedback on your experience participating in PEER and suggestions on how to improve the program. This focus group will take approximately 2 hours INFORMED CONSENT STATEMENT: Your participation in this evaluation is entirely voluntary and it is your choice whether to participate or not. There are no known risks or discomforts associated with participating in this interview Would you like to continue with your participation? Do you mind if we record and transcribe the interview? [Interviewer CHECK the BOX for consent] YES NO Interviewer: EB SM DC CN CF Name: Position: Location: Date: Time of Interview: IMPLEMENTATION ● Introduction Facilitator(s): (5 minutes) • Welcome and thank you for sharing your time today. • My name is [INSERT NAME] and I am one of the Senior Technical Leads for this evaluation. • My team members are [INSERT team names and members introduce themselves and their roles] • We are part of the Evaluation team for the PEER Program. • You are here today because were identified as PEER PIs or Co-PIs. • Our goal today is to have a conversation with you about your experience as PEER researchers, and your unique experience as female scientists • There are no right or wrong comments. • The information you provide will be collapsed and responses will be stripped of your name/personal identifiers ▪ hence I will only have summaries of comments as a group ● Introduction of the Attendees: (10 minutes) • Share your name and your school/major • Ice breaker 11 0 ● Review Agenda: (10 minutes) • Review process ▪ First we will start with introductions – which we already completed ▪ Then I will ask the group a series of questions and we will have a discussion and makes some notes together ▪ For each question: ● We will record as a group the answers ● We will then write out and summarize as a group ▪ There will be one break, at the half-way mark of the Focus Group where we can have snacks and drinks ▪ I will also open the session for questions you might for each other ▪ Finally, we will close the focus group, some final comments/videos and or thoughts • Set expectations ▪ “OK, as a group, first we are going to take a few minutes to set the rules for our group” ● You will each give me some rules we might want to follow ● QUESTIONS QUESTION 1. What inspired you to become a scientist? Probe: - What keeps you in the field of science? - Who do you go to for advice? QUESTION 2. What factors contribute to your scientific productivity? Probe: - What about PEER works, what doesn’t in terms of helping you advance your career, produce papers, conduct research, offer new courses/curriculum? - What do you feel is the value of the products you completed or are working on with PEER funds? - Any specific changes you’ve seen due to these products? QUESTION 3. Thinking about the next generation of scientists in your country – the young men and women you inspire – what type of support do you think they need/will need? Probe: - what type of support do you need to help them be successful? - What contributes to the scientific capacity of a nation, your nation? QUESTION 4. For you, what are the barriers within your country, within your institutions to do quality research, to be innovative and inventive, to make the new discoveries? Probe: - How could the climate within your country/institute change to help you succeed? QUESTION 5. Open Time Some examples: 4. All comments are valid 5. Everyone should be respectful and open to other’s comments 6. The space we are in is a safe and respectful space 11 1 VLOG BOOTH – volunteer “Tell us why you are a scientist and how you plan to change your country with your creativity/innovative power/skills?” VIDEOS/VLOGS for PEER Female PI/Co-PIs OTHER NOTES 11 2 MOST SIGNIFICANT CHANGE PROCESS GUIDE 113 114 115 116 117 On behalf of the USAID Partnerships for Enhanced Engagement in Research (PEER) Program, a team of independent researchers - from the consulting firms of Cultural Practice and QED – are conducting an External Midterm Performance Evaluation. Your participation and honesty will help us provide a more accurate assessment of PEER so the managers may improve the program. The purpose of the evaluation is to assess the management and implementation of the PEER program. This Evaluation is NOT an audit neither is the purpose to assess individual performance of any project or individual. Through this survey, we will be asking you a series of questions about: · your experience with PEER · outcomes and impacts from your participation · prior expectations of PEER · collaborative activities · and changes in scientific capacity This survey should take you between 45 minutes to 1 hour – depending on the number of activities, team size, and phase of your project. The information you provide will be kept confidential and all personally identifiable information will be removed from each survey, as each person will be given a unique numeric ID for analysis. The final evaluation report will be available through the PEER website. Contact Information: if you have any questions about this survey or the evaluation, please contact the Senior Evaluation Technical Lead for this Survey - Dr. Carolyn Fonseca at Peerevaluationsurvey@gmail.com We thank you for your time and help! Introduction and Consent Form 2016 PEER Midterm Performance Evaluation PEER PI/CoPI Survey * 1. Before you start, please click on the box to provide consent to participate in this survey. Yes, I consent to participate 1 The following questions will gather information about you, your current PEER project, and characteristics about your PEER participation. Participant Background Information 2016 PEER Midterm Performance Evaluation PEER PI/CoPI Survey 2. What type of PEER participant are you? PEER Principle Investigator (PI) PEER CoPI PEER Team Member US Scientist Partner (US Agency Grant PI) US Scientist Team Member (US Agency Grant CoPI or other) Other (please specify) Other (please specify) 3. How did you become aware of PEER? (MARK ALL that apply) heard about it from a colleague at work read about it online received a message from a collaborator was invited to participate by the USAID Office (Mission) in my country a person from one of my country's government agencies told me attended a PEER presentation I was invited to participate by the National Academies of Sciences (NAS) 2 4. Under which PEER cycle did you receive funding? (MARK ALL that apply) Science Cycle 1 (awards announced May 2012) Science PIRE Cycle (awards announced December 2012) Science Cycle 2 (awards announced June 2013) Health Cycle 1 (awards announced July 2013) Health Cycle 2 (awards announced August 2014) Science Cycle 3 (awards announced August 2014) Cycle 4 (awards announced August 2015) Cycle 5 (awards announced 2016) 5. What was the size your PEER award/grant? less than $30,000 $30,000-$49,999 $50,000-$99,999 $100,000-$199,999 $200,000-$299,000 more than $300,000 6. At what stage is your PEER project? less than 12 months 12 to less than 18 months 18 to less than 24 months 24 months or more has not started completed do not know 3 Other (please specify) 7. Other than PEER funding, between January 1 to December 31 of 2015, what other types funding did you have for your research: (MARK ALL that apply) my Government my University/Institution USAID (non-PEER funding) other US Government funding (not USAID) Department for International Development United Kingdom (DfID) The Bill and Melinda Gates Foundation The World Bank other organizations outside my country private funding 8. Is this the first time you participated in an international research project? Yes No 9. Is this your first grant? Yes No 10. Is this the first time working with a scientist from another country? Yes No 11. Is this your first time as a Principle Investigator(PI)/CoPI? Yes No 12. Is this the first time you have been funded by an international organization? Yes No 4 13. As a scientist, currently WHERE are you in your career? less than 1 year 1 year to less than 3 years 3 years to less than 5 years 5 years to less than 10 years 10 years or more Other (please specify) 14. What is your current full-time position? Senior/Full Professor Junior/Associate/Assistant Professor University Lecturer Visiting Faculty Research Scientist College Dean/Department Chair/College Rector/ or equivalent Extension Agent/Faculty 15. Are you male or female? Female Male 16. In what country do you live? 5 In this section we will ask you questions about your expectations concerning PEER, your experience, and level of interaction with the various types of participants/stakeholders. Expectations and Experience 2016 PEER Midterm Performance Evaluation PEER PI/CoPI Survey Other (please specify) 17. What did you EXPECT to gain through PEER? (MARK ALL that apply) new/advanced technical and/or scientific skills capacity to acquire more funding for my research new material/expertise to teach courses/classes more graduate students to support my research develop a new product (journal article, technology, model, data, laboratory process, other) build NEW relationships with other scientists grow existing relationships with other scientists provide my community with a solution change the information my government uses to make decisions improvement in/or access to equipment/technology 18. To date, how would you describe your overall EXPERIENCE with the PEER program? very poor poor good very good outstanding too new to PEER and not able to state at this time 6 19. What would you change to improve the PEER call? (MARK ALL that apply) call/solicitation release (how it is publicized/communicated/released to the public) format of the application timing/cycle of the solicitation and/or award funding amount (increase) process in which the funding is dispersed/sent to the scientist time it takes to receive the funding matching process with the US-funded scientist type of research allowable the outputs required of the project reporting frequency (decrease) Other (please specify) 7 Interaction with NAS and Other Groups 2016 PEER Midterm Performance Evaluation PEER PI/CoPI Survey unsatisfactory needs improvement meets expectations exceeds expectations exceptional N/A communicated programmatic information responsiveness to PI/CoPI questions taken action to resolve conflict provided funding in a timely manner communicated expectations accurately concerning reporting or other PEER requirements connected me or provided access to US Scientists Partner(s) Other (please specify) 20. During your time as a PEER PI/CoPI, how well has the National Academies of Sciences (NAS) performed the following: 8 No Interaction Very Negative Somewhat Negative Neither Positive or Negative Somewhat Positive Very Positive N/A my PEER team members my PEER students my institutional leadership US Scientist Partner US Scientist Partner CoPI US Government Funding Agency of my US Scientist Partner my Government representatives National Academies of Sciences (NAS) USAID staff in my country USAID Washington DC US Embassy staff other Government representatives community leaders private sector partners Other (please specify) 21. Please rate your interaction with the following groups: 9 The following questions are focused on improving our understanding of the types of products scientists' are producing under PEER support. In addition, there are questions concerning the use, communication/sharing, and characteristics of the products. PEER Products and Outcomes 2016 PEER Midterm Performance Evaluation PEER PI/CoPI Survey 22. Under PEER support, did you produce any of the following products? (MARK ALL that apply) * if your product is "in-process" and has not been completed DO NOT include; include only those completed patent proposal submitted patent approved peer-reviewed journal article(s)s published peer-reviewed journal article(s) under review peer-reviewed journal article(s) submitted but rejected new grant proposals for new funding opportunities (non-PEER) new grant proposal for another PEER call new courses new training new workshop new training material blogs/vlogs/videos with scientific instruction paper/poster presentation at a NATIONAL scientific conference paper/poster presentation at a INTERNATIONAL scientific conference invited speaker at a NATIONAL event/meeting/conference invited speaker at an INTERNATIONAL event/meeting/conference paper/poster presentation at my Department/Institution new MS/PhD thesis concluded by one of my students during my time in PEER new MS/PhD thesis concluded for myself scientific report(s) tools new theories new technology new method open access data set(s) stakeholder workshops/training technical and/or managerial guidelines/documents market analysis documents/reports policy recommendations/analysis for government agencies business plans programmatic strategies None Other (please specify) 10 Other (please specify) 23. What did you do with your products produced through PEER support? (MARK ALL that apply) shared with other people in my scientific community did nothing with the product(s) sent to USAID posted on institutional webpage/newsletter posted on social media (Twitter, Facebook) sent to US Scientist Partner(s) shared with my University Dean/Rector/Department Chair sent to/shared with an NGO or international organization sent to my collaborators sent to my local Government Agency shared with a local community Other (please specify) 24. On average, what is your most popular/common format to share your research findings/scientific products? via email through personal conversations online platforms at events Other (please specify) 25. In your opinion, how do you think individuals are USING your product(s)/results created from your PEER partnership? (MARK ALL that apply) to change policy at the national level as guidance to communities as information for improved decision-making by my government to build other products/technologies to change policy at the community/village level to improve programs at government agencies to improve evidence in my field of research to improve programs at USAID no one is using my products do not know 11 26. How SATISFIED are you with your PEER experience? extremely unsatisfied unsatisfied neutral satisfied extremely satisfied 12 Scientific Productivity Factors 2016 PEER Midterm Performance Evaluation PEER PI/CoPI Survey no impact positive impact negative impact length of project timeline size of project budget level of support from the National Academies of Sciences (NAS) level of contact with USAID from Washington DC level of contact with USAID from my country infrastructure/equipment level of support from my home institution my own technical skills and capacity access to skilled students access to technical staff frequency of communication with US Scientist Partner quality of communication with US Scientist Partner 27. What impact did the following factors have on your ability to create/produce PEER products (i.e. publications, patents, technologies, new courses, tools, data set, new training materials, and other PEER products)? 13 28. Which of the following do you find MOST difficult in a journal publication process? selecting the topic selecting the appropriate journal understanding the requirements for the selected journal writing the article understanding the comments for revisions from the journal Other (please specify) 14 PEER Effects 2016 PEER Midterm Performance Evaluation PEER PI/CoPI Survey strongly disagree disagree neither agree of disagree agree strongly agree "receiving the PEER grant has helped me to leverage new funding" 29. How strongly do you agree with the following statement: strongly disagree disagree neither agree of disagree agree strongly agree "receiving PEER support has helped me advance my career" 30. How strongly do you agree with the following statement: strongly disagree disagree neither agree of disagree agree strongly agree "as a PEER participant I have been recognized by my institution as a leading scientist" 31. How strongly do you agree with the following statement: strongly disagree disagree neither agree of disagree agree strongly agree "as a PEER participant I have more students attending my courses" 32. How strongly do you agree with the following statement: strongly disagree disagree neither agree of disagree agree strongly agree "as a PEER participant I have been recognized by my colleagues in my field as a leading scientist" 33. How strongly do you agree with the following statement: 15 16 Mentoring and Students 2016 PEER Midterm Performance Evaluation PEER PI/CoPI Survey 34. Are you currently a mentor? Yes No 35. Would you like to have a MENTEE (someone to Mentor)? Yes No 36. Do you currently have a mentor? Yes No 37. Would you like to have a mentor? Yes No 38. What are the barriers to finding a mentor? 17 Number of Students Undergraduate students (BS, BA, or equivalent) Graduate students (MS, MA, or equivalent) Doctoral students (PhD or equivalent) Medical students (MD Medicine, MDV Veterinary medicine, or equivalent) Post-Doctoral scholars research collaborators or technical assistants 39. To date how many FEMALE students have you supported directly with PEER funding? (examples: to work in your lab, support your research, help teach courses, graduate research assistantships, graduate teaching assistantships, paid academic expenses, other) Number of Students Undergraduate students (BS, BA, or equivalent) Graduate students (MS, MA, or equivalent) Doctoral students (PhD or equivalent) Medical students (MD Medicine, MDV Veterinary medicine, or equivalent) Post-Doctoral scholars research collaborators or technical assistants 40. To date how many MALE students have you supported directly with PEER funding? (examples: to work in your lab, support your research, help teach courses, graduate research assistantships, graduate teaching assistantships, paid academic expenses, other) 18 41. With the support of PEER funding, have you provided indirect support for students? (mentoring, travel for presenting research, workshops, new course curriculum, new technical training) Yes No Other (please specify) 19 We would like to understand what types of collaborations you have experienced - under PEER - and which of these has been most productive/helpful. Collaboration and Networks 2016 PEER Midterm Performance Evaluation PEER PI/CoPI Survey Other (please specify) 42. PRIOR to your PEER participation, had you interacted (talked, emailed, held meetings) with any of the following? (MARK ALL that apply) National Academies of Sciences (NAS) USAID US Embassy Staff US Scientists US Science Government Agencies (examples: NOAA, NSF, NASA, NIH, other) my country's National Science institution(s)/academies my country's Government Agencies Rector/Dean of my university Scientists from my Department(s) of my University/Institution Scientists from other Departments Scientists from other Universities in my country Non-Profit organizations Private Sector/For-Profit organizations 20 43. How did you first meet your US Science Partner? (MARK ALL that apply) I worked with this person on prior research projects we did not know each other prior to PEER I hired him/her he/she hired me the US Science Partner had hired me we consulted together I was an invited speaker/scientist/lecturer to the US Scientist Partner's home institution we met at a conference we wrote a paper together the US Scientist Partner was my graduate student I was the US Scientist Partner's graduate student we were in graduate school together the US Scientist Partner attended one of my training/workshop/lectures we met at my University/Institution during one of my US Scientist Partner visits Other (please specify) 44. How many years have you known your US Scientist Partner prior to working with him/her on PEER? never met less than 6 months 6 months to less than 3 years 3 to less than 5 years 5 years or more 21 Other (please specify) 45. What type of support did you receive from the US Scientist Partner in preparing the PEER application/proposal? (MARK ALL that apply) no support, I created the proposal alone we (US Scientist Partner and I) put equal effort into writing the proposal I wrote a larger portion than the US Partner PI I wrote the technical section I wrote the background/context section I helped to write the budget section 46. Would you collaborate with your US Scientist Partner/Team again? Yes No Not at all Slightly Somewhat Moderately Extremely N/A National Academies of Sciences (NAS) USAID US Scientist Partner (or his/her CoPI) my institutional leadership colleagues from my institution Other (please specify) 47. DURING your time as a PEER participant, how BENEFICIAL has your collaboration/interaction been with the following groups for your scientific productivity? 22 Other (please specify) 48. As a result of participating in PEER, CHECK ALL NEW connections with any of the following: (mark only NEW connections - if you had a previous connection with that organization prior to PEER leave BLANK) International donor organizations scientists from other countries USAID institutions/universities from other countries National Academies of Sciences (NAS) Government Agencies other scientific organizations 49. How many professional networks do you belong to? None 1-3 more than 3 no change greatly weakened weakened strengthened greatly strengthened created NEW relationships N/A scientists inside your institution/university/organization 50. To what extent has PEER changed your relationships with: no change greatly weakened weakened strengthened greatly strengthened created NEW relationships N/A scientists outside your institution/university/organization within your country 51. To what extent has PEER changed your relationships with: no change greatly weakened weakened strengthened greatly strengthened created NEW relationships N/A scientists from other countries 52. To what extent has PEER changed your relationships with: 23 no change weakened strengthened created NEW relationships National Academies of Sciences (NAS) USAID 53. To what extent has PEER CHANGED your relationship with the following: 54. What could PEER do to help improve collaborations in the future? 24 For each of the following groups, please select the frequency with which you communicate your PEER results/findings: Frequency of Communication 2016 PEER Midterm Performance Evaluation PEER PI/CoPI Survey Never Once a year or less A few times a year Several times a month Several times a week N/A my institutional leadership my government agencies my local community my US Scientist Partner(s) for PEER the US Government Agency representative for my US Scientist Partner my science agencies or academies non-profit organizations in my country international organizations USAID National Academies of Sciences (NAS) 55. How frequently do you communicate your PEER results with the following: 25 In this section, we ask you about your own personal growth and/or change in your technical capacity from participating in PEER. Capacity Building 2016 PEER Midterm Performance Evaluation PEER PI/CoPI Survey Other (please specify) 56. What kind of TRAINING could have helped you better achieve the goals of your PEER project? (MARK ALL that apply) proposal writing technical training in my field communicating research results peer-reviewed journal paper writing grant management/project implementation strategies 57. In your opinion, what is the MOST IMPORTANT factor to build capacity in science and technology in your country? scientists students government policies 58. How often per YEAR do you present your research at professional conferences? 0-1 times 2-3 times more than 3 times 26 59. NOT including PEER publications, please share your total number of published peer-reviewed journal articles: 0 1-3 4-6 7-10 11-15 more than 15 Other (please specify) 60. NOT including your PEER award(s), please share largest grant/award you have won/received to date in US dollars: (check the box that best reflects the range under which your biggest award would fall under and only count those which you won; do not count those for which you applied but did not win) $0-$4,999 $5,000-$9,999 $10,000-$24,999 $25,000-$49,999 $50,000-$99,999 $100,000-$199,999 more than $200,000 Yes No N/A new evidence/data/methods in my field other professional networks in my field 61. Has your participation in PEER changed your awareness of the following? 27 Yes No N/A infrastructure (labs, equipment) teaching (course/curriculum offerings) information about collaborative opportunities international funding opportunities 62. Has your participation in PEER changed access regarding the following? Yes No N/A collect data analyze data write research articles communicate your results 63. Has your participation in PEER changed your ability to do the following? Yes No N/A prestige within my institution/organization/university prestige within my field career advancement/salary increase 64. Has your participation in PEER changed any of the following? 28 very untrue somewhat untrue neutral somewhat true very true "I can easily convince people to work with me on a project/activity" "I have a hard time leading a team or group of people" "teaching comes naturally to me" "I can easily introduce myself to strangers" "I have trouble sharing my career accomplishments with strangers" "I can easily present in public my research findings, such as at a scientific conference" 65. Thinking about your personality, how likely are each of the following statements? 66. In your opinion, do you feel work/research commitments interfere with your family/home responsibilities? Yes No 67. In your opinion, do you feel family/home responsibilities interfere with your work/research commitments? Yes No 29 The following section has questions about the environment in which you do research. Institutional Characteristics 2016 PEER Midterm Performance Evaluation PEER PI/CoPI Survey not at all very collaborative: suggests there is a cooperative climate where people easily work together, and/work towards a common goal competitive: suggests an environment is competitive and rivalrous can be positive or negative supportive: suggests the institution/university provides support or helps to achieve ones goals, being helpful friendly: suggests your institution is kindly amicable and helpful, encourages positive interactions between people divisive/hostile: the environment is unfriendly, abusive, and/or allows harassment causing the employee to feel alone, uncomfortable, or scared 68. Please rate on a scale from 1-5, where 1 is "not at all" and 5 is "very", how collaborative, competitive, supportive, friendly, and divisive/hostile is your institution's environment: 30 Yes No N/A provide me time off for family issues provide me time for training allow flexibility in my schedule provide funding for personal capacity building activities reduce teaching/course load for more time to do research offer childcare options or support Other (please specify) 69. Does your institution support the following? 70. In your opinion, what are the major barriers to INNOVATION in your institution? (MARK ALL that apply) funding accountability structures grant management capacity lack of access to scientific databases/journals physical infrastructure (labs/equipment) information technology appropriate staff quality of research institutional policies Other (please specify) 31 Recommendations for PEER 2016 PEER Midterm Performance Evaluation PEER PI/CoPI Survey 71. What do you like best about PEER? 1. 2. 3. 72. If you could change any 3 things in PEER what would those be and why? 73. What do you think are the biggest obstacles/barriers for scientists in your country to access/participate in PEER? (MARK ALL that apply) information about funding opportunities information about the PEER call capacity to write a proposal access/matching to a US Scientist partner Other (please specify) 32 PEER Project Characteristics 2016 PEER Midterm Performance Evaluation PEER PI/CoPI Survey Country Country Country Country Other (please specify) 74. From the drop-down menu, select the country(ies) where you are conducting PEER activities/work/research: * if only one country leave the rest blank Primary Field of Focus 75. Please select the field that best describes the primary focus (i.e. 50% or more of your research/work is in this field) of your PEER project: Secondary Field of Focus 76. Does your PEER project have a secondary field focus (i.e. less than 50% of the project activities are in this field)? (IF you DO NOT have one, select "DO NOT HAVE ONE" from the drop down menu) 77. Which aspect of the PEER program is MOST important to you? capacity building activities (training, workshops, technical training) research productivity (new products, technology, new data, other) collaboration and networking (expanding connections to other scientists and/or key partners) 33 Biographical Information 2016 PEER Midterm Performance Evaluation PEER PI/CoPI Survey 78. Please write the name of your current full-time home Institution/Organization: * your privacy will maintained and your information will not be shared with third parties. 79. How old are you? less than 20 years old 20-25 years 26-30 years 31-40 years 41-50 51-60 61 or older 80. What is your marital status? never married/single married unmarried, living with a partner/common law marriage widowed separated divorced 34 Your Education 2016 PEER Midterm Performance Evaluation PEER PI/CoPI Survey Other (please specify) 81. Mark ALL the degrees you have completed: Doctor of Philosophy (PhD) Post-Doctoral Research Medical Doctor (Degree in Medicine) Doctor of Veterinary Medicine Juris Doctor or Doctor of Laws (Law Degree) Masters Degree (MS, MA, MBA or equivalent) Bachelors Degree (BA, BS, AB or equivalent) Year degree was awarded 82. Please provide the following information for your Doctoral/or Highest Degree: primary field of your dissertation Other (please specify) 83. Please select the field that best captures the primary field of your dissertation research for your highest degree? Country Other (please specify) 84. In what country did you receive your PhD or highest degree of education? 35 85. Please write the name of the institution/university/college from which you received your PhD or highest degree of education: no literacy some primary education some secondary education some high school and/or degree or equivalent some college/university course work college/university degree Do not know Mother Father 86. What is the highest educational attainment of your mother and father? 87. If your father has college/university education, was it focused in science? Yes No Do not know 88. If your mother has college/university education, was it focused in science? Yes No Do not know 36 Household Characteristics 2016 PEER Midterm Performance Evaluation PEER PI/CoPI Survey 89. Including yourself, please select the number that best represents your full-time household size. (Include babies, children, elderly and yourself) 1 2-3 4-6 more than 6 total number 0- 5 years of age 6-18 years 19 or older none 90. How many dependents (children/adults) do you have who receive at least half or more of their financial support from you? 91. Are you the sole/only financial provider for your household? Yes No 37 92. Annually, what is your total income in US Dollars? Retired I do not wish to share less than $5,000 $5,001-$10,000 $10,001-$20,000 $20,001-$30,000 $30,001-$40,000 $40,001-$50,000 $50,001-$60,000 $60,001-$70,000 $70,001-$80,000 $80,001-$90,000 $90,001-$100,000 $100,001-$200,000 more than $200,000 38 Please feel free to write any other comments you might wish to share. Other Comments 2016 PEER Midterm Performance Evaluation PEER PI/CoPI Survey 93. Thank you for your time! Final Comments: 39 On behalf of the USAID Partnerships for Enhanced Engagement in Research (PEER) Program, a team of independent researchers - from the consulting firms of Cultural Practice and QED – are conducting an External Midterm Performance Evaluation. Your participation and honesty will help us provide a more accurate assessment of PEER so the managers may improve the program. The purpose of the evaluation is to assess the management and implementation of the PEER program. This Evaluation is NOT an audit neither is the purpose to assess individual performance of any project or individual. Through this survey, we will be asking you a series of questions about: · your experience with PEER · outcomes and impacts from your participation · prior expectations of PEER · collaborative activities · and changes in scientific capacity This survey should take you between 30 to 45 minutes – depending on the number of activities, team size, and phase of your project. The information you provide will be kept confidential and all personally identifiable information will be removed from each survey, as each person will be given a unique numeric ID for analysis. The final evaluation report will be available through the PEER website. Contact Information: if you have any questions about this survey or the evaluation, please contact the Senior Evaluation Technical Lead for this Survey - Dr. Carolyn Fonseca at Peerevaluationsurvey@gmail.com We thank you for your time and help! Introduction and Consent Form 2016 PEER Midterm Performance Evaluation US Partner Scientist Survey * 1. Before you start, please click on the box to provide consent to participate in this survey. Yes, I consent to participate 1 The following questions will gather information about you, your current partnership with the PEER project(s), and characteristics about your participation in PEER. Participant Background Information 2016 PEER Midterm Performance Evaluation US Partner Scientist Survey 2. What type of PEER participant are you? US Scientist Partner (US Agency Grant PI) US Scientist Team Member (US Agency Grant CoPI or other) PEER Principle Investigator (PI) PEER CoPI PEER Team Member Other (please specify) Other (please specify) 3. How did you become aware of PEER? (MARK ALL that apply) heard about it from a colleague at work read about it online received a message from a collaborator invited to participate by my USG funding Agency (i.e. NOAA, USGS, NSF, NASA, NIH, etc.) invited to participate by the USAID Office attended a PEER presentation invited to participate by the National Academies of Sciences (NAS) 2 4. Who funds your current USG grant partnered with PEER? National Aeronautics and Space Administration (NASA) National Science Foundation (NSF) National Institutes of Health (NIH) National Oceanic and Atmospheric Administration (NOAA) Smithsonian Institution U.S. Department of Agriculture (USDA) USDA Agricultural Research Service (ARS) USDA Forest Service USDA National Institute of Food and Agriculture (NIFA) U.S. Geological Survey (USGS) General Electric India Other (please specify) 5. At what stage of your USG grant did you add the PEER PI/Project? less than 12 months 12 to less than 18 months 18 to less than 24 months 24 months to less than 3 years 3 years or more 3 6. What is the size of your USG grant/award? (i.e. the grant you have from your USG funding agency to which your PEER partner has been added) less than $30,000 $30,000-$49,999 $50,000-$99,999 $100,000-$199,999 $200,000-$299,999 $300,000-$399,999 $400,000-$499,999 $500,000 to $1 Million more than $1 Million 7. Please select the PEER Cycle(s) in which you have partnered with PEER projects? (MARK ALL that apply) Science Cycle 1 (awards announced May 2012) Science PIRE Cycle (awards announced December 2012) Science Cycle 2 (awards announced June 2013) Health Cycle 1 (awards announced July 2013) Health Cycle 2 (awards announced August 2014) Science Cycle 3 (awards announced August 2014) Cycle 4 (awards announced August 2015) Cycle 5 (awards announced 2016) 8. To date, what is your total number of PEER partnerships? (i.e. number of PEER Projects for which you have been a US Partnering Scientist) 9. How many years did you know your PEER PI prior to working with them on PEER? Did not know them at all less than 6 months 6 months to less than 3 years 3 to less than 5 years 5 years or more 4 Other (please specify) 10. Other than your USG Grant/Award, what other type of funding were you receiving between January 1 to December 31 of 2015? (MARK ALL that apply) my University/Institution other USAID funding (non-PEER funding) other US Government funding (not PEER related) The Bill and Melinda Gates Foundation The World Bank private funding 11. For the USG grant connected to the PEER Partner, is it your first time as a Principle Investigator(PI)/CoPI? Yes No 12. Is this the first time you participated in an international research project? Yes No 13. Is this your first grant? Yes No 14. Is this the first time working with a scientist from another country? Yes No 5 15. As a scientist, currently WHERE are you in your career? less than 1 year 1 year to less than 3 years 3 years to less than 5 years 5 years to less than 10 years 10 years or more Other (please specify) 16. What is your current full-time position? Senior/Full Professor Junior/Associate/Assistant Professor University Lecturer Visiting Faculty Research Scientist College Dean/Department Chair/College Rector/ or equivalent Extension Agent/Faculty 17. Are you male or female? Female Male 6 In this section we want to know more about your USG funded Grant partnered with PEER, your contribution, and the research connection to the PEER project. Your U.S. Government (USG) Grant 2016 PEER Midterm Performance Evaluation US Partner Scientist Survey 18. How closely is the PEER project tied to your USG funded award? not at all somewhat very 19. How did partnering with PEER PI expand your research? not at all somewhat supportive very N/A level of support 20. How supportive was/is your USG funding agency of your PEER participation? unsatisfactory needs improvement meets expectations exceeds expectations exceptional N/A communicated PEER programmatic information responsiveness to my PEER related questions taken action to resolve conflict connected me or provided access to PEER Scientists Other (please specify) 21. During your time as a US Partnering Scientist to PEER, how well has your Program Officer at your Funding Agency performed the following: 7 22. Did you provide other types of funding to support your PEER participation? yes no yes no Travel Students Equipment Other (please specify) 23. Did you request supplemental funding from your USG funding agency to support your PEER participation? 24. Was your request for supplemental funding for PEER participation/activities approved from your USG funding agency? yes no pending approval 8 This section is focused on obtaining information about your interaction with the PEER PI. Interactions with PEER PIs 2016 PEER Midterm Performance Evaluation US Partner Scientist Survey 25. What was the primary reason for taking on a PEER project? 26. How many potential PEER PI partners contacted you? 1 2-3 more than 3 Other (please specify) 27. How did your current PEER PI partner contact you? phone call email through a colleague through the PEER Team through my USG Funding Agency 28. What was the PRIMARY reason you chose your current PEER PI partner? scientist expertise and qualifications previous experience with the scientist recommended by a colleague/friend location of research topic of research the project augmented my current grant Other (please specify) 9 Other (please specify) 29. What type of support did you provide the PEER PI in preparing the PEER application/proposal? (MARK ALL that apply) no support, the PEER PI created the proposal alone we (PEER PI and I) put equal effort into writing the proposal I wrote a larger portion than the PEER PI I wrote the technical section I wrote the background/context section l I helped to write the budget section 30. Would you collaborate with your PEER PI again? Yes No 31. PRIOR to PEER, had you submitted to any other research grant proposals with your current PEER PI partner? yes no 10 32. If you knew your PEER PI prior to partnering through PEER, in what way did you know him/her? (MARK ALL that apply) we did not know each other prior to PEER I had worked with this person on prior research projects I had hired him/her he/she hired me we consulted together I was an invited speaker/scientist/lecturer to the Scientist's home institution we met at a conference we wrote a paper together the US Scientist Partner was my graduate student I was the partnering PEER PI graduate student we were in graduate school together the partnering PEER PI attended one of my training/workshop/lectures my institution has an existing relationship with the PEER PI's home institution Other (please specify) 11 No Interaction Very Negative Somewhat Negative Neither Positive or Negative Somewhat Positive Very Positive N/A PEER PI PEER CoPI my institutional leadership my US Government Funding Agency National Academies of Sciences (NAS) USAID PEER PI's institution PEER PI's students other PEER team members PEER PI country's government agencies Other (please specify) 33. Please rate your interaction with the following groups: 34. How could PEER improve the matching/partnering process? 12 In this section we will ask you questions about your expectations concerning PEER, your experience, and level of interaction with the various types of participants/stakeholders. Expectations and Experience 2016 PEER Midterm Performance Evaluation US Partner Scientist Survey Other (please specify) 35. What did you EXPECT to gain through your PEER partnership? (MARK ALL that apply) connection to scientists in the country where I do research access to new data experience working in a developing country new/advanced technical and/or scientific skills develop a new product (journal article, technology, model, data, laboratory process, other) build NEW relationships with other scientists grow existing relationships with other scientists new information for decision-making by policy makers new information for USAID expansion of my current research help scientists build capacity help scientists improve/increase their scientific productivity 36. To date, how would you describe your overall EXPERIENCE with the PEER program? very poor poor good very good outstanding to new to PEER and not able to state at this time strongly disagree disagree neither agree of disagree agree strongly agree "being a PEER Scientist Partner has helped me leverage more funding" 37. How strongly do you agree with the following statement: 13 strongly disagree disagree neither agree of disagree agree strongly agree "being a PEER Scientist Partner has changed my approach to education/student training" 38. How strongly do you agree with the following statement: strongly disagree disagree neither agree of disagree agree strongly agree "being a PEER Scientist Partner has directly benefited my students/research assistants" 39. How strongly do you agree with the following statement: strongly disagree disagree neither agree of disagree agree strongly agree "partnering with PEER has helped me advance in my career" 40. How strongly do you agree with the following statement: strongly disagree disagree neither agree of disagree agree strongly agree "as a Peer Scientist Partner the scale/scope of my research has increased" 41. How strongly do you agree with the following statement: strongly disagree disagree neither agree of disagree agree strongly agree "as a Peer Scientist Partner my scientific productivity (i.e. journal publications, creation of new technology, creation of new data/scientific information) has increased" 42. How strongly do you agree with the following statement: 14 43. What would you change to improve the PEER partnership? (MARK ALL that apply) how the PEER PI solicits my participation provide funding for the US Partner scientist length of the partnership timing/cycle of the solicitation and/or award funding amount for the PEER PI matching process with the PEER PI type of research allowable the outputs required of the project Other (please specify) #1 Benefit #2 Benefit #3 Benefit 44. List the top three ways in which you benefited directly from the PEER partnership: 15 We would like to understand what types of collaborations you have experienced as a US Scientist Partner in PEER, as well as to learn which of these interactions has been most productive/helpful. Collaboration and Networks 2016 PEER Midterm Performance Evaluation US Partner Scientist Survey Other (please specify) 45. PRIOR to your PEER participation, had you interacted (talked, emailed, held meetings) with any of the following? (MARK ALL that apply) National Academies of Sciences (NAS) USAID US Embassy Staff US Scientists US Science Government Agencies (examples: NOAA, NSF, NASA, NIH, other) my country's National Science institution(s)/academies my country's Government Agencies Rector/Dean of my university Scientists from my Department Scientists from other Departments Scientists from other Universities in my country Non-Profit organizations Private Sector/For-Profit organizations 16 46. When selecting your PEER PI partner, what 3 key characteristics must be present? (MARK ONLY your top 3) ability/openness to communicate with me ability to collaborate effectively previous experience collaborating with US scientists previous experience working on international research projects previous experience publishing peer-reviewed journal articles previous success at attaining grant funding level of prestige within the PEER PI's field in his/her country firsthand knowledge of the quality of his/her research a good fit with my style of grant/project management a good fit with my personality clarity in the work/research objectives his/her project goals must align with my project goals Other (please specify) 47. Do you expect your collaboration with the PEER PI to continue after your USG funded grant (USG award tied to PEER) ends/closes? (if your award/grant has already closed and you are still collaborating with your PEER PI select 'yes') Yes No 48. Do you expect your collaboration with the PEER PI to continue after their PEER funding has ended? (if the PEER PI's funding has ended and you are still collaborating then select 'yes') Yes No Not at all Slightly Somewhat Moderately Extremely N/A National Academies of Sciences (NAS) USAID PEER PI/CoPI PEER PI/CoPI institution USG Funding Agency Other (please specify) 49. DURING your time as a PEER US Scientist Partner, how BENEFICIAL have these collaborations/interactions been to you? 17 no change greatly weakened weakened strengthened greatly strengthened created NEW relationships N/A scientists inside your institution/university/organization 50. To what extent has PEER changed your relationships with: no change greatly weakened weakened strengthened greatly strengthened created NEW relationships N/A scientists outside your institution/university/organization within your PEER PI's country 51. To what extent has PEER changed your relationships with: no change improved weakened created NEW relationships your USG funding agency National Academies of Sciences (NAS) USAID 52. To what extent has PEER CHANGED your relationship with the following: 53. What could PEER do to help improve collaborations in the future? 54. How many professional networks do you belong to? None 1-3 more than 3 18 The following questions are focused on improving our understanding of the types of products scientists' are producing under PEER support. In addition, there are questions concerning the use, communication/sharing, and characteristics of the products. PEER Products and Outcomes 2016 PEER Midterm Performance Evaluation US Partner Scientist Survey 55. Under your PEER partnership, were any of the following produced? (MARK ALL that apply) * if your product is "in-process" and has not been completed DO NOT include; include only those completed patent proposal submitted patent approved peer-reviewed journal article(s)s published peer-reviewed journal article(s) under review peer-reviewed journal article(s) submitted but rejected new grant proposals for new funding opportunities (non-PEER) new grant proposal for another PEER call new courses new training new workshop new training material blogs/vlogs/videos with scientific instruction paper/poster presentation at a NATIONAL scientific conference paper/poster presentation at a INTERNATIONAL scientific conference invited speaker at a NATIONAL event/meeting/conference invited speaker at an INTERNATIONAL event/meeting/conference paper/poster presentation at my Department/Institution new MS/PhD thesis concluded by one of my students during my time in PEER new MS/PhD thesis concluded for myself scientific report(s) tools new theories new technology new method open access data set(s) stakeholder workshops/training technical and/or managerial guidelines/documents market analysis documents/reports policy recommendations/analysis for government agencies business plans programmatic strategies None Other (please specify) 19 Other (please specify) 56. What did you do with these PEER products? (MARK ALL that apply) shared with other people in my scientific community did nothing with the product(s) sent to USAID posted on institutional webpage/newsletter posted on social media (Twitter, Facebook) sent to US Scientist Partner(s) shared with my University Dean/Rector/Department Chair sent to/shared with an NGO or international organization sent to my collaborators sent to my USG funding agency shared with a local community Other (please specify) 57. In your opinion, how do you think individuals are USING the products created from your PEER partnership? (MARK ALL that apply) to help change policy at the national level to provide guidance to communities to provide information for improved decision-making by my government to help build other products/technologies to help change policy at the community/village level to improve programs at government agencies to improve evidence in my field of research to improve programs at USAID no one is using my products do not know 20 58. How SATISFIED are you with your PEER experience? extremely unsatisfied unsatisfied neutral satisfied extremely satisfied 21 Scientific Productivity Factors 2016 PEER Midterm Performance Evaluation US Partner Scientist Survey no impact positive impact negative impact length of project timeline size of project budget lack of budget for your participation/collaboration level of support from the National Academies of Sciences (NAS) level of contact with USAID level of support from my USG funding agency existing infrastructure/equipment level of support from my home institution my own technical skills and capacity access to skilled students level of technical capacity of PEER Scientists location of project frequency of communication quality of communication 59. What impact did the following factors have on your ability to provide PEER products? 22 60. Please share your total number of published peer-reviewed journal articles: 0 1-3 4-6 7-10 11-15 more than 15 Other (please specify) 61. How many peer-reviewed journal articles have you published with your partnering PEER PI? 0 1-3 4-6 7-10 11-15 more than 15 Other (please specify) 23 Mentoring and Students 2016 PEER Midterm Performance Evaluation US Partner Scientist Survey 62. Are you currently a mentor? Yes No 63. Would you like to have a MENTEE (someone to Mentor)? Yes No 64. Do you currently have a mentor? Yes No 65. Would you like to have a mentor? Yes No 66. What are the barriers to finding a mentor? 24 Number of Students Undergraduate students (BS, BA, or equivalent) Graduate students (MS, MA, or equivalent) Doctoral students (PhD or equivalent) Medical students (MD Medicine, MDV Veterinary medicine, or equivalent) Post-Doctoral scholars research collaborators or technical assistants 67. To date how many FEMALE students have you supported directly with your USG grant/award partnered with PEER? (examples: to work in your lab, support your research, help teach courses, graduate research assistantships, graduate teaching assistantships, paid academic expenses, other) Number of Students Undergraduate students (BS, BA, or equivalent) Graduate students (MS, MA, or equivalent) Doctoral students (PhD or equivalent) Medical students (MD Medicine, MDV Veterinary medicine, or equivalent) Post-Doctoral scholars research collaborators or technical assistants 68. To date how many MALE students have you supported directly with your USG grant/award partnered with PEER? (examples: to work in your lab, support your research, help teach courses, graduate research assistantships, graduate teaching assistantships, paid academic expenses, other) 69. Did you fund (with your own research money/resources) any PEER PI/CoPI students? (research assistantship, travel for presenting research, workshops, technical training) Yes No 25 70. Did your home institution fund any PEER PI/CoPI students? (research assistantship, travel for presenting research, workshops, technical training) Yes No 26 In this section, we ask you about the type of capacity building activities you believe would be most helpful for the PEER PI or should be required in all PEER projects. Capacity Building 2016 PEER Midterm Performance Evaluation US Partner Scientist Survey Other (please specify) 71. What kind of training should be required for all PEER PIs? (MARK ALL that apply) proposal writing advanced technical training communicating research results peer-reviewed journal paper writing grant management/project implementation strategies Other (please specify) 72. If you were provided funding, what type of capacity building activities do you think you could provide your PEER PI to help them be more successful? (MARK ALL that apply) proposal writing advanced technical training communicating research results peer-reviewed journal paper writing grant management/project implementation strategies collaborating on international projects mentoring and managing graduate students 27 73. In your experience, what factor is most important for a country in building scientific capacity? number of scientists availability of quality students government policies availability of funding collaborative opportunities willingness/interest by scientists infrastructure/scientific equipment 74. How often per YEAR do you present your research at professional conferences? 0-1 2-3 more than 3 Yes No N/A new evidence/data/methods in my field other professional networks in my field 75. Has your participation in PEER changed your awareness of the following? Yes No N/A infrastructure (labs, equipment) teaching (course/curriculum offerings) scientists at your PEER PI institution/university data information about collaborative opportunities international funding opportunities 76. Has your participation in PEER changed access regarding the following? 28 Yes No N/A collect data analyze data write research articles communicate your results 77. Has your participation in PEER changed your ability to do the following? very untrue somewhat untrue neutral somewhat true very true "I can easily convince people to work with me on a project/activity" "I have a hard time leading a team or group of people" "teaching comes naturally to me" "I can easily introduce myself to strangers" "I have trouble sharing my career accomplishments with strangers" "I can easily present in public my research findings, such as at a scientific conference" 78. Thinking about your personality, how likely are each of the following statements: 79. In your opinion, do you feel work/research commitments interfere with your family/home responsibilities? Yes No 80. In your opinion, do you feel family/home responsibilities interfere with your work/research commitments? Yes No 29 The following section has questions about the environment in which you do research. Institutional Characteristics 2016 PEER Midterm Performance Evaluation US Partner Scientist Survey not at all very collaborative: suggests there is a cooperative climate where people easily work together, and/work towards a common goal competitive: suggests an environment is competitive and rivalrous can be positive or negative supportive: suggests the institution/university provides support or helps to achieve ones goals, being helpful friendly: suggests your institution is kindly amicable and helpful, encourages positive interactions between people divisive/hostile: the environment is unfriendly, abusive, and/or allows harassment causing the employee to feel alone, uncomfortable, or scared 81. Please rate on a scale from 1-5, where 1 is "not at all" and 5 is "very", how collaborative, competitive, supportive, friendly, and divisive/hostile is your institution's environment: 30 Yes No N/A provide me time off for family issues provide me time for training allow flexibility in my schedule provide funding for personal capacity building activities reduce teaching/course load for more time to do research offer childcare options or support Other (please specify) 82. Does your institution support the following? not at all somewhat supportive very N/A level of support 83. How supportive was/is your home institution of your PEER participation? 84. In your opinion, what are the major barriers to INNOVATION in your institution? (MARK ALL that apply) funding accountability structures grant management capacity lack of access to scientific databases/journals physical infrastructure (labs/equipment) information technology appropriate staff quality of research institutional policies Other (please specify) 31 Recommendations for PEER 2016 PEER Midterm Performance Evaluation US Partner Scientist Survey 85. What do you like best about PEER? 1. 2. 3. 86. If you could change any 3 things in PEER what would those be and why? 87. What do you think are the biggest obstacles/barriers for US scientists to participate in PEER? (MARK ALL that apply) information about the PEER call access/matching to PEER PI funding amount time-frame of call with respect to my USG funded grant Other (please specify) 32 PEER Project Characteristics 2016 PEER Midterm Performance Evaluation US Partner Scientist Survey Country Country Country Country Other (please specify) 88. In what country(ies) are you conducting PEER activities/work/research? * if only one country leave the rest blank Primary Field of Focus 89. Please select the field that best describes the primary focus (i.e. 50% or more of your project is in this field) of your partnering PEER project: Secondary Field of Focus 90. Does your partnering PEER project have a secondary field focus (i.e. less than 50% of the project activities are in this field)? (IF you DO NOT have one, select "DO NOT HAVE ONE" from the drop down menu) 91. Which aspect of the PEER program is MOST important to you? capacity building activities (training, workshops, technical training) research productivity (new products, technology, new data, other) collaboration and networking (expanding connections to other scientists and/or key partners) 33 Biographical Information 2016 PEER Midterm Performance Evaluation US Partner Scientist Survey 92. Please write the name of your current full-time home Institution/Organization: * your privacy will be maintained and any information you provide will not be shared 93. How old are you? 19 years old or younger 20-25 years 26-30 years 31-40 years 41-50 51-60 61 or older 94. What is your marital status? never married/single married unmarried, living with a partner/common law marriage widowed separated divorced 34 Your Education 2016 PEER Midterm Performance Evaluation US Partner Scientist Survey Other (please specify) 95. Mark ALL the degrees you have completed: Doctor of Philosophy (PhD) Post-Doctoral Research Medical Doctor (Degree in Medicine) Doctor of Veterinary Medicine Juris Doctor or Doctor of Laws (Law Degree) Masters Degree (MS, MA, MBA or equivalent) Bachelors Degree (BA, BS, AB or equivalent) Year degree was awarded 96. Please provide the following information for your Doctoral/or Highest Degree: primary field of your dissertation Other (please specify) 97. Please select the field that best captures the primary field of your dissertation research for your highest degree? Country Other (please specify) 98. In what country did you receive your PhD or highest degree of education? 35 99. Please write the name of the institution/university/college from which you received your PhD or highest degree of education: no literacy some primary education some secondary education some high school and/or degree or equivalent some college/university course work college/university degree Do no know Mother Father 100. What is the highest educational attainment of your mother and father? 101. If your father has college/university education, was it focused in science? Yes No Do not know 102. If your mother has college/university education, was it focused in science? Yes No Do not know 36 Household Characteristics 2016 PEER Midterm Performance Evaluation US Partner Scientist Survey 103. Including yourself, please select the number that best represents your full-time household size. (Include babies, children, elderly and yourself) 1 2-3 4-6 more than 6 total number 0- 5 years of age 6-18 years 19 or older none 104. How many dependents (children/adults) do you have who receive at least half or more of their financial support from you? 105. Are you the sole/only financial provider for your household? Yes No 37 106. Annually, what is your total income in US Dollars? Retired I do not wish to share less than $5,000 $5,001-$10,000 $10,001-$20,000 $20,001-$30,000 $30,001-$40,000 $40,001-$50,000 $50,001-$60,000 $60,001-$70,000 $70,001-$80,000 $80,001-$90,000 $90,001-$100,000 $100,001-$200,000 more than $200,000 38 Please feel free to write any other comments you might wish to share. Other Comments 2016 PEER Midterm Performance Evaluation US Partner Scientist Survey 107. Thank you for your time! Final Comments: 39 11 9 Desk Review Inventory  Document  Code  Contents  Relate d Evalua tion Questi ons  File Location  FINAL Science Results Framework  1  Lab Results Framework Science Results Framework Science RF with activities mapped PEER Results Framework  All  PEER EVALUATION\Results framework and monitoring plans  Monitoring and Evaluation Performance Plan (April 2016)  2  Background on project (objectives, financial, etc.) PEER Theory of Change Developing the Performance Monitoring Plan PEER’s Performance Indicator Data Table Implementation of the PMP Reporting Evaluation Annex A: Frequency and Timing of Key Data Collection Activities Annex B: Special Indicators Annex C: Record of Indicator Changes Annex D: M&E Roles and Responsibilities Annex E: M&E Outputs Annex F: Data Quality Checklist  All  PEER EVALUATION\Results framework and monitoring plans  Policy and Program Change Framework  3  Pathway and Research Impacts Pathway Discussion of challenges Ideas for categorizing program or policy change PEER and HESN examples  Q2  PEER EVALUATION\Results framework and monitoring plans  National Academies of Sciences Quarterly Reports (18) (Oct - Dec 2011 to Jan-Mar 2016)  4 (projec t - code)  Significant Activities Problems and Challenges in Program Implementation Evaluation of Accomplishments Progress on Goals and Objectives Project Updates (Jul 2012 onward) Update on Expenditures During the Quarter (and adjustments) Supporting Documentation or Products Peer Applicants by Eligibility and by Country, Region, Field of Study and Region  All  PEER EVALUATION\Quarterly_ Reports-2016-06-23 12 0  National Academies of Sciences PEER Health Quarterly Reports (4) (Jan - Mar to Oct - Dec 2014)  5 (with additio nal codes)  Significant Activities Problems and Challenges in Program Implementation Update on Expenditures During the Quarter (and adjustments) PEER Health Program Expenses Plans for upcoming activities Project updates (Apr - Jun 2014 onward)  All  PEER EVALUATION\Quarterly_ Reports-2016-06-24  Draft Framework for Assessing Policy and Program Change  6  Challenges Pathway to Research Implementation Categorizing impact of program or policy change Examples PEER and HESN Current Science Objective Indicators and Proposed changes  Q2  PEER EVALUATION\PEER Monitoring Data  Funding and Buy￾ins  7  Project, country, PI, Partner Award Amount, U.S. partner, Grant, Funding Buy-in listed by stakeholder Summary of funding type by Cycle Supplement funding and purpose of funding  Q1, Q3  PEER EVALUATION\PEER Monitoring Data  Indicator Analysis and quality CP  8  Matches old indicators (2011 - 2016) with new indicators (2017-2020). Includes: Indicator Code, Indicator Definition, Unit of Measure, Collection Method, Source  All  PEER EVALUATION\PEER Monitoring Data  Monitoring Data NAS  9  Monitoring Data (x indicator and x year of project cycle): PEER Science - C1, C2, C3 PEER PIRE PEER Health - C1, C2 Project information with PI and U.S. partner contact information  All  PEER EVALUATION\PEER Monitoring Data  PEER Program trainings and events as of Mar 2016  10  Data on PEER program trainings and events: Start Date of Training or Event Country of training or event Name of Training or Event Type of Training or Event Topic or Description # of USAID staff participants # of non-USAID staff participants # of female non-USAID staff participants # of male non-USAID staff participants  Q3  PEER EVALUATION\PEER Monitoring Data 121  Projects master program and policy short list  11  Analysis of xx research to program/policy change. Includes: Cycle Grant Number Project name Total budget Country Primary field of research Last Name First Name Gende r E -mail of principal investigator Principal investigator's institution Research Result type FY 16 Impact Stage of Research Notes on stage Influence level of research adopters Date of latest site visit Response to follow -up actions  All  PEER EVALUATION \PEER Monitoring Data  Engagement Plan to Evaluate the Impact of STIP Public Goods Investments (May 2016)  12  Description of Lab investments in Science, Technology, Innovation and Partnerships  Q1  PEER EVALUATION \Lab Public Goods Investment Study -Dahlberg  Kickoff Meeting PEER presentation  13  Background on the PEER program  All  PEER EVALUATION \Kick - off meeting  HESN Midterm Performance Evaluation Report  14  Final HESN Evaluation and Annexes  Q1  PEER EVALUATION \HESN  PEER Evidence to Action Supplements  15  PEER EVIDENCE TO ACTION SUPPLEMENT RECIPIENTS 2016 Description of supplemental project objectives for 10 projects  Q2  PEER EVALUATION \Evidence to action supplements  Feed the Future: BIFAD Review of the CRSPs (August 2012)  16  Review of the CRSP model  Q1  PEER EVALUATION \Document s from possible comparator organizations \Feed the Future  The CGIAR at 40 and Beyond (June 2011)  17  Description of impacts over the past 40 years CGIAR Financial Highlights (971 - 2011)  Q1  PEER EVALUATION \Document s from possible comparator organizations \Feed the Future  Feed the Future Innovation Lab Fact Sheet (June 2016)  18  Brief description of each Feed the Future Lab  Q1  PEER EVALUATION \Document s from possible comparator organizations \Feed the Future 12 2  World Vegetable Center/AVRDC (n.d.) "International research on vegetable improvement in East and Southern Africa: Adoption, impact and returns"  19  Impacts of agricultural research on vegetable production in East and Southern Africa from 1990-2015. Describes returns on investment.  Q1 (Q2 )  PEER EVALUATION\Document s from possible comparator organizations\Feed the Future  The non-academic impact of international development research in UK Higher Education (May 2015)  20  "This report is based on a short study investigating the nature of impact relating to international development arising from research in UK Higher Education Institutions (HEIs) and prepared for the UK Collaborative on Development Sciences (UKCDS) - a group of 14 UK government departments and research funders working in international development. There are two areas of interest to UKCDS that have prompted this study: 1. The impact of UK HEI research arising from studies supported by the Department for International Development (DFID) 2. The impact of UK HEI research impact case studies that relate to the topic ‘international development’ more generally"  Q1 (Q4)  PEER EVALUATION\Document s from possible comparator organizations\UKCDS  UKCDS Science Impact Stories Link  21  These are projects funded by one of 17 UK-based funding agencies. They were selected because of their evidence-to-impact. Check out the last 2 paragraphs of each. These stories are well framed with the big picture, the question, the team of researchers, the results, the impact. http://www.ukcds.org.uk/the-global￾impact-of-uk-research http://h/  Q1 (Q2)  PEER EVALUATION\Document s from possible comparator organizations\UKCDS  WOTRO Science for Global Development - Research Uptake  22  Presentation of "Research Uptake" approach   PEER EVALUATION\Document s from possible comparator organizations\WOTRO￾Netherlands 12 3  WOTRO Theory of Change and Impact Pathways  23  Presentation Objectives of this presentation ‒ To give an overview on the concepts of Theory of Change (ToC) and Research Impact Pathway (IP) ‒ Understanding on the application of the Theory of Change approach and Research Impact Pathways ‒ To provide guidance on formulating policy oriented outcomes & outputs and the development of indicators  Q1 (Q2)  PEER EVALUATION\Document s from possible comparator organizations  Comparator Programs Contacts  24  Contacts from USAID/BFS, WORTO, UKCDS, and Sida  Q1  PEER EVALUATION\contact information  Contact Info NAS  25  Contacts from NAS  All  PEER EVALUATION\contact information  Mission and Regional Technical Bureau Contact List  26  Contacts from USAID Missions and Regional Technical Bureaus  All  PEER EVALUATION\contact information  USAID/NAS Site Visit Reports Indonesia (August 2015) - FOLDER  27  Seven (7) Site Visit Reports and TDY Executive Summary Site visit reports include: Summary Feed-back for Objective (1)/ Status of PEER project USAID/NAS follow up items to provide support to this project  All  PEER EVALUATION\Backgroun d materials Indonesia\Site Visit Indonesia Aug 2015  Indonesia CDCS (2014 - 2018)  28  Overview of development objectives in Indonesia  All  PEER EVALUATION\Backgroun d materials Indonesia  Projects Master (June 30, 2016) (updated 8.9.16)  29  Grant Number Project name Manager Total budget Country Region City Special categories (if applicable) Primary field of research Secondary field of research Last Name First Name Gender Citizenship E-mail of principal investigator Telephone number Principal investigator's institution Co-PI(s)  All  PEER EVALUATION 124 Other developing country institutions involved Actual start date for your PEER Science proposal Actual end date for your PEER Science proposal Grant closed? (all reports and pubs in?) Past PEER PI? U.S. partner Gender of U.S. Partner U.S. partner's e -mail U.S. partner's institution U.S. partner's funding agency FY 16 Impact Stage of Research (options in comment bubble) Notes on stage Influence level of research adopters What was the objective of the research project? Did the PI complete the research project? What are successful outcomes of the project? (papers, students trained, etc) What challenges did the PI have implementing the project? Are there evidence to policy impacts? Or program change? Description of program or policy impact or progress made towards a program or policy impact Date evidence to policy impact counted or last day of relevant reporting period) Follow up actions on this project for the PEER or Mission team Is there a potential research user/ scaler? List names of potential research users/ scalers Date of latest site visit Notes Response to follow -up actions  2013 BKK Survey  30  Summary of PEER Science Participants Feedback on a conference  Q3, Q4  PEER EVALUATION \Survey Results  2013 PEER Science Participants’ Conference Feedback  31  Additional feedback participants' experience at the conference  Q3, Q4  PEER EVALUATION \Survey Results  Lima 2015  32  Feedback on a Forum (Lima)  Q3 and Q4  PEER EVALUATION \Survey Results  PEER Amman Survey 4 -16  33  Feedback on a Forum (Amman)  Q3 and Q5  PEER EVALUATION \Survey Results  PEER Health Cycle 2 Partnership  34  Report for PEER Health Cycle 2 Partnership Visits, PI experience with proposal development  Q1  PEER EVALUATION \Survey Results  PI Publications  35  2 publications (Njoroge)  Q2  PEER 12 5 (Kenya) EVALUATION\Kenya background materials/  PI Publications (Kenya)  36  4 publications (Wandiga)  Q2  PEER EVALUATION\Kenya background materials/  PI Publications (Kenya)  37  1 publication (Owino)  Q2  PEER EVALUATION\Kenya background materials/  PI Publications (Kenya)  38  1 publication (Maina)  Q2  PEER EVALUATION\Kenya background materials/  PI Publications (Kenya)  39  1 publication (Baaru)  Q2  PEER EVALUATION\Kenya background materials/  PI Publications (Kenya)  40  2 publications (Nyingi)  Q2  PEER EVALUATION\Kenya background materials/  PI Publications (Kenya)  41  1 publication (Gakuya)  Q2  PEER EVALUATION\Kenya background materials/  PI Publications (Kenya)  42  2 publication (Kimiywe)  Q3  PEER EVALUATION\Kenya background materials/  PEER Grant Initiation Cycles 2- 4 (All projects)  43  Cycle 2-4:  PEER Baseline Data Form Basic Project Information Infrastructure Development (Prior grants submitted, publications, educational impacts, supplementary information) PEER Grant Set-Up Form (contact info)  Q4  Foundant  PEER Quarterly Reports and Annual Reports (PIs)  Cycles 2-4 (all projects)  44   Project Name Summary Project events Major Equipment Purchased Challenges Additional Information Photos Documents  All  Foundant  Event Participant Lists  45  Documents announcing individual participants’' participation in an event  Q3  Foundant  Baseline Assessment  46  This baseline assessment was used to determine the impact that courses had on participants. Includes questions on barriers to conducting research, number of research and grant applications, awards/public recognition, access to mentors and mentors at institution, guidance for implementing research, writing grant proposals, professional development,  Q2, Q3, Q4  126 access to research publications  Cycle 1 Projects - Annual Reports - Bangladesh  47  1-97 (2013, 2014, 2015, Application) 1-225 (2013, 2014, 2015, Application)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Reports - Bolivia  48  1-358 (2013, 2014 and application)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Reports - Columbia  49  1-31 (2013, 2014, 2015 and application)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Reports - Ecuador  50  1-108 (Application, 2013, 2014, 2015, 2016, Annual Financial Report 2016) 1-384 (Application, 2013, 2014, 2015)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - Egypt  51  1-368 (Application, 2013)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - El Salvador  52  1-354 (Application, 2013, 2015)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - Ethiopia  53  1-289 (Application, 2013, 2014, 2015)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - Georgia  54  1-170 (Application, 2013, 2014, 2015)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - Ghana  55  1-142 (Application, 2013, 2014)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - India  56  1-32 (Application, 2013, 2014, 2015) PP -27 (Application, 2013, 2014, 2015)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - Indonesia  57  1-21 (Application, 2013, 2014) 1-90 (2013, 2014, 2015) 1-102 (Application, 2013, 2014, 2015) 1-152 (Application, 2013, 2014, 2015) 1-205 (Application, 2013, 2014, 2015) 1-208 (Application, 2013 2014, 2015) 1-235 (Application, 2013, 2014, 2015)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - Jordan  58  1-146 (Application, 2013, 2014)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive 127  Cycle 1 Projects - Annual Report - Kenya  59  1-198 (Application, 2013, 2014, 2015) 1-207 (Application, 2013, 2014) 1-382 (Application, 2013, 2014, 2015)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - Lebanon  60  1-84 (Application, 2013, 2014, 2015) 1-91 (Application, 2013, 2014) 1-121 (Application, 2013, 2014, 2015) 1-163 (Application, 2013, 2014, 2015) 1-228 (Application, 2013, 2014)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - Malawi  61  1-307 (Application, 2013, 2014, 2015)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - Mexico  62  PP -10 (Application, 2013, 2014, 2015)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - Mongolia  63  1-15 (Application, 2013, 2014, 2015) 1-98 (Application, 2013, 2014)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - Morocco  64  1-375 (Application ,2013, 2014)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - Nepal  65  1-183 (Application, 2013, 2014)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - Peru  66  1-353 (Application, 2013, 2014, 2015 + 2015 Technical Report)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - Philippines  67  1-34 (Application, 2013, 2014, 2015) 1-177 (Application, 2013, 2014, 2015)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - Sri Lanka  68  1-194 (Application, 2014, 2015)  All  PEER EVALUATIO N \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - Tanzania  69  1-232 (Application, 2013, 2015)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - Uzbekistan  70  1-41 (Application, 2013, 2014, 2015)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive  Cycle 1 Projects - Annual Report - Vietnam  71  1-243 (Application, 2013, 2014, 2015) 1-319 (Application, 2013, 2014, 2015)  All  PEER EVALUATION \Annual Reports + Applications - Google Drive 13 1 ANNEX IV: SOURCES OF INFORMATION INDIVIDUAL KEY INFORMANT INTERVIEWS Name Organization Position/Title PEER Stakeholder Category Dalal Najib NAS Senior Program Officer NAS Staff Daniel Placht NAS Program Associate NAS Staff Kelly Robbins NAS Program Director NAS Staff Lynnette Lusenaka NAS Program Officer, Africa NAS Staff George Mitri University of Balamand, Lebanon Associate Professor, PI PI/Co-PI Sediqa Hassani Ibn-e-Sina University, Afghanistan Instructor, PI PI/Co-PI Gerrie Tuitert Netherlands Organisation for Scientific Research NWO- Science for Global Development WOTRO Senior policy officer Comparator Yaso Kunaratnam UK Collaborative on Development Sciences (UKCDS) Research & Policy Officer Comparator Anjali Kuma USAID Global Development Lab PEER Regional Advisor for Middle East and Latin America (AAAS Fellow) USAID PEER Staff Cameron Bess USAID Global Development Lab PEER Senior Regional Advisor for Southeast Asia USAID PEER Staff Clare Muhoro USAID Global Development Lab Private Sector Partner Advisor USAID PEER Staff Jason Porter USAID Global Development Lab PEER Regional Advisor for Africa and Central Asia (AAAS Fellow) USAID PEER Staff Jason Landrum USAID/El Salvador Regional Mission Science and Technology Advisor USAID Mission Staff Marina Vardanyan USAID/Armenia Energy and Water Advisor USAID Mission Staff Hubert Sylney USAID/Haiti PEER POC, Program Office USAID Mission Staff Patrick Meyer USAID/Bangladesh; formerly USAID/Armenia Climate Adaptation Group Leader, EG Office USAID Mission Staff Maina Matir-Torres USAID/Peru PSC, Environment Office USAID Mission Staff Alexandre Mancuso USAID/Brazil Program Officer/Environment Program Manager USAID Mission Staff Christine Lee NASA Scientific Applications Engineer (JPL)/ Associate Program Manager USG Agency Partner Marc Parascandola NIH/NCI Epidemiologist and Program Director USG Agency Partner Lara Campbell NSF Program Officer USG Agency 13 2 Name Organization Position/Title PEER Stakeholder Category Dalal Najib NAS Senior Program Officer NAS Staff Daniel Placht NAS Program Associate NAS Staff Kelly Robbins NAS Program Director NAS Staff Lynnette Lusenaka NAS Program Officer, Africa NAS Staff George Mitri University of Balamand, Lebanon Associate Professor, PI PI/Co-PI Sediqa Hassani Ibn-e-Sina University, Afghanistan Instructor, PI PI/Co-PI Partner Scott Miller Smithsonian Institute Deputy Undersecretary US Partner Scientist; USG Agency Partner Harry Jenter USGS Deputy Chief, Office of Surface Water USG Agency Partner Adam Kurniawan Balang NGO, Indonesia General Director of Balang NGO Local Collaborator Dr. Tom Clements Wildlife Conservation Society NGO, Indonesia Regional Director South East Easia WCS Local Collaborator Farida Handayani Institute for Vector and Reservoir Control Research and Development, National Institute of Health Research and Development, Ministry of Health Rep. of Indonesia PI PI/Co-PI Heri Kuswanto Institute Technology of Surabaya (ITS), Indonesia Director of Graduate Statistics Program, PI PI/Co-PI Faisal Hossain University of Washington Associate Professor, PI PI/Co-PI Jeff Dozier UC Santa Barbara Distinguished Professor US Partner Scientist Rob Reilinger MIT Principal Research Scientist US Partner Scientist Dr. Sang Putu Kaler Surata Mahasaraswati University, Indonesia PI Cycle 1 PI/Co-PI Dr. Brandon Sitzmann USAID/RDMA Regional S&T Advisor/POC for PEER USAID Mission Laura Kiige UNICEF, Kenya Child and Maternal Nutrition Coordinator Local Collaborator Chris Okange Ministry of Education, Science and Technology Directorate of Science and Standards Local Government Dr. Mary Baaru Kenyatta University Lecturer, PI PI/Co-PI Dr. Jane Ambuko University of Nairobi, Kabete Co-PI PI/Co-PI 13 3 Name Organization Position/Title PEER Stakeholder Category Dalal Najib NAS Senior Program Officer NAS Staff Daniel Placht NAS Program Associate NAS Staff Kelly Robbins NAS Program Director NAS Staff Lynnette Lusenaka NAS Program Officer, Africa NAS Staff George Mitri University of Balamand, Lebanon Associate Professor, PI PI/Co-PI Sediqa Hassani Ibn-e-Sina University, Afghanistan Instructor, PI PI/Co-PI Vincent Obanda Kenya Wildlife Service Scientist, Co-PI PI/Co-PI Moses Otiende Kenya Wildlife Service Lab Director, PI PI/Co-PI Nicolas Marita GeoPower Africa Professor of Geophysics, PI PI/Co-PI Dr. Joyce Maina University of Nairobi, Department of Animal Production Senior Researcher, PI PI/Co-PI Owino Joseph South Eastern Kenya University PI PI/Co-PI Shem Wandiga University of Nairobi Chair of the Chemistry Dept, PI PI/Co-PI Dr. Vincent O. Onywera Kenyatta University Deputy Vice Chancellor University Administration Bruce Walker Georgia Tech University, Sonification Lab Professor US Partner Scientist GROUP KEY INFORMANT INTERVIEWS Coun t Organization/ Group Name # Attende d Participant Name Organization Position Category 1 FTF Innovations Lab 2 Clara Cohen USAID/BFS Science Policy Advisor Comparator 2 Jerry Glover USAID/BFS Advisor 3 ICARE, Armenia 2 Vardan Urutyan ICARE Research Director PI/Co-PI 4 Lusine Tadevosyan ICARE Project Coordinator 5 USAID PEER Leadership 2 Callie Raulfs￾Wang USAID Global Development Lab PEER Program Manager USAID PEER Staff 6 Annica Wayman USAID Global Development Lab Research Team Manager 7 NIFA Staff 2 Mike McGirr USDA/NIFA National Program Leader USG Partner Agency 13 4 Coun t Organization/ Group Name # Attende d Participant Name Organization Position Category 8 Patty Fulton USDA/NIFA National Program Leader 9 Planet NI Staff 2 Eloisa Acha Planet NI (National Instruments) Program Director PEER Partner Agency 10 Jimmy Hwang Planet NI (National Instruments) Marketing Manager (former PEER POC) 11 Center for International Forestry Research (CIFOR) 2 Dr. Ani Adiwinata CIFOR PI PI/Co-PI 12 Linda Yuliani CIFOR Research Team 13 Developing a bioeconomy in Indonesia: Identification of novel microorganisms and microbial enzymes from Indonesian peatland and buffaloes to improve bioconversion of oil palm residues 3 Dr. Amadeus Pribowo Indonesia International Institute for Life Sciences (i3L) PI PI/Co-PI 14 Dr. Irnayuli Sitepu Indonesia International Institute for Life Sciences (i3L) Co-PI 15 Florencia F. Dewanto Indonesia International Institute for Life Sciences (i3L) Research Team (Undergrad student) 16 Kangaroo Mother Care, University of Indonesia 7 Prof. Dr. Hadi Pratomo Faculty of Public Health, University of Indonesia PI PI/Co-PI 17 Dr. Asri Adisasmita Faculty of Public Health, University of Indonesia Co-PI 18 Yulia Izati Faculty of Public Health, University of Indonesia Research Team 19 Septyana Choirunisa Faculty of Public Health, University of Indonesia Research Team 20 Tiara Amelia Faculty of Public Health, University of Indonesia Research Team 21 Dwi Muliahani Rizky Faculty of Public Health, University of Admin 13 5 Coun t Organization/ Group Name # Attende d Participant Name Organization Position Category Indonesia 22 Yayuk Sri Rahayu West Java District Health Officer Health Agency of Karawang District Local Government 23 Baby Friendly Hospital Initiative 6 Fransisca Handy Agung Centre of Health Research, University of Indonesia Podiatrist, PI PI/Co-PI 24 Fitra Yelda Faculty of Public Health, University of Indonesia Research Team 25 Luluk Ishardini Faculty of Public Health, University of Indonesia Research Team 26 Dr. Sabarinah Faculty of Public Health, University of Indonesia Vice Dean University Admin 27 Hendri Hartati Faculty of Public Health, University of Indonesia Admin 28 Rita Damayanti Faculty of Public Health, University of Indonesia Dean 29 Effect of Early Pollution on Maternal Health 2 Nikmah Salamia Idris Cipto Mangunkusumo National General Hospital, University of Indonesia PI PI/Co-PI 30 Nina Dwi Cipto Mangunkusumo National General Hospital, University of Indonesia Co-PI 31 STEM Skills and Problem-Based Learning (PBL) 3 Kamarza Mulia, Associate Prof. Chemical Engineering, University of Indonesia PI PI/Co-PI 32 Dr. Elsa Krisanti Mulia Chemical Engineering UI Co-PI 33 Dr. Tania Chemical Engineering UI Research Team 34 Citizen Science 5 Dr. I Made Gunadarma PI PI/Co-PI 13 6 Coun t Organization/ Group Name # Attende d Participant Name Organization Position Category Solutions for National Biodiversity Data Needs: Developing a Plant Checklist for West Kalimantan, Indonesia Wiryana University 35 Andreas Hadiyono Gunadarma University PhD student, Programmer 36 Kartika Dwintaputri Siregar Gunadarma University Research Team 37 Sutresna Wati Gunadarma University Finance 38 Astie Darmayanti Gunadarma University Admin 39 Sustainable Conversion of Oil Palm Lignocellulosic Waste into Pentanol using Metabolically Engineered Microbes 6 Dr. Fransiskus Xaverius Ivan Surya University, Indonesia PI PI/Co-PI 40 Dr. Yalun Arifin Surya University, Indonesia Co-PI 41 Valentine Kheng Surya University, Indonesia Research Team 42 Isadonna Fortune Tengganu Surya University, Indonesia Research Team 43 Indra Memdi Khoris Surya University, Indonesia Research Team 44 Debby Laurentina Surya University, Indonesia Research Team 45 Randomization￾based Curriculum for Introductory Statistics 2 Dr. Kie Van Ivanky Saputra Pelita Harapan University, Indonesia PI PI/Co-PI 46 Rosabella Pelita Harapan University, Indonesia Research Team 47 Pharmacheck 4 Dr. Iwan Ariawan Faculty of Public Health, Center of Health, University of Indonesia PI PI/Co-PI 48 Nugroho S Faculty of Public Health, Center of Health, University of Indonesia Research Team 49 Fitra Yelda Faculty of Public Health, Center of Health, University of Indonesia Research Team 50 Yudarini Faculty of Public Health, Center of Health, University of Indonesia Research Team 51 USAID/Indonesi a 2 Clara Davis USAID/Indonesia AAAS Fellow, Science and USAID Mission 13 7 Coun t Organization/ Group Name # Attende d Participant Name Organization Position Category Technology Advisor and PEER POC 52 Peter Cronin USAID/Indonesia Education Office Director 53 BFCI, Kenya 3 (PI Judith Kimiywe also present) Miriam Chesire County Government Subcounty Nutrition Officer Local Government 54 William NGotie County Government Subcounty Community Strategy Focal Person 55 Wildlife Trafficking - ICIPE 3 Dr. Lillian Wambua ICIPE PI PI/Co-PI 56 Dr. Jandouwe Villinger ICIPE Co-PI 57 Mahmood G. National Museum of Kenya Co-PI 58 Kenyan Forestry Research Institute KEFRI 2 Stephen Kiama KEFRI Research Scientist, PI PI/Co-PI 59 Richard Onwonga KEFRI Co-PI 60 Witchweed/ Striga (KU) 3 Steve Runo Kenyatta University Senior Lecturer, PI PI/Co-PI 61 Hilda Kithinji Kenyatta University Student 62 Dorothy Kenyatta University Student 63 Baby Friendly Community Initiative (BFCI) 5 (3 students present) Dr. Judith Kimiywe Kenyatta University PI PI/Co-PI 64 Dr. Elizabeth Murage African Population and Health Research Centre (APHRC) Co-PI PI/Co-PI; Collaborateo r 65 Mwangaza Assistive Technology for the Blind 3 Dr. Miheso￾O’Connor Kenyatta University Mathematics Specialist, PI PI/Co-PI 66 Nick Twoli Kenyatta University Science Advisor, Lead Researcher 67 Bernard Mugo Kenyatta University Lead Researcher 68 Harnessing Genomics of Edible African Solanaceae Plants 3 Dr. Willis Owino Jomo Kenyatta University of Agriculture and Technology (JKUAT) PI PI/Co-PI 69 Elias Mibai Jomo Kenyatta University of Student 13 8 Coun t Organization/ Group Name # Attende d Participant Name Organization Position Category Agriculture and Technology (JKUAT 70 Grace Mungai Jomo Kenyatta University of Agriculture and Technology (JKUAT Student 71 USAID/Kenya Education Office 2 Robert (Wick) Powers USAID/Kenya Office Director, Education and Youth USAID Mission 72 Dr. Lucy Kithome USAID/Kenya Project Management Specialist, Education Technology 73 USAID/Kenya 3 Mikaele Lauridsen USAID/Kenya Environment Office Environment Officer USAID Mission 74 Alex Albertine USAID/Kenya Strategic Planning and Analysis Office Mission Economist, Team Lead 75 Kyra Zogbekor USAID/Kenya Strategic Planning and Analysis Office evaluation team lead for Program Office LIST OF FOCUS GROUPS Focus Group Name # Participants Host Organization Category PEER Supported Students - Kenya 21 Kenyatta University Student PEER Supported Female PIs - Kenya 8 Kenyatta University PI/Co-PI PEER Supported Students - Indonesia 8 University of Indonesia, Depok Student PEER Supported Female PIs - Indonesia 6 University of Indonesia, Depok PI/Co-PI MOST SIGNIFICANT CHANGE WORKSHOPS Country # Participants Host Organization Category(ies) Kenya 18 Kenyatta University PI/Co-PI/Researchers Indonesia 12 University of Indonesia, Depok PI/Co-PI/Researchers 13 9 DISCLOSURE OF ANY CONFLICTS OF INTEREST [The Evaluation Policy requires that evaluation reports include a signed statement by each evaluation team member regarding any conflicts of interest. A suggested format is provided below.] Name Title Organization Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations 14 0 being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date Findings, Conclusions and Recommendations (FCR) Tracking Matrix 14 3 ANNEX V: POLICY CHANGE LITERATURE REVIEW Research on policy change has been extensive, particularly in the of field of policy sciences, education and in on US government programs. Early policy literature focused on theories explaining top-down and bottom-up approaches. Recent newer theories (Multiple Streams, Advocacy Coalition Theory, Networks) view policy change in a non-linear process and more reflective of dynamic systems considering the mental models and preferences of individuals. Theories are taking into account ‘human’ behavior like incentives, group think, and collective action to provide deeper explanations around why policies are created, changed, and implemented. Policy change in the field of public policy often is defined as change in the existing policy or addition of new innovative policies. Policy change theories/frameworks suggest change occurs through coalitions (Sabatier and Jenkins-Smith), elite power, institutions (Ostrom), multiple streams (Kingdon), leaps/shocks (Baumgartner and Jones) or incrementally (Cerna, 2013). Some of the newest theories from behavioral economists are around “nudging effects” (Thaler and Sunstain, 2008) whereby change occurs through indirect suggestions for non-forced compliance. These frameworks often are more useful for some policy arenas than others. For example, when looking at budget policies in the US much of the change has been almost flat – meaning little to no change has occurred for very long periods of time. There are also short periods of time that had very large and extensive policy revisions/changes. The ‘Punctuated Equilibrium Model’ offered by Baumgartner and Jones would be a good fit to explain how this pattern might be occurring. Where as policy changes in local governments around water use/restrictions, policies for this area might have a better fit with Sabatier and Jenkins-Smith’s Advocacy Coalition Framework which is focused on core beliefs of groups and the impact of external factors on their views. Theories help to provide explanations about how a policy might be changed or created or in some cases eliminated. Policy continuation is more likely than change as this often requires a lot of effort to change the “status-quo”, those in power or benefiting from the current system are not likely to promote changes, and it is very costly to change institutionalized norms. Across the policy change literature, several key factors are relevant to the PEER Evaluation. First, policy change has often been studied and observed over long time periods (10-40 years) especially in complex systems with multiple layers of government. This implies that to observe policy change in a period of 3 years (an average PEER cycle) these projects would likely need to be those already directly connected to the policy process within the system. Those projects not tied to a policy process are less likely to show a direct impact on policy change. That being said, as mentioned above, not all policy change can be explained linearly, and in some instances, such as when external shocks occur, windows of opportunity open and these projects/innovations/solutions could be accepted becoming a significant piece to policy change. - Identifying the connection of the PEER work/research to a specific policy process and its ‘weight’ in that process (direct effect, marginal impact, indirect effect, other), will help MARK/target those projects to track for policy change (this will improve the efficiency of management efforts) - Prior to PEER project implementation or at early stages (year 1), an assessment of the policy arena should be conducted to determine when and where the PEER product might be used (this may not even be the product itself but the increased capacity of the scientists, or recognition within the government of the value of the research, other) 14 4 - PEER projects not directly connected to a policy process can still affect a policy outcome but are less likely to do so in a 3 year period; PEER management and PEER PIs should work together to track “windows” of opportunities when decision-makers may be open to new ideas/solutions Second, some theories can have more explanatory power of policy change in certain fields when compared to others. This may be due to how much the theory has been applied to diverse contexts, or to the type of parameters in the framework. Recently is the literature producing work where policy theories are being applied/tested in developing country contexts (Sabatier and Weible, 2014). Much of the policy theory work in the past has been done in the US and Europe, where government structures and institutions have been stable, with long histories, and with detailed information about the different players in the policy arena. Third, if the goal of a policy change is to change a behavior/condition/outcome/impact, formal policy through governmental processes may not be required. The impact/change desired could be achieved through non-required behavioral incentives, redefinition of the problem/issue, and changes in individual mental models. In other words, efforts to change policy (for certain types of arenas), which is often difficult expensive and slow, may be directed on changing core believe, incentivizing populations, and creating environments conducive to supporting the desired outcome. - If PEER wishes to change for example rules around energy use, the projects may wish to transfer efforts to change government rules/polices towards activities to incentivize the population to behave differently (in those countries and for those issues were applicable) - A policy change may not lead to change in the desired impact/outcome of a project and therefore consideration should be taken in determining the various factors in the causal model, and whether policy change will lead to the intended societal benefit Some policy types can take priority over others. The level of conflict, ambiguity (Matland, 1995), and number of beneficiaries are some factors that may affect the how policy-makers prioritize competing issues. Policy change can also occur a the indiviaul, system or network Types of Policy Frameworks and Theories The following is a list of various existing theories of policy process and change. They have been simplified for purposes of this report. For more in-depth and complete descriptions of these see Sabatier and Weible (2014) and Sandfort and Moulton (2014), both of these books are keystone literature in the field of policy change. Systems Theory – David Easton is one of the earlier policy scholars and his theoretical view looked at policy change in government in the US from a systems perspective – as a series of events under a larger political system, organic and changing. His work influenced thinking around policy from a top-down and bottom up approach and was focused on power, political and authority to control linked to implementation success or failure (Sandfort and Moulton, 2014). 14 5 Figure 15. Easton, David (1965). A Systems Analysis of Political Life, New York, S.32. Another linear model is the policy stages model or policy making model. Meier (1991) provided a linear view of how policies might be created or changed. However, the is model is often believed to be limited as most decision-making environments are complex with multiple connections at different levels between differing actors/stakeholders. Figure 16. Policy process model by Meier (1991). Advocacy Coalition Framework (ACF) – Sabatier and Jenkins-Smith Often a leading theory of change for the environmental policy field, the ACF suggests coalitions are made up of individuals with similar beliefs/core beliefs. This model can work well under conditions of high ambiguity, but tends to occur in over longer periods of time (10 years or more) and likely not reflective of short-term changes. ACF intends to incorporate multiple type of actors, include larger socioeconomic environment, and understand competition coalition dynamics. This complex process has been suggested to be one of the more comprehensive theories attempting to cover the entire context under which policy change occurs. The focus of this theory is the ‘policy subsystem’ in which the coalitions compete and use different types of policy tools. 14 6 Figure 17. ACF Model - Sabatier and Weible (2014). Multiple Streams – Kingdon Policy Streams Theory by Kingdon (1984) was influenced by work from Cohen who developed the Garbage Can Model in which solutions, problems, and process come together in a soup from which change occurs. Kingdon was less abstract about how these factors interact suggesting a ‘window of opportunity’ – an event or shock – opens in which solutions and problems joins with the help of a ‘policy entrepreneur’. This person can push through a policy agenda the merged solution and problem. The Policy Streams Theory by Kingdon is reflective of non-linear processes as he suggests solutions exist independent of problems and vice versa. This is usually opposite to some of the other theories based on rational behavior. Institutional Analysis and Development (IAD) – Ostrom This model is based on rational choice theory, and part of the work which won Ostrom the Nobel Prize in Economics. Her IAD model is interested in looking at the conditions that have to be present for groups to solve collective action problems. Institutions play a role in the political system, and in past research to explain changes in policy, they were ignored. Ostrom’s work looks at how “institutions” (defined widely as formal and/or information rules, norms) affect the rules people generate (Cairney and Heikkila, 2014). Figure 18. The IAD model by Ostrom (2005). 14 7 This framework has often been applied to the management of the ‘commons’. Punctuated Equilibrium – Baumgartner and Jones Their theory tries to explain the presence of long periods of policy stability and continuity with then short periods of large changes. Power and agenda setting are present with many ideas competing for attention but not one specific taking hold. However, something happens (an external shock to the system), and a change occurs. Decision makers cannot consider all problems and thus focus on a disproportionate number of issues (Bounded Rationality); these policy makers pay disproportionate attention on a small number of policies. Change requires critical mass to overcome, whereby attention shifts – the ‘equilibrium’ is disrupted – external pressure reaches a tipping point and causes these punctuations “burst of attention” and then so new approaches are considered. Behavioral Economics and ‘Nudges’ – Thaler and Sunstain These scholars, much like Kahneman and Tversky, look at how psychology influences choices, where positive reinforcement might be effective in causing a behavioral change. The incentives are determined and used to create the required behavioral change and outcome. These can often work better than mandates or rules. Definition of a ‘nudge’ by Thaler and Sunstain: “A nudge, as we will use the term, is any aspect of the choice architecture that alters people's behavior in a predictable way without forbidding any options or significantly changing their economic incentives. To count as a mere nudge, the intervention must be easy and cheap to avoid. Nudges are not mandates. Putting fruit at eye level counts as a nudge. Banning junk food does not.” One of their most famous experiments and applications of their theory was in the Amsterdam Airport urinals where they placed houseflies to change behavior for improved “aim” as non-force compliance. Table 1. P-olicy process and change theories and level of research activity for each one (Cairney and Heikkila, 2014- Table 10.2). Acronyms as follows: MSA – Multiple Streams Analysis, PET – Punctuated Equilibrium Theory, SCF – Social Construction Framework, PFT – Policy Feedback Theory, ACF – Advocacy Coalition Framework, NPF – Narrative Policy Framework, IAD – Institutional Analysis and Development Framework, and DOI – Diffusion of Innovation. 14 8 Other excellent online resources: - Dr. Paul Cairney podcasts (he is leading scholar in public policy processes and theories and has a very user-friendly podcast to describe theories in policy change) https://paulcairney.wordpress.com/1000-words/ - Dr. Christopher Weible webpage for leading articles in the ACT and he was Sabatier’s (the creator of ACT) student and scholar now leading the field in policy change and processes http://www.ucdenver.edu/academics/colleges/SPA/FacultyStaff/Faculty/Pages/ChristopherWeible.aspx - Dr. Gordon Kingsley leading expert in policy process in government, STEM and Science and Technology policy, as well as organizational theory http://www.iac.gatech.edu/people/faculty/kingsley References “Effective implementation in practice: Integrating public policy and management” (2014). Jodi Sandfort and Stephanie Moulton. Jossey Bass and Wiley and Company Publications. “Theory of Policy Process” (2014). Sabatier, P. and Weible, C. Third Edition (Chicago: Westview PRESS). “Nudge” (2008). Thaler, R. and Sunstain, C. Penguin Books. “Agendas, Alternatives, and Public Policies” (1984). Kingdon, J. New York: Haper Collins. “A comparison of Theories of the Policy Process” (2014). Paul Cairney and Tanya Heikkila in (eds) Sabatier, P. and Weible, C. (2014) Theories of the Policy Process, Third Edition (Chicago: Westview PRESS). PDF “The nature of policy change and implementation: A review of different theoretical approaches” (2013). Lucie Cerna, OECD Report. 14 9 "Synthesizing the Implementation Literature: The Ambiguity-Conflict Model of Policy Implementation." Matland, Richard. Journal of Public Administration Research & Theory April 1995 ANNEX VI DESCRIPTIVE STATISTICS Table 1. PEER PI Responses of Research Location by Country and Cycle PEER Countries (Q74) Science Cycle 1 Science PIRE Cycle Science Cycle 2 Health Cycle 1 Health Cycle 2 Science Cycle 3 Cycle 4 Cycle 5 Two Cycles TOTAL Afghanistan 0 0 0 0 0 0 4 0 0 4 Armenia 0 0 1 0 0 1 1 0 0 3 Bangladesh 1 0 1 1 0 0 1 1 0 5 Bhutan 0 0 0 0 0 0 1 0 0 1 Brazil 0 0 2 0 0 2 3 0 0 7 Cambodia 0 0 2 1 1 1 0 0 0 5 Cameroon 0 0 1 0 0 0 1 0 0 2 Colombia 1 0 1 0 0 0 0 0 0 2 Dominican Republic 0 0 1 0 0 0 0 0 0 1 Ecuador 1 0 0 0 0 0 0 0 0 1 Egypt 0 0 1 0 0 0 0 0 0 1 Ethiopia 1 0 0 0 0 1 1 0 0 3 Georgia 1 0 0 0 0 0 0 0 0 1 Ghana 0 0 1 0 0 1 1 0 0 3 India 0 0 3 0 0 1 1 0 0 5 Indonesia 4 0 5 2 5 3 5 0 1 25 Japan 0 0 0 0 0 1 0 0 0 1 Jordan 1 0 1 0 0 1 1 0 0 4 Kenya 1 0 4 1 0 2 9 0 0 17 Kyrgyzstan 0 0 0 0 0 0 1 0 0 1 Laos 0 0 0 0 0 0 0 0 1 1 Lebanon 2 0 0 0 0 1 0 0 2 5 Malawi 1 0 0 0 0 0 0 0 0 1 Maldives 0 0 0 0 0 0 1 0 0 1 PEER Countries (Q74) Science Cycle 1 Science PIRE Cycle Science Cycle 2 Health Cycle 1 Health Cycle 2 Science Cycle 3 Cycle 4 Cycle 5 Two Cycles TOTAL Mali 0 0 0 1 0 0 0 0 0 1 Mexico 0 1 0 0 0 0 0 0 0 1 Mongolia 0 0 1 0 0 0 0 0 0 1 Mozambique 0 0 0 1 0 0 0 0 0 1 Nepal 1 0 0 0 0 0 0 0 0 1 Nicaragua 0 0 1 0 0 0 0 0 0 1 Nigeria 0 0 2 0 0 0 0 0 0 2 Pakistan 0 0 0 0 0 0 1 0 0 1 Palestinian Territories 0 0 1 0 0 0 0 0 0 1 Peru 0 0 1 0 0 0 1 0 1 3 Philippines 2 0 1 0 2 1 0 0 0 6 Senegal 0 0 2 0 0 0 0 0 0 2 South Africa 0 0 3 0 0 1 2 0 0 6 Sri Lanka 1 0 0 0 0 1 0 0 0 2 Tanzania 0 0 1 0 0 1 0 0 0 2 Thailand 1 0 1 0 0 1 0 0 0 3 Tunisia 0 0 1 0 0 0 0 0 0 1 Uganda 0 0 0 0 0 1 0 0 0 1 United Arab Emirates 0 0 1 0 0 0 0 0 0 1 United States of America 0 0 0 0 0 3 2 0 0 5 Uzbekistan 0 0 0 0 0 0 2 0 0 2 Vietnam 2 1 2 0 0 4 1 0 1 11 Total 21 2 42 7 8 28 40 1 6 155 Annex V Table 2 : PEER PI/Co-PI Survey data – satisfaction level with PEER by cycle and career stage. Career Stage PEER Satisfaction Level Cycle 4 Cycle 5 Health Cycle 1 Health Cycle 2 Science Cycle 1 Science Cycle 2 Science Cycle 3 Science PIRE Cycle Projects with Two Cycles 1 year to less than 3 years extremely satisfied 2 1 1 extremely unsatisfied neutral satisfied 1 unsatisfied 3 years to less than 5 years extremely satisfied 1 1 5 1 extremely unsatisfied neutral satisfied 5 1 1 unsatisfied 5 years to less than 10 years extremely satisfied 4 1 1 7 2 1 1 extremely unsatisfied 1 1 neutral satisfied 5 1 1 3 1 1 unsatisfied 10 years or more extremely satisfied 14 1 2 15 19 9 3 extremely unsatisfied 1 3 neutral 1 2 satisfied 13 1 5 5 4 9 9 1 unsatisfied 2 Table 3: PEER PI/Co-PI Survey data – Sector Career Stage Cycle 4 Cycle 5 Health Cycle 1 Health Cycle 2 Science Cycle 1 Science Cycle 2 Science Cycle 3 Science PIRE Cycle Projects with Two Cycles Sector Career Stage Cycle 4 Cycle 5 Health Cycle 1 Health Cycle 2 Science Cycle 1 Science Cycle 2 Science Cycle 3 Science PIRE Cycle Projects with Two Cycles Agriculture 1 year to less than 3 years 10 years or more 2 5 1 3 years to less than 5 years 5 years to less than 10 years 3 Atmospheric Sciences/Disaster Management 1 year to less than 3 years 10 years or more 2 2 2 1 3 years to less than 5 years 5 years to less than 10 years 1 1 Biodiversity 1 year to less than 3 years 1 10 years or more 3 1 1 2 6 4 1 3 years to less than 5 years 3 5 years to less than 10 years 1 2 1 Biology 1 year to less than 3 years 10 years or more 1 1 3 years to less than 5 years 5 years to less than 10 years Chemistry 1 year to less than 3 years 10 years or more 3 years to less than 5 years 1 5 years to less than 10 years Climate 1 year to less than 3 years 1 1 10 years or more 3 2 2 3 years to less than 5 years 5 years to less than 10 years 2 Conservation 1 year to less than 3 years 10 years or more 2 2 1 3 years to less than 5 years 1 5 years to less than 10 years Sector Career Stage Cycle 4 Cycle 5 Health Cycle 1 Health Cycle 2 Science Cycle 1 Science Cycle 2 Science Cycle 3 Science PIRE Cycle Projects with Two Cycles Education 1 year to less than 3 years 10 years or more 1 1 1 3 years to less than 5 years 5 years to less than 10 years 1 Energy 1 year to less than 3 years 10 years or more 1 3 3 years to less than 5 years 1 5 years to less than 10 years Engineering 1 year to less than 3 years 1 10 years or more 1 2 1 3 years to less than 5 years 5 years to less than 10 years 1 1 1 Environment 1 year to less than 3 years 10 years or more 3 2 1 1 3 years to less than 5 years 3 1 5 years to less than 10 years 2 2 Food Security 1 year to less than 3 years 10 years or more 5 3 years to less than 5 years 5 years to less than 10 years 1 Health 1 year to less than 3 years 10 years or more 2 5 5 1 3 years to less than 5 years 1 5 years to less than 10 years 1 1 Marine Biology/Oceanography 1 year to less than 3 years 10 years or more 3 1 1 3 years to less than 5 years 5 years to less than 10 years Sector Career Stage Cycle 4 Cycle 5 Health Cycle 1 Health Cycle 2 Science Cycle 1 Science Cycle 2 Science Cycle 3 Science PIRE Cycle Projects with Two Cycles Molecular Biology 1 year to less than 3 years 10 years or more 1 2 3 years to less than 5 years 5 years to less than 10 years 1 Natural Resources 1 year to less than 3 years 10 years or more 3 3 1 2 3 years to less than 5 years 1 5 years to less than 10 years 1 Soil Sciences 1 year to less than 3 years 10 years or more 1 3 years to less than 5 years 5 years to less than 10 years Water 1 year to less than 3 years 10 years or more 6 2 2 2 2 3 years to less than 5 years 1 1 5 years to less than 10 years 3 1 2 15 6 Table 4. CI for key variables requested related to stakeholder perception _Q33_recog~d 175 2.445714 .1077055 2.233137 2.658292 Variable Obs Mean Std. Err. [95% Conf. Interval] _Q31_recog~n 174 2.454023 .1055141 2.245762 2.662284 Variable Obs Mean Std. Err. [95% Conf. Interval] _Q32_mores~s 172 2.5 .0905204 2.321319 2.678681 Variable Obs Mean Std. Err. [95% Conf. Interval] _Q30_advan~r 175 3.005714 .1073634 2.793812 3.217617 Variable Obs Mean Std. Err. [95% Conf. Interval] _Q18_overa~E 177 3.299435 .1250488 3.052647 3.546223 Variable Obs Mean Std. Err. [95% Conf. Interval] _Q26_PEERs~n 174 2.293103 .1111742 2.073671 2.512536 Variable Obs Mean Std. Err. [95% Conf. Interval] _Q46_Colla~n 170 1.958824 .0152845 1.92865 1.988997 Variable Obs Mean Std. Err. [95% Conf. Interval] _Q44_Years~r 171 3.152047 .1109807 2.932969 3.371124 Variable Obs Mean Std. Err. [95% Conf. Interval] _Q43_wedid~r 52 1 0 1 1 Variable Obs Mean Std. Err. [95% Conf. Interval] _Q29_lever~g 174 2.614943 .103939 2.409791 2.820094 Variable Obs Mean Std. Err. [95% Conf. Interval] _Q28_publi~s 170 4.058824 .1575804 3.747744 4.369903 Variable Obs Mean Std. Err. [95% Conf. Interval] 15 7 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523