THIS PUBLICATION WAS PRODUCED AT THE REQUEST OF THE UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT. IT WAS PREPARED INDEPENDENTLY BY MATHEMATICA, AND THE AUTHORS INCLUDE AUDREY-MARIE MOORE, EDITH FELIX, AND JOSH MEUTH ALLDREDGE. COMMUNITY MEMBERS USE THE EZYAGRIC APP (RESILIENTAFRICA NETWORK) RESEARCH, EVIDENCE, AND THE GLOBAL INNOVATION ECOSYSTEM A PERFORMANCE EVALUATION OF THE USE AND UTILITY OF THE HIGHER EDUCATION SOLUTIONS NETWORK TO SOLVE DEVELOPMENT CHALLENGES USAID.GOV PERFORMANCE EVALUATION OF HESN | ii ACKNOWLEDGEMENTS We greatly appreciate the contributions of the many people who have helped make this evaluation report possible. We especially thank the U.S. Agency for International Development (USAID)’s Center for Development Research team, including, Ticora Jones, Karen Fowle, Maggie Linak, Brian Bingham, and Michelle L'Archeveque Jones, who together provided us guidance and support throughout. We also thank Danice Brown Guzmán from the University of Notre Dame’s Pulte Institute for Global Development for her review and oversight throughout. This report would not have been possible without the contributions from staff and leadership of the eight Higher Education Solutions Network (HESN) Development Labs, including: the AidData Center for Development Policy (Ariel BenYishay, Jessica Wells, Kathy Nolan), the Comprehensive Initiative on Technology Evaluation (CITE) (Bish Sanyal, Kendra Leith, Dan Frey), the International Development Innovation Network (IDIN) (Amy Smith, Laura Budzyna) the ResilientAfrica Network (RAN) (William Bazeyo, Roy William Mayega, Julius Ssentongo, Nathan Tumuhamye, Ronald Kayiwa, Dorothy Okello, Dennis Chirawura, Harriet Adong, Natasha Kassami, Deborah Naatujuna), the Center for Conflict and Development (ConDev) (Edwin Price), the Development Impact Lab (DIL) (Carson Christiano, Samuel Fishman), the Global Center for Food Systems Innovation (GCFSI) (Eric Crawford), and the Social Entrepreneurship Accelerator at Duke (SEAD) (Kimberly Bardy Langsam). We would like to thank them for sharing materials and other information, responding to emails, as well as spending time on the phone helping us to understand their activities. Additionally, we are extremely grateful to all the other stakeholders who participated in surveys and interviews for the evaluation including current and former staff at various USAID missions and operating units, local government ministry representatives, partner organizations, researchers, and innovators. We would also like to thank several other colleagues who have made important contributions to this report. Randall Blair provided thoughtful comments on the draft report. Cindy George coordinated the editing of the report. Sheena Flowers and Sherly Friedlander enhanced the content by putting it in an attractive and intuitive format. USAID.GOV PERFORMANCE EVALUATION OF HESN | iii ABSTRACT At all levels, high quality education provides a foundation for economic growth, improved health, and peaceful, resilient societies. It also supports countries on their journey to self-reliance (USAID 2018), as higher education institutions (HEIs) play a critical role by spurring innovations to solve global development problems in low- and middle-income countries (LMICs). USAID launched the Higher Education Solutions Network (HESN) program in 2012 to strengthen the role of HEIs in development. Since its founding, HESN has invested $115 million in eight HESN Development Labs in universities throughout the United States and in Africa. These funds build lasting partnerships between HEIs and diverse stakeholders in LMICs, which allows them to finance workshops and learning events, produce high-quality data sets and new methodologies, and carry out a broad array of research. USAID commissioned this evaluation to help stakeholders understand (1) the conditions and models that generate effective partnerships with USAID Missions, Bureaus, and Independent Offices (MBIOs); (2) how HESN worked to improve the use of research findings; and (3) their potential LMIC policy impacts. The mixed-methods performance evaluation uses survey, bibliometric, administrative, interview, and focus group data to examine the use and utility of the core- and buy-in-funded activities to USAID and its development partners. The evaluation shows that the eight HESN Development Labs played a vital role in achieving HESN’s goals in collaboration with USAID, policymakers, and other partners. HESN-supported researchers and development practitioners created more than 900 products and technologies that address poverty-related challenges in communities worldwide. The HESN Development Labs developed 291 data sets or data￾related technologies, engaged more than 1,200 partners in 83 countries, and directly affected 7.1 million beneficiaries across 35 countries. The evaluation results show that HESN activities funded through core investments from USAID tended to generate products that were more useful to a public audience (for example, data sets shared and reused by other researchers). In contrast, activities funded directly by MBIOs (“buy-in activities”) were most useful in meeting the strategic goals of specific countries. Products supported using a hybrid of core and buy-in funding often led to follow-on work for the HESN Development Labs. Because they served a broader audience, these products also tended to be more cost-efficient. Recommendations from the evaluation suggest that USAID (1) link U.S. government-funded research and innovation activities to USAID country development strategies, (2) maintain flexible scopes of work that allow for cross-sectoral research, (3) continue investing in technology and evidence-based research and innovation, and (4) continue to use a combination of core- and buy-in-funded activities to meet the Agency’s research objectives. The cost-efficiency results also suggest that investments in universities located in LMICs may obtain the best value for money for research and innovation projects. USAID.GOV PERFORMANCE EVALUATION OF HESN | iv TABLE OF CONTENTS ABBREVIATIONS AND ACRONYMS......................................................................................................... viii GLOSSARY ..................................................................................................................................................ix EXECUTIVE SUMMARY .............................................................................................................................. x I. INTRODUCTION................................................................................................................................... 1 II. PURPOSE OF THE EVALUATION....................................................................................................... 2 III. HESN BACKGROUND.......................................................................................................................... 4 A. Overview of HESN......................................................................................................................... 4 1. Description of the HESN program ......................................................................................... 4 2. Implementing organizations (HESN Development Labs)...................................................... 5 3. Target population................................................................................................................... 7 IV. EVALUATION DESIGN AND METHODOLOGY .................................................................................. 9 A. Evaluation methodology ................................................................................................................ 9 1. Implementation study........................................................................................................... 10 2. Analysis of outcomes........................................................................................................... 10 3. Cost analysis ....................................................................................................................... 11 B. Data collection process ............................................................................................................... 11 C. Limitations ................................................................................................................................... 12 V. FINDINGS ........................................................................................................................................... 14 A. Implementation Summary............................................................................................................ 15 B. Implementation facilitators and barriers ...................................................................................... 21 Structural and institutional elements (Research Question 3)...................................................... 22 C. Results and outcomes ................................................................................................................. 26 The utility of HESN for USAID staff (Research Question 1)........................................................ 27 The utility of HESN for policymakers and implementers (Research Question 2)........................ 33 HESN influence on HEIs (Research Question 5)........................................................................ 40 D. HESN Influence on Stakeholder Careers.................................................................................... 44 E. Cost analysis of HESN ................................................................................................................ 48 1. Cost data and assumptions ................................................................................................. 48 2. Cost findings ........................................................................................................................ 50 F. HESN sustainability ..................................................................................................................... 52 VI. CONCLUSIONS.................................................................................................................................. 55 USAID.GOV PERFORMANCE EVALUATION OF HESN | v VII. RECOMMENDATIONS...................................................................................................................... 61 ANNEX A: SOURCES OF INFORMATION ...............................................................................................A.1 ANNEX B: DATA COLLECTION METHODS.............................................................................................B.1 ANNEX C: CASE STUDIES.......................................................................................................................C.1 ANNEX D: EVALUATION DESIGN REPORT ...........................................................................................D.1 ANNEX E: SURVEY INSTRUMENTS AND INTERVIEW PROTOCOLS..................................................E.1 ANNEX F: EVALUATION TEAM................................................................................................................F.1 ANNEX G: LASER BRIEF......................................................................................................................... G.1 USAID.GOV PERFORMANCE EVALUATION OF HESN | vi Tables 1. HESN Development Labs ............................................................................................................... xi II.1. HESN International Development Labs ........................................................................................... 2 IV.1. Evaluation questions and components ............................................................................................ 9 V.1. Differences between core- and buy-in-funded activities ................................................................15 V.2. Activities that survey respondents identified as best representing HESN goals ...........................21 V.3. Average citation levels by pre-2015 researcher output group .......................................................45 V.4. Cost per unique beneficiary of HESN innovations, by HESN Development Lab ..........................50 V.5. Cost per event and participants .....................................................................................................51 V.6. Cost per output...............................................................................................................................51 Figures 1. Main objectives of HESN ................................................................................................................ xi 2. HESN at a glance...........................................................................................................................xiv III.1. HESN Objectives ............................................................................................................................. 4 III.2. HESN results chain.......................................................................................................................... 5 III.3. Location of HESN Development Labs ............................................................................................. 6 III.4. HESN stakeholder summary............................................................................................................ 7 IV.1. Data collection process ..................................................................................................................11 V.1. Summary of the overall evaluation findings ...................................................................................14 V.2. HESN implementation and evaluation timeline..............................................................................15 V.3. HESN Development Lab outputs produced by year, by HESN Development Lab........................16 V.4. Number of HESN outputs by HESN Development Lab and type ..................................................18 V.5. Number of activities in which surveyed HESN stakeholders participated .....................................19 V.6. Word frequency of interviewees, by stakeholder group................................................................. 20 V.7. Summary of implementation facilitators and barriers.....................................................................21 V.8. Summary of results and outcomes ................................................................................................27 V.9. Perceptions and use of HESN outputs by USAID MBIOs .............................................................28 V.10. CITE (with partner George Washington University) Feed the Future Market System Monitoring activity with USAID/Uganda .........................................................................................29 V.11. AidData's geocoded aid management platforms ...........................................................................30 V.12. DIL cash-benchmarking study in Rwanda .....................................................................................32 V.13. Perceptions and use of HESN outputs by USAID MBIOs for policymaking ..................................33 V.14. HESN Development Lab average results on USAID program and policy change scorecard .......................................................................................................................................35 USAID.GOV PERFORMANCE EVALUATION OF HESN | vii V.15. HESN unique partners and partner touches ..................................................................................36 V.16. Instances of partner engagement, by country................................................................................36 V.17. C-Innova in Colombia.....................................................................................................................39 V.18. RootiO radio innovation supported by RAN...................................................................................40 V.19. HESN impacts on HEIs: Percentage of respondents who agree or strongly agree with key statements ...............................................................................................................................41 V.20. HESN at a glance...........................................................................................................................43 V.21. Average citations early in HESN and after HESN support concluded (researchers grouped by PhD year)....................................................................................................................46 V.22. Annual citations of sampled scholars during and after the HESN period, by career stage ...........47 V.23. HESN Development Lab institutionalization ..................................................................................52 V.24. AidData sustainability case study ..................................................................................................53 V.25. The number of new data sets, data-related technologies, classes, or disciplines created by HESN-funded Development Labs .............................................................................................54 USAID.GOV PERFORMANCE EVALUATION OF HESN | viii ABBREVIATIONS AND ACRONYMS AOR agreement officer’s representative CITE Comprehensive Initiative on Technology Evaluation CDD Center for Digital Development CDR USAID Center for Development Research ConDev Center on Conflict and Development DIL Development Impact Lab DIV Development Innovation Ventures DRC Democratic Republic of the Congo GCFSI Global Center for Food Systems Innovation GIS geographic information system HEI higher education institution HESN Higher Education Solutions Network IDDS International Development Design Summits IDIN International Development Innovation Network LMIC low- and middle-income country MBIO USAID Missions, Bureaus, and Independent Offices MIT Massachusetts Institute of Technology OECD Organization for Economic Cooperation and Development OU operating unit PPL Bureau for Policy, Planning, and Learning RILabs Resilience Innovation Labs SEAD Social Entrepreneurship Accelerator at Duke SOGE Scaling Off-Grid Energy UDS University of Development Studies UN United Nations USAID United States Agency for International Development USD United States dollars USG United States government UTEC University of Engineering and Technology USAID.GOV PERFORMANCE EVALUATION OF HESN | ix GLOSSARY Bibliometric analysis. The use of statistical methods to analyze books, articles, and other publications. These methods are traditionally used in the field of library and information science, but researchers are increasingly using the methods to understand the scholarship contributions of research-focused programs. Cooperative agreement. A legal instrument used when the principal purpose is to transfer anything of value to a recipient to accomplish a public purpose of support or stimulation authorized by federal statute and when the government anticipates USAID's substantial involvement.1 Cost-efficiency. Such an analysis allows us to compare the costs of an intervention to the program's outputs (such as cost per workshop, cost per report, or cost per trainee). This analysis is useful when we want to understand whether different delivery models can produce a given output more efficiently. This type of analysis also allows us to understand how the context of particular interventions and their characteristics drive the cost output (Walls et al. 2020). Cross-sectoral approach. An approach that considers the combined outcomes and results from different sectors or methodologies when designing and implementing activities. Higher Education Solutions Network (HESN) Development Labs. Research and development groups or centers housed by seven world-class universities working directly to evaluate and strengthen real￾world innovations in development. Missions, Bureaus, and Independent Offices (MBIOs). Field-based missions or independent offices, within broader USAID, that carry out the agency’s development work. Sector-specific approach. An approach that focuses on designing and implementing activities within one sector. 1 www.usaid.gov USAID.GOV PERFORMANCE EVALUATION OF HESN | x EXECUTIVE SUMMARY A. BACKGROUND AND PURPOSE We increasingly see a growing interest in HEI contributions to development across donors. The United Kingdom’s Department for International Development recently implemented a new partnership program for HEIs, and in 2015 USAID designated one of their four education priorities as the “strengthening of higher education and workforce development programs.”2 About 20 percent of World Bank funds support higher education, and 34 percent of the total education funding in the Development Assistance Committee3 of OECD countries went to higher education in 2015 (McCowan 2016). Donor investments in higher education systems in low- and middle-income countries (LMICs) have also increased since 2015— a trend that the USAID-funded HESN program led beginning in 2012.4 McCowan (2016) cites three main reasons that the international community is investing more in higher education and research. First, the changing nature of the global political economy means that people value knowledge to improve economic competitiveness. Academic experience drives technological development, and it provides graduates with higher-level skill sets to adapt and use in the workplace. Second, by working with communities and outside organizations, HEIs can offer professional capacity-building as an input for international development programs. Third, people who complete tertiary education experience higher return rates for the time and money they invest than those who complete less-advanced education levels. USAID’s $115 million investment in core HESN projects has generated more than $200 million in leveraged funds from academic, private sector, governmental, and other stakeholders for development projects and an additional $56 million in equity, debt, and philanthropic support for entrepreneurs affected and supported through the HESN Development Labs. These resources supported more than 900 new and existing innovations, trained more than 2,000 professionals, and helped more than 7 million beneficiaries worldwide. HESN is a set of cooperative agreements between USAID and seven universities designed to channel the ingenuity and expertise of university students, researchers, and faculty toward global development. The HESN program operated through two USAID funding streams: (1) dedicated core funding that supports overarching HESN program goals, and (2) buy-in funding from various USAID MBIOs supports their specific research and innovation activities. USAID provided core funding to the HESN Development Labs to build capacity by pursuing development-focused research, such as geocoding global databases. In contrast, USAID MBIOs support “buy-in” projects of interest to those MBIOs by providing funds to the HESN program and engaging with HESN agreement officers' representatives5 (AORs) to directly negotiate project scopes of work with HESN Development Labs. Implementation of HESN began in 2012, and USAID provided core funding to the partner universities through 2018. MBIOs have been able to “buy in” to HESN since 2015.6 The goal of HESN 1.0 was to create an international, interdisciplinary network of HESN Development Labs to solve a variety of development challenges. USAID believed that investing in these HESN 2 https://link.springer.com/article/10.1007/s10734-016-0035-7 and https://files.eric.ed.gov/fulltext/ED594404.pdf 3 The Development Assistance Committee was established in 1961. Its present members are Australia, Austria, Belgium, Canada, Denmark, Finland, France, Germany, Greece, Ireland, Italy, Japan, Korea, Luxembourg, Netherlands, New Zealand, Norway, Portugal, Spain, Sweden, Switzerland, the United Kingdom, the United States and the European Union Institutions. 4 The period of performance under the core HESN Cooperative Agreements with Labs ran from fall 2012 to fall 2017; however, some labs extended their period of performance by signing agreements with USAID missions and operating units. 5 AORs serve as technical experts and managers for USAID's grants and cooperative agreements, regularly interfacing with partners and checking in on project progress. They function under a delegation of responsibility from USAID agreement officers. 6 A “buy-in” is a process that USAID MBIOs can use to support activities of interest under HESN. The missions and operating units provide supplemental funding (from their own budgets) to the HESN cooperative agreement to support the research or innovation efforts of specific lab(s) that serve the needs of the MBIO. USAID.GOV PERFORMANCE EVALUATION OF HESN | xi Development Labs would improve the quality, access, and use of data supporting evidence-based development. USAID also believed that the investments would accelerate the creation, testing, and scaling of technological innovations and catalyze a global ecosystem of people and institutions to share knowledge and lessons and build the capacity to solve critical development problems. The program also engaged the next generation of development practitioners by training students in international development. Since HESN 1.0 (henceforth simply HESN) ended in 2018, USAID is implementing HESN 2.0 (LASER) and is planning for HESN 3.0. Figure 1 highlights the objectives of HESN 1.0. Figure 1. Main objectives of HESN Source: HESN Midterm Performance Evaluation, 2016. To meet these objectives, the HESN Development Labs sought to produce research and innovation advancements as public goods (using core funding) and satisfy the immediate research and innovation needs of MBIOs. In their pursuit of the latter goal, the HESN Development Labs had incentives to demonstrate the value and potential application of specific research and innovation products, which could be observed and subsequently sought out by other missions and operating units. Table 1 highlights each of the eight HESN Development Labs, their host university, and the approach each entity established for its research. Table 1. HESN Development Labs NAME HOST UNIVERSITY HESN DEVELOPMENT LAB APPROACH AidData Center for Development Policy College of William and Mary Cross-sectoral Center for Conflict and Development (ConDev) Texas A&M University Sector-specific Development Impact Lab (DIL) University of California, Berkeley Cross-sectoral Global Center for Food Systems Innovation (GCFSI) Michigan State University Sector-specific Comprehensive Initiative on Technology Evaluation (CITE) Massachusetts Institute of Technology Cross-sector International Development Innovation Network (IDIN) Massachusetts Institute of Technology Cross-sector ResilientAfrica Network (RAN) Makerere University Cross-sector Social Entrepreneurship Accelerator at Duke (SEAD) Duke University Sector-specific Source: HESN documents, survey, and interview data. USAID.GOV PERFORMANCE EVALUATION OF HESN | xii B. KEY EVALUATION QUESTIONS USAID wants to understand (1) the conditions and models that generate effective partnerships with MBIOs, (2) the way HESN worked to improve the use of research findings, and (3) their potential LMIC policy impacts. To this end, USAID formulated the following questions for this evaluation: 1. To what extent have mission partners applied learnings from HESN 1.0 research or outputs to their programs? a. What was the usefulness of the buy-ins to their programs (how pertinent, relevant, and timely were the outputs)? b. What was the utility of buy-ins to programs (how much did policymakers and stakeholders use the outputs)? c. To what extent did the usefulness and utility of the buy-ins vary by funding source? 2. What was the partnership's perceived utility among stakeholders (USAID mission, the HESN Development Labs, and policymakers)? 3. Which structural or institutional elements of the partnership contributed to different levels of usefulness to mission programming and decision making? Which elements contributed to utility? 4. Which process elements of the partnership contributed to different levels of usefulness to mission programming and decision making? Which elements contributed to utility? 5. How much has HESN contributed to changes in HEIs or HEI networks to increase their engagement in international development? To what extent would any changes be sustained? Why or why not? C. EVALUATION METHODOLOGY Mathematica conducted a mixed-methods process evaluation that explores the different HESN funding modalities' use and utility to international development stakeholders. Such evaluations use quantitative and qualitative data to examine the degree to which organizations implemented an intervention as intended (that is, followed the implementation process). The evaluation included these data collection and analysis steps: document review, administrative data review and analysis (with monitoring and reporting indicators), bibliometric analysis,7 online surveys, quantitative analysis, interviews and focus groups with key stakeholders and subsequent qualitative analysis, and cost-efficiency analysis. We selected four HESN Development Labs to explore more deeply during this evaluation: MIT CITE, MIT IDIN, RAN at Makerere University, and AidData at William and Mary. We chose these labs because they were diverse in their approach to activities, conducted activities in numerous countries, and implemented activities using a mix of the three types of funding modalities (core, buy-in, and hybrid). Limitations of the evaluation included (1) recall bias and nonresponse bias on the surveys and interviews; (2) the onset of COVID-19 (which prevented the team from traveling to Uganda and Ghana for data collection and shifted our methods to virtual interviews; (3) and the lack of detailed cost data to conduct an in-depth cost-efficiency analysis. We believe that the quality of the virtual data collection process was strong based on triangulation across multiple interviews and saturation levels within stakeholder groups. We do not think that the shift to virtual data collection had a significant effect on our findings. Our team reached enough people across all stakeholder groups. We also conducted more interviews virtually than we could have during our time in Uganda and Ghana—12 more than we estimated when planning our field visits. 7 Bibliometrics is the use of statistical methods to analyze books, articles, and other publications. USAID.GOV PERFORMANCE EVALUATION OF HESN | xiii D. EVALUATION FINDINGS Summary of the overall evaluation findings 1. Research Question 1. MBIO application of outputs to programs. The application of HESN learnings by USAID mission partners varied according to mission and activity. HESN Development Lab activities developed in collaboration with those of missions and funded as either buy-ins or in collaboration with core funding were more immediately useful than purely core-funded work, because they typically supported the mission’s in-country objectives and strategies. 2. Research Question 2. Perceived utility of the partnership by stakeholders. The partnership with universities allowed USAID to strengthen its ability to undertake and use research. HESN also helped support local innovation ecosystems and helped the HESN Development Labs build their institutional capacity to work with USAID. 3. Research Question 3. Structural and institutional elements that contribute to the different levels of usefulness in programming and decision making. Bureaucratic funding and approval structures (both within HEIs hosting HESN Development Labs and with external partners) delayed research efforts and occasionally hampered the timeliness of research products. When the HESN Development Labs relied on close support from their AOR’s flexibility from USAID and streamlined grant and funding systems in their host HEIs, they were better able to navigate or avoid bureaucratic barriers to the production of timely research. 4. Research Question 4. Process elements that contribute to different levels of usefulness in programming and decision-making. Differences in planning processes and internal timelines between MBIOs and HESN-funded activities led to challenges in commissioning and completing work. Distinct communication styles also made it hard for some HESN Development Labs to convey findings to USAID in practical ways for immediate use. Finally, some HESN Development Labs encountered difficulties adapting to changing monitoring and evaluation processes and reporting obligations. For each process-related issue, clear and regular communication, along with exposure to one another over time, helped the HESN Development Labs and USAID MBIOs overcome those challenging process factors and improve the usefulness of their research. 5. Research Question 5. Contributions to changes in HEIs. HESN funding led to the development of more than 96 new courses and disciplines at HEIs. The host universities for RAN, AidData, MIT CITE, and IDIN also institutionalized their respective labs by providing funding for them to continue their international development work. 6. Cost analysis Outcomes. HESN generated more than $256 million in leveraged funds. That investment was more than two dollars leveraged for every dollar of U.S. government funding provided to HESN. Capacity-building events and investments in innovations were more cost￾efficient than other types of outputs produced under the program. The average cost per participant for an event ranged from $371 to $3,515. For innovations, the average cost per unique HESN beneficiary (number of beneficiaries per innovation/HESN Development Lab innovation expenditure) ranged from $2,273 to $81,564. These estimates are dependent on the number of outputs, products, or innovations produced by the HESN Development Labs. 7. HESN core, buy-in, and combined-funding activities directly benefited 7.1 million people worldwide. USAID.GOV PERFORMANCE EVALUATION OF HESN | xiv HESN addressed several needs of stakeholders in the development field. It improved the access that USAID and development partners have to quality data and evidence, and MBIOs found that HESN￾produced research had applicability for their work. USAID, HEIs, and local policymakers used HESN￾produced evidence and data as inputs to decision making and program design. HESN publications helped faculty and students increase their publication record and provided public data sets for other researchers to advance knowledge. HESN also supported entrepreneurs worldwide with opportunities to develop, test, and implement hundreds of technologies and ideas that helped communities overcome development challenges. It also created a global ecosystem of people and institutions that continue to work together to solve complex international development challenges. HESN supported the creation of 96 new courses, programs, and disciplines. It influenced 160 institutions and created more than 291 data sets or data-related technologies, tools, and approaches. According to HESN monitoring data, the most common HESN outputs included workshops and publications, followed by knowledge sharing and collaborative platforms. The program also spurred 993 innovations and has reached nearly 7.1 million direct beneficiaries to date (Figure ES.2). The HESN Development Labs varied in their program and policy change impacts, as suggested by the impact scores shown in the first cell of Figure 2. USAID devised the HESN program or policy change impact score to assess (1) the geographic reach, and (2) the degree of program or policy influence of a HESN Development Lab’s research outputs, with 6 being the highest score for policy and program influence. Using this metric, MIT IDIN and AidData had the highest impact scores, and SEAD had the lowest. Figure 2 highlights the main contributions of HESN. Figure 2. HESN at a glance *The HESN Program or policy change impact score was devised by USAID to assess the geographic reach and influence of a Lab’s outputs. Source: HESN monitoring data. Notes: Fowle et al. (2020). The HESN program or policy change impact score to assess the geographic reach and influence of a HESN Development Lab’s research outputs, with 6 being the highest score of policy and program influence. The labs submitted scores and justifications to USAID biannually. The limited cost-efficiency analysis finds that USAID’s investment in HESN brought value to the agency. For example, the HESN Development Labs leveraged $2.23 for every dollar that USAID invested in their work. The cost analysis also shows that the program's overall cost was $16.20 per unique, direct USAID.GOV PERFORMANCE EVALUATION OF HESN | xv beneficiary8. When we examined the cost-efficiency of the outputs that the HESN Development Labs produced, RAN and IDIN were consistently the most cost-efficient regardless of the type of output or product (that is, workshop, report, or event). These two labs worked primarily overseas, so the costs per output were lower than those of the other two HESN Development Labs. RAN is based in Uganda, which also lowers their overall operating costs relative to the labs in the United States. Across all four labs, workshops and the production of data sets tended to be the two most cost-efficient activities. HESN did face several structural, institutional, and process challenges during implementation. For example, internal university structures presented challenges for the HESN Development Labs during the start-up phase. Several labs specifically noted that the university’s bureaucracy often slowed their ability to hire partners and obtain approvals for work, travel, and staff. The labs also faced external barriers with partners that reduced their ability to complete activities efficiently. These ranged from difficulties moving money to partner organizations and grantees to challenges in communicating and collaborating with subcontractors and partners who were less able to complete the work. For example, because of other workload commitments, the HESN Development Labs sometimes struggled to engage experts from their HEIs, so they hired consultants to complete activities. The consultants occasionally lacked the same level of experience and rigor as the HEI faculty, which led to lower-quality products. The HESN Development Labs also indicated that they struggled to grow into a robust inter-lab network. They cited such factors as divergent focus areas, time constraints, and staffing limitations as reasons they struggled to work together. HESN Development Lab staff also noted that the limited pools of funding for buy-ins and core work implicitly fostered a sense of competition. The HESN Development Labs and MBIOs also struggled, in some cases, to find common ground. The barriers they cited as challenges included the misalignment of needs and goals, the bureaucracy of awarding activities, and staff turnover in both the HEIs and the MBIOs. E. RECOMMENDATIONS The following section provides USAID with recommendations that can help facilitate and improve the use and utility of research programs. Improving how MBIOs apply research or outputs to their programs. Adhere to a co-design model to engage USAID missions, operating units, and local governments in developing relevant and applicable activities. Although HESN processes included a co-design model for activities, mission staff often felt that activities were actually co-designed with all relevant stakeholders. Missions highly encouraged their Washington colleagues to adhere to a co-design process with MBIOs on future research programs. The co-design process can ensure that missions include funding in their country development cooperation strategies (CDCS) planning processes for new projects that meet country needs. The co-design process can be helpful at both the project level and the individual research activity level. HESN 2.0 appears to use a co-design process in aspects of its work. However, the process should also include local government if USAID wants to have a policy impact and uptake of research to drive evidence-based decision making. Because local governments were not the primary audience for deliverables, they were left out of original HESN activities. Maintain the focus on “growth mindset” and flexibility. As discussed in the report and our conclusions, stakeholders repeatedly identified the focus on flexibility and a “growth mindset” as the elements that most facilitated the success of HESN activities. To allow implementers to explore new avenues, build new partnerships, and adjust as they learn along the way, USAID should look for ways to maintain this flexibility on research-related programs. Failure often leads 8 The overall cost per unique beneficiary is calculated by dividing the total USAID investment in HESN of $115 million by the total number of unique beneficiaries, 7.1 million. USAID.GOV PERFORMANCE EVALUATION OF HESN | xvi to innovation, so being open to failure and supporting participants in their efforts to try again can facilitate learning. Improving the utility of research programs for stakeholders. Continue to focus on evidence, technology, and innovation. Many HESN activities contributed to development outcomes that are changing and affecting lives in LMICs—reaching nearly 35 million direct and indirect beneficiaries. USAID should continue to design and implement programs that focus on evidence, technology, and innovation, contributing to international development in both large and small ways. Continue to invest in capacity building. HESN successfully invested in different kinds of capacity building over time. USAID should continue to support local capacity building, because it empowers local participants to adapt and contribute to changes. The focus should be on “action without harm.” Assess the contributions of workshops and summits that identify local needs and gaps. The USAID-funded LASER program is using a model that engages MBIOs and local stakeholders to identify needs and gaps in development. The model allows the implementing university then to design interventions and activities that support local stakeholders. It is an interesting model that should be evaluated to determine the extent to which it contributes to MBIO future programming and the use of outcomes and products by stakeholders. Overcoming structural and institutional barriers. Monitor and reconcile diverse incentive systems. Differing incentive systems can present structural barriers to reaching goals. For example, universities are motivated to publish research findings, whereas USAID is motivated to serve local populations. USAID should consider these differing incentive systems under HESN 2.0 and work with universities to ensure that stakeholders use research in practical ways. HESN’s work provides examples of ways to reconcile the incentives. Development Gateway with AidData developed and implemented geocoding work that met the mission and government partners' needs. RAN worked hard to help researchers see the value of moving away from the “ivory tower” of research to working directly with communities. Although reconciling various incentive systems is not easy, lessons learned under HESN 1.0, including the importance of trust building, open communication, and flexibility, can help USAID and its partners move toward workable solutions. Also, as mentioned earlier, using workshops, summits, and events to reconcile different research agendas or identify local development needs also provides examples of successfully aligning incentive systems and can reconcile the different research agendas through the co-design process mentioned above. Find ways to align university research and USAID missions’ timelines to support the use of evidence. The differing timelines to complete deliverables between HEIs and donors were structural barriers that researchers faced in trying to complete activities. HEIs are accustomed to having two or three years to complete research products, but USAID MBIOs need quick evidence to take advantage of country policy opportunities. The HESN Development Labs suggested that the lack of “quick evidence” caused USAID leadership to lose interest and made it harder for them to get funding to continue some of the work. USAID staff felt intense pressure to show results to continue getting resources to implement HESN. The challenge noted by the HESN Development Labs and USAID AORs related to the type of research conducted under HESN, which they felt was not always oriented toward quick results. Therefore, the HESN Development Labs had to balance that need with the time necessary for quality research. Although challenging and sometimes outside USAID’s control, trying to expand on the buy-in system might help meet USAID missions’ shorter-term research needs. Strengthening process elements to improve research program utility. Provide examples of communication products that MBIOs feel best help them communicate findings and results. USAID has numerous examples of concise, well-prepared policy briefs and USAID.GOV PERFORMANCE EVALUATION OF HESN | xvii infographics that it can identify and use as examples. When preparing scopes of work or buy-in activities, MBIOs could include examples of communication pieces that successfully reach their target audiences. These examples can help academic institutions understand how to convert research into evidence and results that are usable and understandable to missions and policymakers. The most successful communication pieces (1) synthesize and frame evidence in a way that policymakers demand and understand information; (2) are provided at the “right time,” meaning that MBIOs can deliver the results during a time that key decision-makers in the mission or government are planning and have a need for the information; and (3) demonstrate an understanding of the policy audience and their planning processes. Staff from several HESN Development Labs and USAID also recommended hiring (or including) communication experts on research teams to facilitate the development of improved products that meet the needs of the client. Consider integrating stakeholder analysis as part of a new research activity. Stakeholder analyses can help implementers better understand the political and institutional context for research in a specific context, including the incentive systems for collaboration and uptake of research findings. These types of analyses can help ground findings and provide implementers and MBIOs with guidance on the types of products that will reach different audiences, as well as identify any potential roadblocks to the use of the evidence. Stakeholder analyses can also facilitate collaborations by helping all stakeholders understand the power dynamics among participating stakeholders. Sustaining changes at HEIs Engage directly with the government, universities, and the private sector to increase and sustain future funding. One of the biggest challenges in the innovations work was that, although the financing allowed innovators to work on new ideas, the budget rarely allowed them to pilot, test, and incubate the innovations at larger scales. Innovators noted that they needed time, funding, and space to “incubate” their ideas properly. A HESN Development Lab leader noted that “the original idea to improve products required a connection to the private sector…. However, legal issues and the timing of research, whose idea you take, who is in charge….There is a mismatch between the time and pace it takes to do university research compared to the private sector, and that limited their interest and engagement in HESN.” Future programs need to consider how to engage different actors successfully and bring them in to support the innovation process. For example, in Colombia, innovators noted that universities often provided additional funding for them to work on their ideas. In Brazil, the government frequently releases funding calls that allow innovators to apply for grants to support their ideas. In other countries, both international and local companies sometimes create “incubators” or “hubs” that allow innovators to develop their ideas. USAID and its implementers need to consider what works in the context of each country while ensuring that some support exists to help foster and sustain innovation. Build a more robust network among the labs. Consider eliminating the disincentives to collaboration among HESN Development Labs. New programs should build funding and activities for implementers, whether they are universities, NGOs, or private-sector firms, to collaborate and bring their networks together. These types of connections can help grow the global learning ecosystem. They also help sustain research work as the networks continue to collaborate in the post-program cycle, including seeking new funding for other activities emanating from USAID collaborations. “[To improve impact]...it would be good if there was broader participation and many more meetings to find the relevant questions yet unanswered that are most relevant to the mission strategies. It’s about identifying the challenges, so that information solutions are useful to them. Having more of a process for identification would be good, and more analysis of challenges before jumping in.” —Lab Staff USAID.GOV PERFORMANCE EVALUATION OF HESN | xviii Other Recommendations Improve collection of cost data to support more-detailed cost-analysis. USAID is rolling out its cost guidance under new programs, which will improve their ability to collect and report cost information in the future. It will also help evaluators conduct better cost analysis. In line with USAID’s guidance, we recommend that, at a minimum, future research programs collect and report on the following data: ● Determine appropriate cost categories for research programs. Categories should include elements such as program management, researcher time, travel, and other direct costs. ● Implementers should collect and report the level of effort, associate travel, and ODCs for each product or deliverable completed under the research project. The detailed disaggregation will allow USAID to determine the cost per deliverable quickly. ● For programs that have both core- and buy-in-funded activities, implementers need to track the cost of deliverables by type of funding. ● If USAID wants to complete a cost-effectiveness analysis, ensure that the implementer is able to set up some type of impact assessment from the beginning of the program. Impact estimates are crucial to comparing cost effectiveness. ● Ensure that shared costs across activities are accounted for throughout the life of a project and are reflected in the product costing and reporting. ● Ensure that implementers review USAID cost guidance and adapt it for the project in the monitoring and evaluation plan. USAID.GOV PERFORMANCE EVALUATION OF HESN | 1 I. INTRODUCTION HEIs are critical contributors to international development, and we see growing interest in HEI contributions across donors. The United Kingdom’s Department for International Development recently implemented a new partnership program for HEIs, and in 2015, USAID designated one of their four education priorities as “the strengthening of higher education and workforce development programs.”9 About 20 percent of World Bank funds support higher education, and 34 percent of the total education funding in the Development Assistance Committee10 of OECD countries went to higher education in 2015 (McCowan 2016). We have also seen an uptick in donor investments in the higher education systems in LMICs since 2015—a trend that the USAID-funded HESN program led beginning in 2012.11 McCowan (2016) cites three main reasons that the international community is investing more in higher education and research. First, the changing nature of the global political economy means that people value knowledge to improve economic competitiveness. Academic experience drives technological development, and it provides graduates with greater skills to adapt and use in the workplace. Second, HEIs provide professional capacity building, an essential piece of the international development puzzle. Third, research shows higher individual rates of return (that is, higher salaries) for those who completed tertiary education. USAID is one of the donors that work closely with HEIs to develop and use research to improve economic and social development in LMICs. In 2012, as part of their investment in HEIs, USAID formed a partnership with seven top universities to create the HESN program, designed to channel the ingenuity of university students, researchers, and faculty toward global development, which will drive technological development. The HESN Development Labs research, incubate, test, and accelerate solutions in partnership with local universities, organizations, and communities to deliver the most significant impact and continue to develop the capacity of researchers, students, and faculty to meet these challenges. The network has created a vibrant framework of cooperation between USAID missions, local actors, development professionals, and academics to tackle the complexities of modern-day development challenges. This report provides the results of the HESN performance evaluation. Section II details the purpose of the evaluation. Section III provides details on HESN and its objectives. Section IV provides an overview of the evaluation methodology, and Section V details the findings. Sections VI and VII present the conclusions and recommendations from the evaluation of HESN. 9 https://link.springer.com/article/10.1007/s10734-016-0035-7 and https://files.eric.ed.gov/fulltext/ED594404.pdf 10 The Development Assistance Committee of the OECD was established in 1961. Its present members are Australia, Austria, Belgium, Canada, Denmark, Finland, France, Germany, Greece, Ireland, Italy, Japan, Korea, Luxembourg, Netherlands, New Zealand, Norway, Portugal, Spain, Sweden, Switzerland, the United Kingdom, the United States and the European Union Institutions. 11 The period of performance under the core HESN Cooperative Agreements with Labs ran from fall 2012 to fall 2017; however, some labs extended their period of performance by signing agreements with USAID missions and operating units. USAID.GOV PERFORMANCE EVALUATION OF HESN | 2 II. PURPOSE OF THE EVALUATION As part of the trend in donor commitment to higher education, the USAID-funded Expanding the Reach of Impact Evaluation consortium has partnered with USAID's Global Development Lab to assess the long￾term impacts of HESN.12 The HESN-funded Development Labs include (1) the College of William and Mary’s AidData Center for Policy Development; (2) the Massachusetts Institute of Technology (MIT) Comprehensive Initiative on Technology Evaluation (CITE); (3) the MIT International Development Innovation Network (IDIN); (4) Makerere University ResilientAfrica Network; (5) Duke University Social Entrepreneurship Accelerator at Duke (SEAD); (6) University of Texas A&M Center on Conflict and Development (ConDev); (7) University of California, Berkeley, Development Impact Lab (DIL); and (8) Michigan State University Global Center for Food Systems Innovation (GCFSI) (see Table II.1). USAID invested $115 million in these centers to support the development of research and innovation. This original investment in core-funded activities leveraged more than $200 million of additional funds from academic, private-sector, governmental, and other stakeholders for development projects, as well as an additional $56 million in equity, debt, and philanthropic support for entrepreneurs. The funding also supported more than 500 new and existing innovations and trained more than 2,000 professionals worldwide. Table II.1. HESN International Development Labs LAB NAME HOST UNIVERSITY LAB APPROACH AidData Center for Development Policy College of William and Mary Cross-sectoral Center on Conflict and Development (ConDev) Texas A&M University Sector-specific Development Impact Lab (DIL) University of California, Berkeley Cross-specific Global Center for Food Systems Innovation Michigan State University Sector-specific Comprehensive Initiative on Technology Evaluation Massachusetts Institute of Technology Cross-sector International Development Innovation Network (IDIN) Massachusetts Institute of Technology Cross-sector ResilientAfrica Network Makerere University Cross-sector Social Entrepreneurship Accelerator at Duke (SEAD) Duke University Sector-specific Source: HESN documents, survey, and interview data. As the current programs have shifted more toward mission engagement, the Center for Development Research (CDR) is interested in understanding the utility and impact of HESN 1.0 projects on USAID missions, local LMIC partners, and researchers. USAID also wants to understand the conditions and models that generate effective partnerships with USAID missions, how the activities contributed to improved use of research findings, and the potential LMIC policy impacts. The evaluation questions for the present study include: 12 USAID directs Partnerships for Enhanced Engagement in Research (PEER) to support researchers and scientists in developing countries as they seek solutions to development challenges. Through PEER and partnerships with other U.S. government agencies, scholars in more than 50 countries have gained access to $50 million for research projects in water management, agriculture, energy, and other sectors. USAID.GOV PERFORMANCE EVALUATION OF HESN | 3 1. To what extent have mission partners applied learnings from HESN 1.0 research or outputs to their programs? a. What was the usefulness of the buy-ins to their programs (how pertinent, relevant, and timely were the outputs)? b. What was the utility of buy-ins to programs (how much did policymakers and stakeholders use the outputs)? c. To what extent did the usefulness and utility of the buy-ins vary by funding source? 2. What was the partnership’s perceived utility among stakeholders (USAID mission, HESN Development Labs, and policymakers)? 3. Which structural or institutional elements of the partnership contributed to different levels of usefulness to mission programming and decision making? Which elements contributed to utility? 4. Which process elements of the partnership contributed to different levels of usefulness to mission programming and decision making? Which elements contributed to utility? 5. To what extent has HESN contributed to changes in HEIs or in HEI networks that increase their engagement in international development? To what extent would any changes be sustained? Why or why not? The evaluation serves three primary audiences: (1) USAID Washington, missions, and operating units; (2) the HESN Development Labs; and (3) other development stakeholders and members of the public who are interested in how donor funding can support HEIs. Evaluation results will support USAID in three ways: 1. They will support USAID and CDR to design and implement new research and higher education programs. The findings will help adjust designs to overcome and mitigate any structural or process barriers and improve the utility of research outcomes on future programs. 2. They will highlight the value-added of USAID investments in research and innovation. 3. They will show how a global ecosystem of innovators can use ideas to help solve development challenges in their communities. USAID.GOV PERFORMANCE EVALUATION OF HESN | 4 III. HESN BACKGROUND A. OVERVIEW OF HESN 1. Description of the HESN program HESN is a cooperative agreement between USAID and seven top universities. It is designed to channel the ingenuity of university students, researchers, and faculty toward global development through a set of core objectives (funded through USAID Washington) and buy-in activities from MBIOs. (A “buy-in” is a process that USAID MBIOs can use to support activities of interest under HESN.) The missions and operating units provide supplemental funding (from their own budgets) to the HESN cooperative agreement to support the research or innovation efforts of specific labs that serve the needs of the MBIO. USAID designed HESN to meet three core objectives, highlighted in Figure III.1. Figure III.1. HESN Objectives Notes: Source is HESN Midterm Performance Evaluation, 2016. USAID invested in the HESN Development Labs to improve the quality, access, and use of data to support evidence-based development. The agency also believed that the investments would accelerate the creation, testing, and scaling of technological innovations and catalyze a global network of institutions to share knowledge and lessons and build the capacity to solve critical development problems. Figure III.2 shows the HESN results chain. USAID.GOV PERFORMANCE EVALUATION OF HESN | 5 Figure III.2. HESN results chain Source: USAID, 2012. At the launch of the partnerships in 2012, each university established a HESN Development Lab (two at MIT) and created a network of researchers and practitioners with a mission to revolutionize development through science and innovation. The eight HESN Development Labs researched, incubated, tested, and implemented solutions to development challenges and collaborated with local universities, organizations, and communities to deliver the most useful research for development practitioners. The network sought to create a framework of cooperation between USAID Washington and missions, local actors, development professionals, and academics to tackle the complexities of modern-day development challenges. Soon after the initiation of their awards, the HESN Development Labs began working with their respective partners (such as other HEIs) and MBIOs to address development challenges across the sectors. The HESN partnership funded these activities in three ways: (1) HESN core funding, (2) MBIO funding (as a buy-in), or (3) co-funding by both HESN and MBIOs. The buy-in opportunities often came from MBIOs, who reached out to the AORs. In other cases, the HESN Development Labs, with the support and their AORs, reached out to the MBIOs directly and engaged them about possible activities. Several of the labs, such as AidData, also pursued buy-in activities by directly engaging with MBIOs. 2. Implementing organizations (HESN Development Labs) The research expertise of the HESN Development Labs and the services offered to missions and partners varied across the partnerships. The labs might have had global research and innovation goals, but they were based primarily in U.S. universities. In contrast, one HESN Development Lab, ResilientAfrica Network, is located overseas at Makerere University in Kampala, Uganda (Figure III.3). USAID.GOV PERFORMANCE EVALUATION OF HESN | 6 Figure III.3. Location of HESN Development Labs The following examples show the breadth of work conducted by four of the HESN Development Labs. ● At MIT, the Comprehensive Initiative on Technology Evaluation and the International Development Innovation Network (MIT IDIN) offers multiple research and innovation capabilities. MIT CITE was launched in 2012 “to develop new methods for product evaluation and the impacts and potential of technology in terms of suitability, scalability, and sustainability.” Since 2017, the lab has expanded its work areas to include such varied topics as fairness in machine learning, inclusive systems innovation, and digital financial services. MIT IDIN, also founded in 2012, was inspired by, and continued the implementation of, International Development Design Summits (IDDS) organized by MIT D-Lab, the Olin College of Engineering, and local partners since 2007. IDIN organized 17 IDDS between 2012 and 2017, bringing together international partners and local stakeholders in communities worldwide to co-design solutions to challenges in poverty, health, and food security. Since 2017, IDIN has continued this work with a volunteer steering committee and students engaging with communities through class projects and fieldwork (MIT CITE 2020; MIT D-Lab/IDIN 2020). ● AidData, a research lab at the College of William and Mary, was launched to produce, disseminate, and analyze geospatial data to track the impact of aid flows. Since 2012, AidData has released numerous geospatial tools and methods for researchers and practitioners worldwide. The lab also conducts geospatial impact evaluations to aid in development decision making. These products are of particular interest to bilateral donors interested in assessing the impact of their aid dollars on a specific region. Through HESN core funding and buy-ins, USAID missions and government agencies in LMICs (including Niger, Uganda, and Nepal) benefited from geospatial impact evaluations and increased data access, and even trained on geospatial data management and analysis. For example, AidData’s geocoding allowed Nepal’s Ministry of Finance to map and better understand the country’s universe of aid projects and address regional imbalances in development finance. Through the HESN project, AidData has expanded its focus and now also offers research capabilities in such areas as artificial intelligence and the perspectives and goals of development policy leaders in LMICs worldwide (AidData 2020). USAID.GOV PERFORMANCE EVALUATION OF HESN | 7 ● The ResilientAfrica Network works through four HESN Resilience Innovation Labs and numerous universities across Africa to conduct research and identify and support innovations that strengthen community resilience. The HESN Development Lab selects, supports, and funds innovations that address regional issues ranging from vulnerability to drought, urbanization, and the impact of HIV/AIDS on livelihoods. The innovations provide a myriad of solutions, from digital financial literacy training and E-Health for refugees to reproductive health information to refugees and low-cost maize threshers (RAN 2017). 3. Target population The target populations for the HESN program include HESN Development Lab staff, partners, researchers, and innovators; USAID MBIOs; policymakers in target countries; U.S. universities, including students and faculty; and local communities in USAID partner countries. Figure III.4 shows the HESN stakeholders and their roles in terms of supplying or demanding research. Figure III.4. HESN stakeholder summary Source: Created by Mathematica based on an analysis of data provided by the HESN Development Labs and USAID, 2020. The stakeholders that support the supply side of HESN include faculty, staff, and students at the eight HESN Development Labs, which are responsible for designing activities and supplying information, research, tools, and training to other stakeholders. The project roles included improving data quality, participating in meetings, conducting workshops, identifying mutually useful activities, and designing research and data-related tools to support demand-side stakeholders. The labs delivered training activities and supported innovators to learn, design, and implement their ideas. The role of the HESN Development Lab partners (including other universities) included providing support to the labs and supplying other faculty, staff, and students who could help carry out the work. On the demand side, a series of different stakeholders, such as policymakers, MBIOs, U.S. and global universities, and local communities drove the need for better data and data-related technologies and innovations. Policymakers demand high-quality data to inform decision-making throughout the government. The MBIOs demand data, capacity-building, and research to meet their CDCS objectives and support the target country in advancing their development goals. The U.S. and global universities demand data sets to support new research for faculty and students. The local communities and USAID.GOV PERFORMANCE EVALUATION OF HESN | 8 innovators require funding to create new technologies and capacity-building to overcome their development challenges. HESN also included several stakeholders who supported the HESN Development Labs and enabled them to meet their objectives. These included the AORs, who helped the labs negotiate USAID systems and supported them to network with MBIOs. Supporting stakeholders also included university administrations that enabled the labs to participate in HESN and ultimately helped their sustainability. Finally, all stakeholders (including MBIOs and HESN partners) that helped disseminate products, tools, and other HESN outputs provided essential supports to the HESN program. USAID.GOV PERFORMANCE EVALUATION OF HESN | 9 IV. EVALUATION DESIGN AND METHODOLOGY A. EVALUATION METHODOLOGY The evaluation’s overall goal is to generate learning that informs USAID decision making to improve future research programs designed and implemented by the agency. The evaluation uses a mixed￾methods approach with three components: (1) an implementation study, (2) outcome analysis, and (3) a cost analysis. The implementation study employs qualitative methods to explain the HESN program’s use and utility to beneficiaries. It supports the analysis of facilitators and barriers to implementation as well as draws out lessons learned. The outcome analysis employs quantitative techniques, including a descriptive analysis of statistics, bibliometrics, and descriptive analysis of monitoring, evaluation, and learning indicator data, to determine the extent to which HESN met its goals and objectives. The third component—the cost analysis—allows us to understand the value-added of HESN activities. In the rest of this section, we describe how each of these components contributes to answering the research questions. Table IV.1. Evaluation questions and components EVALUATION QUESTIONS EVALUATION COMPONENT IMPLEMENTATION STUDY OUTCOMES ANALYSIS COST￾EFFECTIVENESS ANALYSIS To what extent have mission partners applied learnings from HESN 1.0 research or outputs to their programs? X X X What was the partnership’s perceived utility among stakeholders (mission, HESN Development Labs, policymakers)? X X Which structural or institutional elements of the partnership contribute to different levels of usefulness to mission programming and decision making? Utility? X Which process elements of the partnership contribute to different levels of usefulness to mission programming and decision making? Utility? X To what extent has HESN contributed to changes in HEIs or in HEI networks that increase their engagement in international development? To what extent would any changes be sustained? Why or why not? X X X Source: Mathematica design USAID.GOV PERFORMANCE EVALUATION OF HESN | 10 1. Implementation study We selected three countries for qualitative field research. Colombia. Colombia housed a HESN buy-in (valued at almost $1.5 million). The buy-in supports USAID/Colombia with impact evaluation of the Colombia Regional Governance Activity in partnership with the government of Colombia. Ghana. Ghana has a mix of core and buy-in activities and allowed evaluators to gather data related to MIT CITE, IDIN, and RAN work on the innovative technologies. It also allowed us to collect the most data linking to various HESN activities. Uganda. Almost all the HESN Development Labs had activities in Uganda. Uganda also had high uptake by USAID/Uganda and partners. Selection of HESN countries An implementation study examines factors affecting implementation, the processes implementers followed, and the project's results, including how to introduce potential solutions into systems or how to promote their large-scale use and sustainability. The intent is to understand which interventions work in real-world settings, how and why the interventions work, and what test approaches can improve those interventions (Peters et al. 2014). We employed both qualitative and quantitative methods in this component of the evaluation. The research methods included document review, administrative data analysis, online surveys, key informant interviews, and focus groups (see Annex B for details on the sampling, methods, and stakeholder groups). Interview and focus group data focused primarily on four target HESN Development Labs and activities in Colombia, Ghana, and Uganda. We selected AidData, MIT CITE, MIT IDIN, and RAN as the target HESN Development Labs because of the mix of funding modalities used to implement their activities, the range of staff activities, and the different country contexts. The mix allowed us to understand each funding modality's various uses and assess the barriers and facilitators the HESN Development Labs faced in implementing their activities in each country. The remaining instruments collected information from all eight HESN Development Labs and their partners. We used descriptive analysis to explain the results of the online survey and a two-step approach to analyzing qualitative data. We reviewed the recordings and detailed notes from interviews and documented the main findings in an Excel file. Then we uploaded the Excel file with the main findings for each research question into NVivo and coded the results thematically to draw out details and sub-themes under each question. We triangulated the results with findings from the document review and the online survey. 2. Analysis of outcomes The purpose of the outcomes analysis is to understand the extent to which HESN delivered on its medium- and long-term objectives. We conducted a bibliometric analysis, reviewed USAID’s policy change scorecard (Fowle et al. 2020), and analyzed the project’s administrative data. We analyzed the bibliometric and administrative data from all eight HESN Development Labs. Annex B contains the methodological details for the analysis. We used a three-step process to examine the differences in the use and utility of HESN activities and outputs by funding modality. First, we analyzed survey data related to the activities that USAID staff had identified in the online survey as best representing HESN goals. A total of 26 USAID staff survey respondents shared their perspectives on the use and utility of 19 different HESN activities. Second, we reviewed the qualitative interview data and identified 8 additional HESN activities to include in our review of the differences between funding modalities and added the qualitative analysis on the use and utility of these activities to the findings. Of the 27 activities, 5 were core funded, 19 buy-in funded, and 3 jointly funded. Third, to gather additional information on the use and utility of outputs, we crossed these 27 HESN activities with M&E data and information from our document review. We discuss the results of this analysis in the outcomes section. USAID.GOV PERFORMANCE EVALUATION OF HESN | 11 3. Cost analysis Mathematica conducted a series of broad cost analyses for this evaluation. We calculated the amount of leveraged funds per dollar invested and the cost per beneficiary based on USAID’s overall investment in HESN. We also conducted a cost-economy and cost-efficiency analysis for a limited set of cost and output data that we acquired from AidData, CITE, IDIN, and RAN. We describe and present the methodology and results for each type of analysis in Section E. B. DATA COLLECTION PROCESS Mathematica conducted a mixed-methods performance evaluation13 that (1) explored how HESN used funds to generate research and innovation; (2) determined whether outputs and outcomes of HESN activities informed policy; (3) examined whether HESN funding modalities (for example, CDR funded, mission funded, or co-funded) contributed to any differences in the type of research and innovation produced by HESN Development Labs and whether missions and other stakeholders used results and outputs; (4) helped contextualize outcome results by describing the geographic, social, and policy environment in which several HESN Development Labs work; and (5) provided a deeper understanding of how mission engagement of the HESN partnerships contributed to differences in the uptake of research and innovation outputs by missions and other key stakeholders. The performance evaluation conducted one round of data collection that included document review, administrative data (that is, monitoring and reporting indicators), online surveys, interviews, and focus groups with key stakeholders. Figure IV.1 highlights the steps we followed to complete the data collection. Figure IV.1. Data collection process The data collection process began with a review of key documents from USAID and HESN, including the cooperative agreements between USAID and each lab, the labs’ annual reports, project descriptions, labs’ monitoring and evaluation plans, the 2016 midterm HESN evaluation, and the 2018 USAID HESN impact report. We used findings from our close review of those files to develop an online survey that allowed the evaluators to target different audiences and collect information abound gaps identified in the document review. We distributed the survey to key staff at the eight HESN Development Labs, USAID missions and operating units, researchers, innovators, university partners, and HEI leaders. Each audience received a survey targeting specific areas based on their roles and responsibilities under HESN. 13 Mixed-methods process evaluations use both quantitative and qualitative data to examine the degree to which an intervention was implemented as intended (that is, followed the implementation process). USAID.GOV PERFORMANCE EVALUATION OF HESN | 12 Upon completing the survey, our team analyzed the data and identified other gaps that became the focus of the qualitative key informant interviews and focus groups. C. LIMITATIONS Several factors limit the evaluation’s strength, including selection bias, recall bias, and the complications posed by the COVID-19 pandemic. 1. Nonresponse bias. About 36 percent of survey respondents completed the questionnaire partially or fully. Although this percentage is generally satisfactory for online surveys, nonresponse bias implies that the views of 64 percent of potential respondents are absent from our analysis. Further, 38 of the 163 respondents noted in their surveys that they did not want to be interviewed, which narrows our qualitative data sources pool. Finally, 9 (of 82 total) stakeholders to whom we reached out for interviews did not respond to our requests, which again introduces the risk of missed qualitative data. 2. Recall bias. A subset of potential respondents with direct experience of HESN left their HESN￾relevant positions for other areas of work, engaged in new activities, or moved to recent international posts. This situation means that our study did not have access to the institutional memory of all HESN stakeholders. Ten survey respondents (one HEI partner, four innovators, one researcher, and four USAID staff) indicated that they were not familiar with HESN despite being associated with HESN￾supported projects. The lack of familiarity could be related to the funding source for the relevant work. For example, several interview respondents linked the activities on which they worked with the HEI, a lab, or the project's name (for example, SOGE) but did not understand the linkage between their work and the HESN program. It also meant that we could not collect full data from them on their HESN program experience. 3. COVID-19. After completing in-person data collection in Colombia in early March, USAID and Mathematica decided, because of coronavirus travel restrictions, to cancel similar plans in Ghana and Uganda. Video and phone interviews and focus groups allowed us to reach nearly all our target stakeholders. Still, offices of some stakeholder institutions abroad were closed, which delayed interviews with potential respondents. We were also unable to interview policymakers in Uganda and Ghana due to office closures and specific protocols that needed we needed to complete in person. In￾person sessions and accompanying field observations might have provided richer qualitative information than video or phone sessions. 4. Shift to virtual interviews. To complete the evaluation during the coronavirus pandemic, the research team switched to virtual data collection, which presented four main challenges. First, several participants were unfamiliar with WebEx online meeting technology, so it took time to help interviewees connect and use WebEx. This process sometimes shortened the interviews. Second, many of the virtual interviews were across international lines, which risked poor connectivity. Video connections sometimes froze, and we had to re-establish them. Third, although video calls were better than phone calls because they allowed us to see the people we were interviewing, it was not always easy to read people’s body language or interpret visual cues, which usually help interviewers guide an interview. Finally, people’s comfort level with video chatting can affect the depth of their responses. Although the team continually probed respondents about content, culture, social norms, and comfort with video technology, the video mode may have affected the quality of their answers. 5. Mathematica faced several limitations in conducting the cost analysis, including: – The HESN Development Labs were not required to monitor costs based on activities they completed under core funding. When we contacted the labs to obtain cost data for each activity, they noted that they were not able to break down the funding by activity or by cost category. The lack of detailed data at the activity level prevented us from understanding how much of the core fund went into the different types of products that each lab had completed during HESN 1.0. As a result, we had to make broad assumptions and estimate the approximate funding that might go USAID.GOV PERFORMANCE EVALUATION OF HESN | 13 into an activity (similar to weighting). Note that our estimates are rudimentary at best and should be interpreted in that light. – The HESN Development Labs were also unable to break out the buy-in funding by cost category. When estimating the cost of specific buy-in-related outputs, we were unable to account properly for recurrent versus one-time costs, which can provide better estimates for scaling and sustainability purposes. The current findings use the overall funding provided for the buy-in and divide that by the number of products that were produced. – The only data we could obtain for the cost analysis were for name of activity, overall funding for buy-in activities, overall budget for core-funded activities (no breakdown by output or cost categories), and outputs provided in the M&E database. Outputs were often not specifically aligned to the activity listed in the cost spreadsheet, so we had to cross-walk several different files, including program descriptions to try to confirm the main outputs for the sample of activities we reviewed. In many cases, we could not confirm that the outputs listed in the program descriptions coincided with final, approved deliverables. The lack of alignment between the activities, outputs, and funding means that the cost analysis below can provide high-level guidance but should be used carefully. – Mathematica used multiple data sources and data collection methods to triangulate results and strengthen the internal validity of its evaluations. The team used document review, key informant interviews, and quantitative surveys to understand the implementation process and draw conclusions and recommendations to support USAID’s learning process. Although the evaluation has limitations, including the inability to draw causal inferences, recall bias, and nonresponse bias, we believe that the targeted and purposeful nature of the interview process helped improve the quality of the findings from the other instruments. The key information interviews also helped the team to fill in gaps or target response groups who did not respond to the survey. We believe that triangulating the various methods helped ensure that the findings of this evaluation are robust and accurate. USAID.GOV PERFORMANCE EVALUATION OF HESN | 14 V. FINDINGS Chapter V details the evaluation findings. Section A presents a summary of the HESN implementation process and related outputs. Section B presents the structural, institutional, and process barriers and facilitators to implementation (Research Questions 3 and 4). Section C shows results and outcomes related to the remaining research questions. The results’ subsections include the utility of HESN to USAID (Research Question 1), the utility of HESN to the partnership (Research Question 2), and the contributions of HESN to HEIs (Research Question 5). Section D examines the contributions of HESN on faculty careers (bibliometric analysis), and Section E presents the cost analysis. Subsection F presents sustainability findings. Figure V.1 summarizes the overall evaluation findings by research question. Figure V.1. Summary of the overall evaluation findings 1. Research Question 1. MBIO application of outputs to programs. The application of HESN learnings by USAID mission partners varied according to mission and activity. HESN Development Lab activities developed in collaboration with missions and funded as either buy-ins or in collaboration with core funding were more immediately useful than purely core-funded work because they typically supported the mission’s in-country objectives and strategies. 2. Research Question 2. Perceived utility of the partnership by stakeholders. The partnership with universities allowed USAID to strengthen its ability to undertake and use research. HESN also helped support local innovation ecosystems, and it helped the HESN Development Labs build their institutional capacity to work with USAID. 3. Research Question 3. Structural and institutional elements that contribute to the different levels of usefulness in programming and decision making. Bureaucratic funding and approval structures (both within HEIs hosting HESN Development Labs and with external partners) delayed research efforts and occasionally reduced the timeliness of research products. When HESN Development Labs relied on close support from their AORs and flexibility from USAID, as well as streamlined grant and funding systems in their host HEIs, they were better able to navigate or avoid bureaucratic barriers to the production of timely research. 4. Research Question 4. Process elements that contribute to different levels of usefulness in programming and decision making. Differences in planning processes and internal timelines between MBIOs and HESN-funded activities led to challenges in commissioning and completing work. Distinct communication styles also made it hard for some HESN Development Labs to convey findings to USAID in practical ways for immediate use. Finally, some HESN Development Labs encountered difficulties adapting to changing M&E processes and reporting obligations. For each process-related issue, clear and regular communication, along with exposure to one another over time, helped HESN Development Labs and USAID MBIOs overcome those challenging process factors and improve the usefulness of their research. 5. Research Question 5. HESN generated more than $256 million in leveraged funds. That investment was more than two dollars leveraged for every dollar of U.S. government funding provided to HESN. Capacity-building events and investments in innovations were more cost￾efficient than other types of outputs produced under the program. The average cost per participant for an event ranged from $371 to $3,515. For innovations, the average cost per unique HESN beneficiary (number of beneficiaries per innovation/HESN Development Lab innovation expenditure) ranged from $2,273 to $81,564. These estimates are dependent on the number of outputs, products, or innovations produced by the HESN Development Labs USAID.GOV PERFORMANCE EVALUATION OF HESN | 15 Figure V.1. continued 6. Cost analysis Outcomes. HESN generated more than $256 million in leveraged funds. That investment was more than two dollars leveraged for every dollar of U.S. government funding provided to HESN. Capacity-building events and investments in innovations were more cost￾efficient than other types of outputs produced under the program. The average cost per unique HESN beneficiary in the four target HESN Development Labs ranged from $2,273 to $229,913, depending on the number of outputs, products, or innovations the labs produced. 7. HESN core, buy-in, and combined-funding activities directly benefited 7.1 million people worldwide. A. IMPLEMENTATION SUMMARY HESN implementation evolved from 2012 to 2018. Funding for the core activities of HESN began in 2012. Each HESN Development Lab was designed independently, with a unique mission, scope, approach, and level of overall funding. Each received obligated core funds on an annual basis through 2018. The amount of core activity funds allocated to each lab over the five years ranged from a low of $4.4 million to a high of $22.3 million. In 2014, the Office of Acquisition and Assistance at USAID approved a total estimated cost (program ceiling) increase that provided space for additional funding for the MBIOs to begin related but separate activities by working directly with the HESN Development Labs through buy-ins. Figure V.2. highlights the main milestones of HESN. Figure V.2. HESN implementation and evaluation timeline Source: HESN program documents and interviews. The addition of buy-ins enabled different parts of USAID to tap into the labs’ research and innovation offerings and the technical expertise in the various HESN Development Labs and partner organizations. Table V.1 summarizes the different objectives, roles, and core versus buy-in-funded work timelines. Table V.1. Differences between core- and buy-in-funded activities CORE FUNDING BUY-IN HESN activity objectives Exploratory and open-ended; targeted broad international development audience Targeted to MBIO strategic and country priorities; MBIO and country-focused Primarily responsible for research/innovation agenda HESN Development Labs and USAID HESN team MBIO staff in collaboration with HESN Development Labs and USAID HESN AORs Activity timelines 3+ years <3 years Products and deliverables Public goods and viable innovations Inputs to CDCS and in-country development objectives Source: HESN program documents and interviews. USAID.GOV PERFORMANCE EVALUATION OF HESN | 16 The USAID AOR team worked closely with HESN Development Labs to introduce and connect staff to the MBIOs. This step increased the pertinence and relevance of the HESN work to the needs of specific MBIOs. HESN Development Labs had various opportunities to present their work to MBIOs and demonstrate how it could be useful to USAID and needs of developing countries. In some instances, MBIOs saw the value-added of an activity, and the interest led to creating a buy-in activity. In other cases, even if MBIOs did not have an immediate need that HESN Development Labs could meet, they became aware of their offerings and could consider them when a need arose in later years. Some of the MBIO interviewees noted, “We heard about HESN, and when we had a particular need and available funding, we contacted the USAID HESN team to explore which HESN Development Lab’s offerings could best suit our needs.” Once an MBIO decided to work with a HESN Development Lab, it collaborated to develop a scope of work for the buy-in. As a result of this engagement, HESN Development Labs have implemented more than 40 buy-in activities worldwide. 2016 was the peak year of outputs produced by the HESN Development Labs. From 2016 to 2017 (operational years 4 and 5), the tapering of HESN core funding coincided with an increase in MBIO buy-in activities, which began in 2014. These two factors converged to increase the production of deliverables under HESN. The activities supported with core funding were still producing outputs, and buy-in activities from MBIOs delivered new outputs. Figure V.3 shows the products completed by HESN Development Labs each year. The types of products and deliverables include workshops, data innovations, and other research products. HESN Development Labs such as RAN and AidData continued to build on outputs through their work on other activities with both USAID and other donors. For example, the EpiTent innovation (a re-imagined tent for emergency human-led service delivery in hot climates) developed with RAN support has generated income that has helped sustain their work. AidData recently won a $600,000, two-year grant with the Hewlett Foundation to build research partnerships in Africa. They also won a Ford Foundation grant to close the evidence gap on China’s Belt and Road initiative. Figure V.3. HESN Development Lab outputs produced by year, by HESN Development Lab Notes: HESN M&E data. *2013 includes one output completed in 2012: Establishing ConDev. USAID.GOV PERFORMANCE EVALUATION OF HESN | 17 A new regulation introduced in 2017 ensured that all proposed work was relevant to MBIO’s coordinated efforts. In 2017, USAID updated regulation ADS 201 to require that all USAID offices gain mission concurrence on any proposed activities. Although HESN activities had long coordinated and informed missions of their work, missions stated that the new concurrence regulation ensured the work's relevance implemented in-country. The gold standard was for HESN activities to feed directly into mission work; however, it was not always easy to align activities with those implemented by missions under their current country strategies. In these instances, missions simply gave their approval for the work to happen in-country and had limited involvement with the HESN activity. For example, several of RAN’s HESN￾funded innovation activities in Uganda did not align with the mission’s ongoing work at the time. With mission approval, RAN implemented several activities in partnership with other local government and community partners. Although CITE had several mission buy-ins, it also encountered instances where there was a misalignment on the focus of HESN activities. For example, USAID/India did not consider water filters a priority. In this case, CITE implemented water filter activities with other local partners. Even though MBIOs worked within the new regulation, some HESN Development Labs and USAID staff noted that the ADS concurrence approval requirements felt time￾consuming and bureaucratic and limited the HESN Development Lab’s ability to engage missions with speed and flexibility. For example, there were instances where missions approved an activity at the initial review stage. Because of changes in leadership or in the sociopolitical context, missions later rejected the activity at the formal-approval stage. In some cases, missions just rejected an activity without explanation. To help build support for HESN activities among mission staff, one HESN Development Lab recommended having more opportunities or avenues for engaging directly with missions at the country level. HESN Development Labs completed many types of deliverables during the five-year program. According to the M&E data, the most common outputs were workshops and publications, followed by knowledge sharing and collaborative platforms. Workshop examples include IDIN’s Creative Capacity Building sessions, IDDS, and training workshops on sector￾specific areas such as agriculture conservation. Reports included technology assessments from CITE, evaluation reports, and research papers, and knowledge-sharing and collaborative platforms included such systems as AidData’s GeoQuery database. Individual researcher and faculty efforts often contributed to several output types; for example, a data tool may also have an associated report. In the case of IDIN, creative capacity building theories generated both methodology documents and a workshop series. In Figure V.4, we present the outputs of HESN Development Labs by type. “HESN was very intentional about mission engagement and seeking concurrence, especially later. As the agency’s processes changed to mission concurrence, the pendulum swung back toward centralization of priorities (toward USAID central) and made things harder.” —USAID Staff “The system detailed all research proposed by all HESN Development Labs in a given country to the mission director, and then the new concurrence process would ensure the projects were aligned with mission goals. However, sometimes missions would reject proposed projects without any explanation.” —USAID Staff USAID.GOV PERFORMANCE EVALUATION OF HESN | 18 Figure V.4. Number of HESN outputs by HESN Development Lab and type Source: HESN M&E data. According to the online survey,14 the most common dissemination methods for outputs, such as those listed in Figure V.4, were in-person presentations, email, social media, and workshops. The most common dissemination audiences were researchers and innovators in the target country, other stakeholders in the location where the research took place, missions, and policymakers. Survey results show that most stakeholders participated in five or fewer HESN activities, but a quarter of respondents participated in six or more. Engagement of the MBIOs in HESN activities was a top priority for HESN Development Labs. Several labs worked diligently with the USAID HESN team and the MBIOs to identify opportunities to apply their expertise and even shift their focus to meet specific mission or OU objectives. When asked how many activities they participated in since 2015, 75 percent of stakeholders (MBIO staff, HEI staff, researchers, and HESN Development Lab staff) reported participating in five or fewer HESN activities (Evaluation Survey 2019). However, about 25 percent of stakeholders participated in six or more—thus signaling a high engagement level with HESN. Over half these highly engaged stakeholders were USAID AORs and HESN Development Lab staff. In Figure V.5, we show these proportions for the 135 survey respondents. 14 Information from the survey is not representative and focuses on two or three projects identified by each respondent as best meeting HESN goals. USAID.GOV PERFORMANCE EVALUATION OF HESN | 19 Figure V.5. Number of activities in which surveyed HESN stakeholders participated Notes: HESN online survey. A total of 135 respondents, including 28 USAID staff, 52 HESN Development Lab staff, 5 researchers, 46 innovators, and 4 HEI leaders. Stakeholder groups had different experiences with the HESN program. Figure V.6 presents word clouds by stakeholder type. These word clouds indicate how frequently stakeholders mentioned words and themes during their interviews. The word clouds show that policymakers tended to say “systems” more than other stakeholders, whereas innovators and researchers tended to discuss “innovations” and “communities.” HESN Development Lab staff and higher education leaders discussed “innovations,” “funding,” and “research” most frequently. All stakeholders mentioned “development” often. These high￾level word-frequency findings are congruent with our in-depth analysis of the qualitative data. They also align with the fact that stakeholders had different contact points with HESN and took away distinct experiences from their work with the program. USAID.GOV PERFORMANCE EVALUATION OF HESN | 20 Figure V.6. Word frequency of interviewees, by stakeholder group Source: HESN interviews. Each word is represented in a size corresponding to its frequency.15 The final set of HESN activities reflects a wide variety of goals, objectives, sectors, and funding sources. The objectives of HESN activities ranged from collecting primary data to filling data gaps, mapping donor engagement abroad, and conducting evaluations and workshops. Activities supported all social sectors, but the primary focus was agriculture, health, the environment, and education. Survey respondents indicated that the most frequently used funding sources for the activities were HESN core funding, USAID mission, USAID OU, and unspecified USAID (could be HESN/mission/OU). In Table V.2, we provide details related to the activities identified by survey respondents as best representing HESN goals. 15 “Work” and other general words in the cloud are built up by aggregating synonyms and similar words. Work, in this case, captures similar words like “act,” “make,” “operate,” and “function.” Although word frequencies do not offer a rigorous analysis, they do help us visualize the most common areas that interviewees discussed. USAID.GOV PERFORMANCE EVALUATION OF HESN | 21 Table V.2. Activities that survey respondents identified as best representing HESN goals Objectives: The most common objectives of HESN-supported activities were to collect data, build partner capacity, fill data gaps, and conduct impact evaluations. Sectors: The most common sectors in which HESN supported research or innovation were agriculture, health, environment, and education. HESN Development Labs: The majority of respondents identified at least one activity with AidData, and close to one-third identified activities with IDIN. Funding sources (possibly more than one): Funds came from HESN (80 percent), missions (24 percent), OUs (18 percent), and unspecified USAID (12 percent: either core HESN, mission, or OU). Notes: HESN online survey. Some HESN Development Labs are more represented because they had more survey respondents than other HESN Development Labs. Figure V.7. Summary of implementation facilitators and barriers 1. The HESN Development Labs faced a number of internal and external structural barriers, including the university bureaucracy, university student and staff turnover, and learning to work with USAID. 2. Several HESN Development Labs also struggled to set up efficient systems with their partners (for example, a system for transferring funds), which slowed the work completed under HESN. 3. HESN Development Labs, USAID Washington, and MBIOs sometimes struggled to find common ground and agree on scope, the content of products, and timeliness. 4. Communication among all stakeholders proved to be both a barrier and a facilitator to implementation. 5. Implementation facilitators included (1) the vital role of the AORs, who helped HESN Development Labs navigate USAID’s systems; (2) the ability to work directly with in-country universities, researchers, and other partners, thus increasing the relevance of HESN’s work; (3) USAID’s willingness and flexibility within HESN’s cooperative agreement structure that allowed HESN Development Labs to pivot and find new niches; and (4) the ability of HESN Development Labs to collaborate. B. IMPLEMENTATION FACILITATORS AND BARRIERS This section outlines the structural, institutional, and process elements of the partnership that contributed to different levels of usefulness in MBIO programming and decision making. Structural barriers are factors in the enabling environment that impede stakeholders from effectively implementing programs. These barriers are rooted in a lack of agreement among stakeholders on how to proceed, inadequate financial or human resources, lack of clarity about roles and responsibilities, and lack of experience in contractual matters. In contrast, process elements of programs include communication, planning timelines, and leadership changes. USAID.GOV PERFORMANCE EVALUATION OF HESN | 22 Structural and institutional elements (Research Question 3) Internal university structures created challenges for HESN Development Labs during the startup phase. Several HESN Development Labs noted that their university systems' bureaucracy often slowed their ability to set up partnerships and secure approvals for work, travel, and staff. Several universities lacked experience working with USAID, and they needed time to understand their obligations under the cooperative agreements. The HESN Development Labs often lacked sufficient staff to handle all the associated administrative and reporting requirements. One HESN Development Lab noted, “Being a state university never helps, because there are tons of state regulations and procurements rules, and so on…contractual challenges.” Staff at the HESN Development Labs used two key approaches to overcome these challenges. First, they worked closely with their AORs to learn the USAID process (including the M&E reporting process). The AORs played a critical role in facilitating the learning, but they also helped the labs navigate the MBIO and the buy-in process. One HESN staff noted, “Having a good AOR was the linchpin between designing work and engaging with the agency directly to help bring work to light….It is what made us have successful outputs. I cannot overstate the importance and value of good [AORs]. [They] helped us navigate through the agency, and helped us to identify right people to talk to about the work we wanted to do.” Within the university, overcoming internal barriers included getting support from executive management at the university by helping them understand the value added that HESN brought to faculty, students, and research centers. The HESN Development Labs faced several external barriers with partners, which reduced their ability to complete activities efficiently. The challenges ranged from difficulties in transferring funds to partner organizations and grantees to working with local communities. For example, RAN noted that one of its most significant challenges involved the mobilization of 20 African universities. Each university followed its policies and business practices. RAN had to figure out how to adapt to the universities’ various contractual requirements while honoring its contractual obligations to USAID. One way RAN helped the other universities learn was by training staff from each university on how to understand and follow USAID’s agreement and reporting requirements. One RAN staff person noted, “Because funding was from Washington, [and we were all] reporting to Washington, we had to bring these people on board—to show them how not to do anything contrary to mission policy and activities." RAN also faced structural challenges when it transitioned the HESN Development Lab’s authority to the West Africa RILab. Administrative requirements such as hiring staff, setting up the RILab, and delineating clear roles and responsibilities slowed the transition process. The RILab also had to submit documents to RAN and faced delays in document transmission and USAID approval. Finally, RAN also confronted funding delays in the sub-award process; thus, the West Africa RiLab lacked the cash to pay employees while it awaited RAN’s disbursement of funds. The transition took nearly a year to complete, but RAN staff and the AORs continued to work together and focus on solving each problem to get the transition completed. HESN Development Labs also experienced challenges in shifting funding to the local level, including grant funding to local innovators. The difficulties that the HESN Development Labs faced were related to their procurement and payment systems. Because many partners had never worked with USAID, they struggled to understand the agency’s requirements and needed significant time to navigate the bureaucracy. The structural barriers related to navigating rules and regulations across borders and organizations, setting up payment systems, and understanding and complying with requirements is a common challenge facing any entity working internationally. The qualitative results of this evaluation highlight three main strategies that helped the HESN Development Labs navigate these challenges. First, the labs ensured that they had staff who understood the requirements and were able to train any new partners or organizations on the requirements. Second, they engaged their AORs to problem-solve as soon as issues came up, because the AORs had the most experience navigating the USAID systems. Finally, the labs USAID.GOV PERFORMANCE EVALUATION OF HESN | 23 focused on building trust with the MBIOs as well as local partners. By doing so, staff were able to engage and communicate more clearly to solve these structural challenges. The HESN Development Labs faced in-country barriers that also delayed work. We present several examples in the text below. ● Uganda lacked a structure for testing innovations and obtaining permissions for medical devices. Although Makerere University had an institutional review board, it faced an unclear permission process from the government for conducting research on human subjects. ● Various internal barriers impeded innovators’ efforts to move their ideas into the marketplace and then overseas, which limited the impact of the innovations. In particular, the RootIO radio stations project devoted about a year to obtaining licenses from the communication regulators in Uganda for small FM radio stations. However, the licenses soon expired and necessitated a repeat of the entire process. ● When distinct firms or organizations develop aid management platforms for USAID missions and local government agencies, they can use non-compatible systems or compete for winning contracts. In several cases, these in-country hurdles meant that HESN Development Labs had to negotiate with local firms that had previously developed the platforms and needed to provide access or additional technical effort to enable the updated aid management platform. In Ghana, challenges relating to this negotiation stalled the final integration of the aid management platform, and to date, the Ministry of Finance is without the platform it was seeking through HESN. Although many of these challenges were not necessarily solved, they point to two important elements that may help future research programs move past them. First, they highlight the importance of understanding the political economy of stakeholder relationships in different countries and the role that incentives play in creating structural barriers. For example, stakeholder analyses conducted ahead of activity development might have highlighted the challenges RootIO would face with licenses. Understanding the process and incentives would have allowed the HESN Development Lab and its partners to plan for the delays and strategize ahead of time on how to overcome them. Similarly, understanding the compatibility of platform systems—and the incentives or disincentives to cooperate—could have helped in the negotiation process in Ghana. The second critical element is researching the local context before launching activities. The current USAID-funded LASER program, which also seeks to stimulate research for mission needs, is using a process called Research for Development, which involves holding workshops to identify local evidence gaps and potential hurdles to activities. This process is an example of how implementers and MBIOs can work together to prevent—or create action plans to mitigate—the types of structural barriers described above. USAID staff noted one additional barrier related to the structure of the HESN Development Labs. Even though several such labs were able to work across sectors and adapt to meet the client's needs, others, such as ConDev, were sector-specific and had less flexibility to change direction. The sector-specific focus proved to be a challenge for several HESN Development Labs, and it was more pronounced when HESN was positioned in the Center for Development Innovation. The focus on innovative approaches in the early years of HESN allowed the labs based in a cross-sector approach to pivot easily compared to the sector-based labs. About half the staff we interviewed from MBIOs identified problems with the scope, quality, and timeliness of HESN Development Labs' outputs. These stakeholders noted that sometimes the statement of work was misaligned to mission needs; at other times, the quality and utility of products was subpar. For example, one USAID OU struggled to communicate the desired scope of work to the HESN Development Lab they worked with through a buy-in. Staff from the OU felt that the lab wanted to create a “giant learning experience,” but the USAID unit itself had to keep insisting they just wanted a simple tool for internal use with their grantees. The OU staff felt that the lab “wanted to use the [buy-in] money to do what they wanted to get out of it.” PERFORMANCE EVALUATION OF HESN | 24 “Slow building of relationships takes years, and when mission people move, you have to start over.” —Lab Staff “Each lab had its own issues and growing pains that popped up. It really was about building a common understanding with them. What do people expect? What do people need? How do people need it? What is allowed, what is possible, how can we be allowed to do it?” —USAID Staff “There was a different prioritization of working with universities, value of scientific research in development when leadership changed. The change also brought a more challenging fiscal situation and challenges in having people see the value of what we do. However, we rode the ebbs and flows, but it was difficult due to the lack of resources.” —HEI Leader “What was letting us down in Uganda was that regulators were taking so long to grant licenses, which slowed some of the innovations.” —Lab staff “We were taking our subawards from Kampala. The release of funds from Kampala was never regular—we were frustrated every time.” —Innovator “Having a good AOR be the linchpin between designing work and engaging with the agency directly to help bring work to light is what made us have successful outputs. I cannot overstate the importance and value of good AORs. They helped us navigate through the agency to identify the right people to talk with. Both when wanting to pursue something and when something came to them that our lab might be interested in.” —Lab leader “Innovation is a process, and these sometimes have failures, so there needs to be a flexible environment. This is not always available in the development world, because the money being used needs to show impact and change. This makes it difficult, as working with innovations. doesn’t always show success initially.” —Lab staff HESN voices Five mission staff indicated that they had limited use for some of the outputs produced by HESN Development Labs. For example, two missions felt that some HESN Development Lab reports were too technical and featured difficult-to-interpret graphics and language. The missions communicated this feedback to HESN Development Labs, but research teams continued to struggle with adapting report content to less-technical audiences. Some USAID staff interviewed indicated that HESN Development Labs would benefit from having in-house staff with experience in data visualization and the translation of technical research. Another mission thought that one of the U.S.-based HESN Development Lab’s evaluation design did not align with the program’s theory of change. Some of the outcomes measured by the evaluation aligned more with the high-level theory of change for the program instead of the specific outcomes that the program could directly affect. Although the high-level theory of change prepared for each USAID-funded programs should inform evaluation designs, HESN Development Labs need to have a good understanding of program components and expected outcomes. They also believed that the evaluation outputs demonstrated a lack of understanding of the country's context, undercutting the ability of the mission to interpret findings and present recommendations. Accordingly, mission staff put the findings of that evaluation to limited use. Another mission faced challenges with the quality of an assessment report produced by a HESN Development Lab. The mission had to explain to the HESN Development Lab how to organize the report, particularly as the first draft lacked a section on “capacities and areas of opportunities” that the scope of work requested that the lab develop. Mission staff also thought that the HESN Development Lab’s methodology was not technically sound; therefore, they were reluctant to believe that the findings were accurate. Two missions also noted that HESN Development Labs experienced delays in reporting final outputs, which affected the missions’ ability to use the results to guide time-sensitive decision making. In these instances, the mission staff interviewed indicated that they used draft outputs to inform their decision-making but expressed that they would have preferred to have access to the final output. Interviewees explained that competing work demands faced by university-based researchers explained most of these delays. Staff turnover in the MBIOs limited HESN Development Labs’ ability to work with missions. With staffing shifts, some MBIO employees did not see the value of what HESN was trying to accomplish. HESN advocates within USAID worked to support and help the HESN Development Labs. Still, the institutional circumstances (political will, staff stretched thin) after the program’s first year proved extremely challenging. A change in the USAID senior team often meant a shift in goals and objectives, leaving HESN Development Labs to restart discussions with the MBIO. In some cases, the new mission staff rejected ideas that the HESN Development Labs has spent months discussing with the previous senior-level staff. Although working with MBIOs to identify local needs and gaps can help overcome this obstacle, it is an ongoing challenge inherent in the structure of any organization that experiences staff turnover. USAID.GOV PERFORMANCE EVALUATION OF HESN | 25 The HESN Development Labs differed in their levels of entrepreneurialism and self-advocacy. Several interviewees noted that not all HESN Development Labs exhibited the same level of entrepreneurism and self-advocacy. Several labs explored ways to engage with USAID and support the agency’s work. However, several labs were either constrained by funding limits or could not expand beyond their sectors. One interviewee noted that “some HESN Development Labs were also just better at marketing themselves than others.” The AORs wanted to ensure that all the HESN Development Labs participated in core- and buy-in-funded activities, but the constraints limited several labs from acquiring buy-in activities. Several interviewees from the HESN Development Labs noted that the funding constraints in the latter years and these other structural constraints led to a perceived level of competition among the HESN Development Labs, which affected their willingness to collaborate. Although HESN Development Labs faced this series of challenges, the facilitating factors that pushed them towards success included (1) the vital role of the AORs who helped HESN Development Labs negotiate USAID’s systems; (2) the ability to work directly with in-country universities and researchers, thus increasing the relevance of HESN’s work; (3) USAID’s willingness and flexibility within HESN’s cooperative agreement structure that allowed HESN Development Labs to pivot and find new niches; (4) the ability of HESN Development Labs to collaborate within their consortia and in local communities; and (5) HESN Development Lab staff and MBIO focus on clear communication and trust￾building among stakeholders that placed a focus on problem-solving. Process Elements (Research Question 4) Differences in planning and internal timelines between missions and HESN-funded activities led to challenges in commissioning and completing work. Stakeholders from all groups consistently mentioned the importance of effective communication and noted two main barriers to timelines and communication. First, missions often required fast turnaround times for short, easily digestible products, but the universities were accustomed to producing longer reports. The misaligned expectations led to a lot of formative feedback between the HESN Development Labs and missions to agree on the content and length of deliverables. The second challenge related to USAID engagement in the review and feedback process, which also consumed considerable time across several review rounds. For example, one HESN Development Lab took nearly five months (planned for one month) to finalize a scope of work because of ongoing rounds of mission input. The reviews focused on the alignment of content and methodology. One HESN Development Lab staff member noted, “The USAID standards and approaches and methodologies were tough to learn, but are now useful to know.” The HESN Development Labs also noted similar challenges with core-funded reports, indicating that they had often worked with other clients who provided little or no feedback on deliverables. The HESN Development Labs were often surprised by the ongoing USAID engagement in activities and the review process. Missions and HESN Development Labs were able to resolve many of these challenges as HESN activities progressed, and it became easier to collaborate. HESN Development Labs occasionally struggled to communicate findings to USAID in practical ways for immediate use. HESN Development Labs had to learn how to convey conclusions in a practical rather than an academic manner. Missions sometimes found that studies or evaluations lacked conclusions or recommendations for immediate implementation at the country level. For example, one mission felt that a HESN Development Lab's evaluation report focused more on theoretical impact measures and correlations and contributed more to the academic literature than to immediate program improvement. The HESN Development Labs differed in their ability to understand and respond to what MBIOs needed for the evaluation results to become practical, relevant, and applicable recommendations. The most fruitful relationships between MBIOs and HESN Development Labs were a product of the labs’ careful absorption of MBIO research goals and the purpose of the evidence as a contribution to MBIO programming. For example, among those HESN Development Labs conducting evaluations for missions, those that identified and adhered to specific mission evidence goals were able to produce action-oriented resources from their findings. In contrast, labs that brought a more purely academic perspective to their USAID.GOV PERFORMANCE EVALUATION OF HESN | 26 research findings produced theoretical reports that tended to disappoint missions, who needed immediate guidance for decision making. M&E indicators changed significantly, such that HESN Development Lab staff had to relearn reporting systems. At least one staff member at each HESN Development Lab noted the challenges of understanding USAID’s reporting system, the required indicators, and the reporting system itself. HESN Development Labs explained that they had to adjust their indicators about once a year to respond areas where the focus had shifted. Initially, the emphasis was on collecting extensive information about activity outputs. Later, the it shifted more toward reporting information about specific innovations and their scale. The focus on scale sometimes exerted pressure on HESN Development Lab staff, pulling them away from the innovation activities. Several HESN Development Labs noted a lack of clear communication about long-term strategies from USAID. HESN Development Labs struggled with the uncertainty about whether there would be a HESN 2.0. It was a challenge to them because they sometimes felt pushed by USAID HESN management to do more long-term work, but they saw it as a risk since they did not know whether long￾term funding support would be there for them. Some HESN Development Lab staff indicated that USAID did not always seem to operate with an overall goal or objective in mind, but instead wanted labs to “work magic.” Some interviewees also noted that USAID missions would occasionally make sudden buy-in requests in narrow domains where HESN Development Labs had advertised little experience. The requests left the HESN Development Lab staff unable to accept the buy-in or scrambling to find adequate technical expertise. For example, one mission requested that GCFSI conduct a supply-chain activity that the HESN Development Lab could not complete without pulling in faculty from other departments, which required additional funding. The lack of a clear long-term strategy for HESN from the USAID HESN team and MBIOs made it difficult for some HESN Development Labs to know how to plan and direct their activities, sometimes limiting their usefulness. C. RESULTS AND OUTCOMES This section presents the results and outcomes related to Research Questions 1, 2, and 5. It focuses on the use and perceived utility of HESN activities and outputs by MBIO staff and partners. It also addresses changes to HEIs and the longer-term sustainability of HESN activities. “It took much more time to complete the reporting than initially planned. I felt like I needed to hire more staff and get additional money to do that [reporting], and it wasn't something that I would recommend for assistant professors to get involved in. It is too demanding.” —HESN Development Lab Staff USAID.GOV PERFORMANCE EVALUATION OF HESN | 27 Figure V.8. Summary of results and outcomes 1. The majority of HESN outputs were useful to USAID and in-country partners. The core-funded work tended to be the most useful to the broader international development community, as these products tended to act like “public goods” (for example, data sets). Buy-in-funded products were the most useful to MBIOs, because they supported their strategic planning processes and in￾country objectives. The joint funding modality (core plus buy-in) served the broadest audience and allowed MBIOs to build on core-funded activities, adapting products for their own use. Because of the high number of beneficiaries served, these types of products tended to be the most cost efficient. 2. Missions and operating units used outputs to increase learning, support work with partners, and educate stakeholders. 3. HESN activities allowed the labs and MBIOs to engage local policymakers in using results and outputs for decision making. 4. HESN brought together a diverse group of stakeholders and demonstrated various ways that they could work together. HEIs involved with HESN improved their ability to work with the donor community; policymakers used HESN outputs to address local development challenges; innovators developed and implement their ideas with HESN support; and students gained fieldwork experience while working on HESN activities. 5. HESN led to the development of more than 96 different courses, disciplines, and programs. The HESN Development Labs also developed hundreds of data sets, research reports, and innovations over the five-year period. The utility of HESN for USAID staff (Research Question 1) A large share of HESN stakeholders indicated that HESN outputs were useful to the MBIOs. Over half the survey respondents agreed that the HESN Development Labs successfully delivered outputs when USAID needed information (Figure V.9). USAID staff (including those in MBIOs) also generally agreed that HESN generated opportunities for USAID to broaden the evidence base used for internal planning purposes and that HESN was relevant for planning future programs. Finally, respondents noted that they used HESN products as inputs for their CDCS or program design. USAID.GOV PERFORMANCE EVALUATION OF HESN | 28 Figure V.9. Perceptions and use of HESN outputs by USAID MBIOs Notes: HESN online survey. The numbers in parentheses are the total number of survey respondents. a Respondents include 21 USAID staff and 46 HESN Development Lab staff. b Respondents include 24 USAID staff. c Respondents include 23 USAID staff and 46 HESN Development Lab staff. d Respondents include 17 USAID staff and 46 HESN Development Lab staff. USAID MBIOs used HESN outputs for planning, programming, and decision making. Of the seven missions with representatives interviewed for this evaluation, representatives of two indicated that HESN outputs directly contributed to the development of the CDCS. USAID/Uganda and its partnership with CITE highlights an example of how HESN research contributed to mission planning and programming. The Uganda mission has been working with CITE since 2016 to map and understand the country’s agriculture market system. Through this ongoing collaborative partnership, CITE worked with the mission to identify how best to respond to its needs. As the mission’s priorities have evolved, CITE evolved to remain in sync. The mission used the information and knowledge documented in CITE’s systems mapping and reports produced as part of the Market System Monitoring activity to write the Feed the Future country plan, which runs from 2017 and 2022, and to design project activities (Figure V.10 and additional detail in Annex C). USAID/Guatemala is another example of where one of the mission’s priorities was to improve or facilitate how USAID works on digital financial tools. The mission partnered with USAID’s Center for Digital Development (CDD), which provided CITE with funding to assess digital financial service opportunities. The assessment then guided the mission’s new CDCS and the Feed the Future digital app. USAID.GOV PERFORMANCE EVALUATION OF HESN | 29 Figure V.10. CITE (with partner George Washington University) Feed the Future Market System Monitoring activity with USAID/Uganda • Objective: Mapping and surveys to understand Uganda’s agriculture market system • Funding sources: USAID/Uganda buy-in • Uptake: Key input for Uganda’s Feed the Future country plan. Shifted how USAID/Uganda thinks about monitoring, evaluation, and learning The Uganda activity is an example of a mission’s use of outputs to guide country planning. The mission buy-in involved systems mapping and surveys to increase the understanding of Uganda’s agriculture market system. CITE made esoteric, hard-to-understand content real and usable by practitioners. In addition, as the mission’s priorities evolved, CITE evolved in parallel. The mission used the information and knowledge that came out of the produced maps and reports to write the Feed the Future country plan and to design projects and activities. In addition, the mission changed its way of thinking about monitoring, evaluation, and learning based on the activity toolkit, which emphasized identifying, measuring, and interpreting systems indicators (see additional detail in Annex C). The relevance and utility of HESN activities often led to follow-on work with several MBIOs. USAID’s Scaling Off￾Grid Energy (SOGE) team allocated some funds to study the sustainability of battery technologies. After hearing about HESN, the team decided to conduct the work through HESN and partnered with RAN. The findings from RAN’s SOGE research effort spun off a series of initiatives, including the design and implementation of the Solar E-Waste Challenge awards program. USAID’s Africa Bureau identified ConDev to work on a study focused on perceptions of school safety. After ConDev completed the paper for the first study, the Bureau decided to work with ConDev on a second study focused on bullying and school safety. Core-funded work was more exploratory and conducive to the development of public goods in that it provided HESN Development Labs with the latitude they needed to identify specific interests and niche areas. The usefulness of the core-funded work is based on four characteristics: (1) whether a product achieved its intended purpose; (2) whether the supply chain permitted integration of the product into the chain, giving consumers easier access to the product; (3) whether stakeholders used the product correctly and continually; and (4) whether stakeholders could scale the product. For two reasons, MBIOs were less interested in core-funded activities outside their CDCS plan: (1) a lack of alignment to the country strategy, or (2) the unavailability of a required buy-in funding to complement the existing core funding. In these cases, HESN Development Labs opted to carry out core-funded work with partners such as local governments, NGOs, research centers, or other organizations. Another reason that some MBIOs had lower levels of interest in HESN activities was related to the benefits of HESN products. Some core￾funded work did not result in an immediate benefit to an MBIO. For example, collecting the geocoded data was supported by core funds and remains available as public goods for future research. However, unless an MBIO invested in adapting that data to its own country, it was less immediately useful for their needs. In Table 1 in Annex D we highlight a selected number of activities that the HESN Development Labs completed over the past five years. The table shows the type of deliverable and whether it was core, buy-in, or hybrid modality. As the deliverables show, most core-funded activities focused on developing data sets or GIS portals, which continue to be available for public use. Examples of core-funded activities “It spun off a series of initiatives that we’ve been quite happy about, so I’d say it seeded the research that then informed design and execution of an award program.” —USAID Staff USAID.GOV PERFORMANCE EVALUATION OF HESN | 30 that had a high level of utility to missions included the IDIN IDDS (summits) and workshops, because the sessions developed the capacity of local innovators to create locally based solutions to community challenges, and the RAN Innovation portfolio, which also served the local communities. Core-funded work was also more conducive to the development of innovations because its flexibility and longer timeline allowed HESN Development Labs to foster innovations and move them through the various stages. For example, workshops helped innovators develop their ideas and create prototypes that were ready to be pilot tested in the field. Buy-ins offered support for some innovation ecosystem activities (for example, pilot testing), but only for a limited time—often for just a few months. Core- plus buy-in-funded (hybrid) activities tended to provide MBIOs with opportunities for follow￾on activities. These activities had longer time frames and often allowed missions to take an activity conducted under core-funding and expand it to the local country served by a USAID mission. For example, AidData developed its geocoding methodology for aid management platforms with core funding. Then, USAID missions such as Ghana and Nepal bought in to the service to make the data relevant to their context for decision makers (Figure V.11). DIL’s Gridwatch activity is another example of how the hybrid-funding model helped meet public use and country-specific needs. Gridwatch provides an expansive, crowd-sourced way for local communities to alert power companies about outages in their area. The University of Michigan and DIL formed a partnership to develop the activity and then adapted it locally through buy-in activities. The USAID/Niger mission funded the Niger Participatory responsive Governance Principal Activity Impact Evaluation under the hybrid model. The results supported the USAID mission in Niger in their design of a follow-on project. Figure V.11. AidData's geocoded aid management platforms • Objective: Provide government ministries with access to geocoded local development data for use in decision making • Funding sources: HESN core and buy-in funding from missions • Uptake: The use of geocoded aid management platforms by policymakers varied across countries AidData worked with several missions on aid management platforms to permit target ministries to make geocoded local development data available for decision making. Platforms seemed to work well when (1) partners worked hand in hand with mission staff, and (2) host countries found the platforms useful and assumed ownership of them. The level of ownership provided some of the initial impetus for adapting the platform, allowing mission staff to import data into their existing systems or processes easily. For example, the Government of Nepal took ownership of the geocoding process, adopted the associated data, and ensured process sustainability. In Ghana, the aid management platform aimed to strengthen the Ministry of Finance’s capacity to analyze data on development programs, work more closely with local governments, and improve planning and accountability in development. The Ministry saw the value and promise in the geocoded data system that AidData and its partner, Development Gateway, had produced. However, because of a misalignment of data platforms, the Ministry never successfully integrated the geocoded data system with its existing databases (developed by a previous contractor). The fully geocoded platform is currently incomplete in the absence of (1) the additional funding needed to bridge the two systems, and (2) contractual requirements that would obligate the previous contractor to support the integration. PERFORMANCE EVALUATION OF HESN | 31 “Really have a map for where we’re going and have information to justify why. Used it to design our activities, and now one of our activities is running with it and using it to guide their work planning.” “The first study got a huge amount of attention within the mission and the outside, and support from the Hill as well. The second study is coming out shortly and I assume will get some attention as well. Even with the first study, there has been a huge upsurge in interest in cash throughout the agency.” "This was an important secondary aspect of our work just to ensure that the technology we are deploying or recommending to country government is sustainable and just has that long-term trajectory in mind.” “I recently put together a quick dashboard based on the data for the mission director, who developed a few talking points based on that. Those are the types of outputs that can have an influence on decision making and policy. The system does not lack potential—it lacks iteration.” “I can say it was a very great report that they provided at the end of the evaluation. As you know, the mission has a large portfolio but not always the resources necessary to implement all the conclusions provided from this kind of evaluation.” “We have been developing the new CDCS and have been getting as much information as possible, as much data as we can to inform the new strategy. The assessment from them informed the country strategy.” “I think findings will be able to guide programming design at the country level: both USAID work and work with the government and the Ministry of Education. Mission would be able to go to the Ministry and say, “Hey, look at these findings you really need to address.” "I think the data would be useful for all our offices because they touch on several issues." USAID Voices on HESN Use and Utility Buy-ins tended to be specific to MBIO needs. Buy-ins led to a higher level of engagement with missions by enabling the HESN Development Labs to co￾create activities. They provided opportunities for MBIOs to make a direct financial investment in activities to be completed by the HESN Development Labs and to be overseen by an activity manager based at the mission. The HESN Development Labs indicated that buy-ins were most useful when missions participated in the early stages of the design process, thereby promoting immediate uptake. Buy-ins were particularly successful when missions carefully weighed the proposed research objectives and possible research methods. Buy-in outputs included assessments, study or evaluation reports, mappings, and data platforms and tools. For example, USAID’s CDD worked with RAN and CITE on several buy-ins that had specific purposes. In its work with RAN, CDD had amassed knowledge about digital IDs in development but needed to build on it with field research. RAN demonstrated experience working with digital ID development, so CDD partnered with RAN to conduct academically rigorous field research. Under the machine-learning activity, CDD needed to tap into the technical expertise that it lacked in-house. With technical experts and prominence in the machine-learning field, CITE offered CDD a strong partnership that enabled the Center to draw attention to machine learning’s potential in development. The HESN Development Lab provided the “just in time” support that was relevant and timely for the mission. Missions used HESN outputs to guide other M&E or programming needs. In Colombia, AidData and Development Gateway set up the Monitor platform that helped the mission decide how to process multilevel information for its M&E system and create new data-processing rules. In Rwanda, the mission used DIL’s cash-benchmarking study to compare the outcomes of household grants equal to the cost of providing a major multi-sector child nutrition program. Study findings influenced USAID/Morocco’s and USAID/Uganda’s decisions to fund these types of programs in the future. USAID/Rwanda subsequently conducted a follow-on study. Missions in Malawi, Liberia, and the Democratic Republic of the Congo (DRC) also decided to conduct cash benchmarking studies (see Figure V.12 and Annex C). “There is more uptake in buy￾ins [than core], because there is a user who is inherently involved in the conception of the research. It is more useful than someone putting research out into the world and just asking, “Here, is anyone interested in this?” —USAID staff USAID.GOV PERFORMANCE EVALUATION OF HESN | 32 Figure V.12. DIL cash-benchmarking study in Rwanda • Objective: Benchmark a major child malnutrition program against what would have occurred if program costs had instead been disbursed as household grants directly to beneficiaries • Funding sources: USAID/Development Innovation Ventures (DIV) for the research, USAID/Rwanda, and Google.org for the intervention. • Uptake: Informed decision making by at least two missions; additional studies in Malawi, Liberia, and the DRC; influenced the formation of a costing working group within USAID/Bureau for Policy, Planning, and Learning (PPL) DIL's cash-benchmarking study in Rwanda is an example of how several missions and OUs applied research findings in their program deliberations. DIL researchers designed the study to benchmark a major child malnutrition program against what would have occurred had the program's funds been distributed directly to beneficiaries. The results show that household grants and in-kind support each affected different outcomes. The implication is that various means of expending program resources generate different types of benefits. An increased understanding of the results that cash transfers can and cannot deliver can improve USAID’s and implementers' ability to identify the best means of providing resources and cost￾effectively spreading them across target beneficiaries (McIntosh and Zeitlin 2018). The study also showcased the importance of thinking about the cost per dollar of various programs through cost analysis. The study drew considerable attention and influenced the decisions of USAID/Morocco and USAID/Uganda to fund these programs. The study also led to additional cash-benchmarking studies in Malawi, Liberia, and the DRC. Finally, the study established a costing working group by the USAID/PPL (see further detail in Annex C). Missions and operating units (OUs) also used outputs from buy-ins to increase learning, support work with partners, and educate stakeholders. After the mission in the DRC heard about HESN, it contacted the USAID HESN team when it needed to conduct an external evaluation of a partner public health school. The team matched the mission with Texas A&M University’s ConDev, whose evaluation of the public health school helped the mission learn how the public health sector's support could assist the school in fulfilling its mandate as a research institution. The evaluation also helped identify what type of public health assistance could be most effective in the DRC. Even though the mission reported a highly positive experience with the HESN Development Lab, it lacked funds for additional buy-ins. In another instance, Duke University SEAD’s diagnostic tool helped USAID/DIV understand its grantees’ strengths and weaknesses while assisting future grantees in setting relevant milestones. The Africa Bureau also shared the lessons it learned from ConDev’s research on school safety with stakeholders to guide education policy. These examples highlight HESN support and could lead MBIOs to demand more research￾oriented activities that helped inform their work. “The goal was to give the grantees access to someone from the outside who could do an evaluation of where they were as an organization.” —USAID staff “We want to make a connection to show to sector, agency, other donors how important it is to be addressing issues of safety in order to improve learning outcomes.” —USAID staff USAID.GOV PERFORMANCE EVALUATION OF HESN | 33 The institutional structure of USAID and relationships between Washington and the field-based staff sometimes colored MBIO perceptions of HESN. Those staff who were reluctant to work with the HESN Development Labs viewed HESN as one of several other programs that USAID/Washington designs and oversees but that, in their opinion, demonstrates little to no alignment with mission priorities. They viewed HESN as less useful to their needs. According to several HESN stakeholders we interviewed in this evaluation, some mission staff believed that HESN was diverting resources away from the “real development work” and toward abstract work conducted by university researchers. They also said that mission staff sometimes felt that teams at USAID headquarters occasionally advised them— without field experience—on what development work should look like. Although this comment was not specific to HESN, the experiences may have contributed to mission's perceptions that the program was not useful. Finally, staff from two missions said that HESN added to their already busy workload. Staff from these missions indicated that, because of all their management and reporting responsibilities, they had limited time to dedicate to HESN. The utility of HESN for policymakers and implementers (Research Question 2) Nearly half of surveyed stakeholders involved with HESN outputs targeting policymakers agreed that HESN presented USAID with more opportunities to engage policymakers in using research evidence. Among survey respondents, 47 percent agreed that HESN generated opportunities for USAID to engage with developing-country policymakers to use evidence to craft new policies, laws, or regulations, and 45 percent agreed that they used the outputs for planning and policymaking purposes (Figure V.13). Similarly, 40 percent of interviewees said that policymakers used the outputs to guide their work. According to interviews, policymakers such as staff of government ministries were more likely to use the outputs if specific government stakeholders participated in discussions about the end product and their subsequent role in the activity. Policymakers were also more likely to support innovations that aligned with their policy agenda. Several of the HESN outputs discussed by interviewees were at too early a stage to have produced a policy impact. For example, some HESN projects were still conducting data analyses in 2019-2020 and had not yet disseminated any findings. HESN Development Labs and partners were also still preparing to present and discuss results with policymakers as late as 2020. Note that policymakers were not always the primary audience for HESN outputs, so the perceived utility of buy￾ins to policymaking should be interpreted cautiously. Figure V.13. Perceptions and use of HESN outputs by USAID MBIOs for policymaking Source: HESN online survey. Notes: Numbers in parentheses are the total number of survey respondents. The total number excludes respondents who indicated that the statement did not apply to their HESN work. a Respondents include 15 USAID staff. b Respondents include 20 USAID staff and 45 HESN Development Lab staff. USAID.GOV PERFORMANCE EVALUATION OF HESN | 34 “We learned with CITE and D￾Lab that you have to go to the community, understand them, work with them, and then give feedback. We’ve always been doing projects, but we didn’t use all our potential. CITE made us understand the complete panorama of the project. CITE has taught us to ensure sustainability of project.” —HEI partner “We are not a research organization ourselves; we are a practitioner. What they brought to the equation is (1) understanding how best to get valid data, good data, and make it significantly meaningful; and (2) designing the research so it resulted in that.” —NGO partner Application of HESN “The solar irrigation pump was one of the flagship projects in which government had a lot of interest and provided $1 to $2 million.” —Lab staff HESN Development Lab research findings and the scale-up of innovations helped policymakers contribute to local development. Policymakers supported several RAN results and innovations that addressed specific development challenges that national or local governments deemed high priority. For example, the local prime minister’s office in Uganda adopted the findings from RAN’s deliberative polling activity on how to structure responses to landslides and floods. An irrigation pump developed by one of RAN’s innovators garnered considerable interest from the Ugandan government, which has now provided nearly $2 million in funding to produce the system and distribute it to farmers. The Ugandan government also adopted a platform that combines microfinance offerings, which permits agricultural producers to obtain financing from other locations when it is not available nearby. In another example, the Rwandan government bought 750 PedalTap units—handwashing gadgets that prevent people from touching the tap when washing their hands—for installation in health centers. Finally, CITE’s research on the packaging of food aid is on track to achieve policy changes. CITE studied options for packaging food aid to protect it from insect infestation. The researchers are now working to bring the results to decision makers with jurisdiction over packaging policies. Implementers and policymakers used findings and methodologies developed by HESN Development Labs to guide their work. CITE’s evaluation report helped the NGO Solar Sister understand its effect on the energy sector and development at large. It also enabled them to communicate the importance of female entrepreneurship to donors and stakeholders. The World Food Program uses the findings from CITE’s work with USAID/Uganda on post-harvest losses to guide related programming. In another example, ConDev designed a methodology for estimating the prevalence of school-related gender-based violence under a buy-in with the Bureau of Africa. The United Nations Educational, Scientific, and Cultural Organization and their Girl Education Initiative are now using the methodology to inform programming. USAID’s Program and Policy Change Scorecard (PPC) shows that HESN Development Labs were influential to policy and international development practice. USAID formulated a two-factor index scoring system to assess research innovation effects on policy and program planning. The agency developed the PPC toward the end of the HESN core-funding, so it was not used to evaluate the HESN Development Labs. However, the policy and change scorecard provides a useful window into the scale and degree of each HESN Development Lab’s effect on policy and programming. USAID.GOV PERFORMANCE EVALUATION OF HESN | 35 In Figure V.14, we present the program and policy change results for the four target HESN Development Labs. The blue bar shows each lab’s average score for the institutional influence of their research and innovation outputs—ranging from 0 (no influence) to 3 (influence on national or multinational policy). The red bar shows each HESN Development Lab’s average implementation status score related to research and innovation accomplishments. The gray bar presents the combined average of the two scores.16 For example, AidData had institutional influence on a larger geographic scale than the other HESN Development Labs because nearly all its work was national or multinational and easily lent itself to government and stakeholder implementation (that is, geospatial analysis). Other HESN Development Labs achieved higher implementation status scores. Figure V.14. HESN Development Lab average results on USAID program and policy change scorecard Source: HESN M&E data. HESN influence on partnerships in the broader international community. Partners such as local government agencies, NGOs, the private sector, and other universities helped HESN Development Labs achieve their goals. In Figures V.15 and V.16, we present the number of partners and “partner touches”17 by the HESN Development Lab and the global distribution of partnerships. In Figure V.15, we demonstrate that the number of partners and partner touches varied by HESN Development Lab. DIL accounted for the highest number of partners (371) and partner touches (618) as compared to SEAD (43 partners, 390 contacts) and AidData (107 partners, 289 partner touches). The large number of touches completed by DIL is related to their focus on outreach with NGOs, commercial enterprises, other HEIs, and government agencies. CONDEV and GCFSI had a similar number of unique partners (167 and 138 respectively) and partner touches (391 and 380 respectively). 16 The scores range from 0 if the research or innovation remained just an idea, to 1 if it led to a plan; to 2 if it led to policy design; and to 3 if it directly guided policy implementation. 17 We define partner touches as individual instances of partner contact, such as a meeting with a target government agency. USAID.GOV PERFORMANCE EVALUATION OF HESN | 36 Figure V.15. HESN unique partners and partner touches Source: HESN M&E data. To understand the expansiveness of HESN’s global reach, we mapped the partner touches around the world. In Figure V.16, we show that the United States accounted for the highest number (996), an expected outcome because seven of eight HESN Development Labs were based in the United States and carried out partnership-building activities domestically. The next-highest engagement countries were Uganda with 293 partner touches and India with 236, where several HESN Development Labs conducted extensive research and innovation activities in both locations. HESN reached every continent, including Australia. Figure V.16. Instances of partner engagement, by country Source: HESN M&E data. USAID.GOV PERFORMANCE EVALUATION OF HESN | 37 Almost all survey respondents agreed that HESN activities provide a positive example of how stakeholders can work together in partnership. They also felt that HESN expanded opportunities to collaborate in the design and implementation of international development research. About 78 percent of respondents concluded that HESN increased the opportunities for participating with developing-country policymakers to exchange research findings. The partnership between USAID and the HESN Development Labs strengthened their capacity to work with USAID. HESN Development Labs and researchers agreed that greater access to USAID was one of the significant benefits of the HESN partnership. For example, one researcher explained that HESN expedited his team’s access to USAID staff, which made it easier to arrange meetings with a range of MBIOs. The access helped them develop a better understanding of the goals and objectives MBIOs, which transferred to the design of activities that were more relevant for USAID. The partnership also helped the university systems learn how to manage and meet USAID’s requirements, including completing all associated paperwork and reporting on M&E indicators. HESN Development Labs and the universities gained the experience and knowledge to compete for future USAID research opportunities. The increased capacity helped several of the institutions win additional projects and funding outside HESN funds. The HESN partnership enabled USAID to understand and capitalize on research, innovations, models, and methodologies that universities provide. HESN offered an opportunity for USAID to build on its earlier work with universities. Through HESN, USAID has been able to work with HESN Development Labs and their consortia of partners in various ways, including implementing scopes of work, collaborating to identify gaps in thinking, and applying new models and methodologies in international development work. Among the essential results missions identified was the chance to work with HESN Development Lab staff and researchers from reputable universities who offered innovative ideas and helped revise their thinking. Inter-HESN Development Lab collaborations on specific HESN activities existed but was limited. The USAID HESN team sought diligently to offer HESN Development Labs the opportunity to meet with each other and create a community. The team brought the labs together during annual TechCon events to help them get to know each other and explore synergies for collaboration on activities. For example, TechCon events helped RAN innovators learn about DIL’s Big Ideas competition. The experience helped RAN innovators win Big Ideas grants, and the innovators now help coordinate the large-scale running of the program, including mentorship of teams for Big Ideas. RAN and other HESN Development Labs have forged the most robust collaborations. RAN’s partnerships with, for example, AidData and IDIN helped RAN build its capacity and adapt some of its partners’ methodologies. Moreover, RAN’s vast network in Uganda and other African countries has made it easier for other HESN Development Labs to work in those countries. HESN facilitated the creation of partnerships between and among various actors, contributing to the implementation of research and innovation activities and the development of useful outputs. AidData engaged in collaborations with diverse actors and adapted its geocoding work to achieve more effective partnerships. Under one USAID buy-in, AidData worked with the Africa Bureau, the U.S. Centers for Disease Control and Prevention, and the NGO Together for Girls. These partners complemented one another and produced useful findings and outputs related to school violence for the Africa Bureau. “That is part of the power of partnerships: knowledge and technology transfer.” —HESN Development Lab staff “If USAID really cares about engagement with researchers, they must have some sort of mechanism like this that allows researchers to get access and work at some level.” —Researcher USAID.GOV PERFORMANCE EVALUATION OF HESN | 38 AidData analyzed data to which it otherwise would not have had access; Together for Girls provided sector and dissemination expertise. As another example, AidData’s consortium partner Development Gateway adapted its geocoding work to achieve more effective partnerships with government stakeholders. Development Gateway learned to reorder its approach so that it could offer geocoded data to in-country governments at the needed level of granularity while providing deeper geocoded data sets to university partners for further learning. CITE built the capacity of a variety of local NGOs to provide better products and services to marginalized and vulnerable populations. For example, CITE worked with organizations and universities to test water filters in Uganda and India. In India, CITE worked with Mercy Corps, the Self￾Employed Women’s Association in Ahmedabad, and other partners to evaluate solar water pumps to guide decisions on purchasing them. In Indonesia, CITE’s evaluation of wheelchairs determined their effect on targeted populations and guided service delivery and wheelchair design for organizations providing services to people with disabilities. Through its network of 20 universities, RAN exchanged expertise and produced outputs targeting specific resilience issues. At the start of its HESN work, RAN conducted a resilience assessment and then convened a range of experts from its regional HESN Development Labs to synthesize the findings, which guided the design and focus of RAN’s four regional Resilience Innovation HESN Development Labs and innovation activities. ResilientAfrica Network’s RILabs include (1) the Eastern Africa RILab based in Uganda and hosted by Makerere University, (2) the West Africa RILab based in Ghana and hosted by the University for Development Studies (UDS), (3) the Horn of Africa RILab based in Ethiopia and hosted by Jimma University, and (4) the Southern Africa RILab based in South Africa at the University of Pretoria. Activities and their related outputs have benefited from partnerships with universities, local government entities, and other local organizations (see innovation ecosystem findings below for more detail). RAN also partnered with other organizations to build their capacity. For example, they assisted the United Nations Development Programme in establishing a climate innovation HESN Development Lab in Nairobi. HESN funding also enabled IDIN’s consortium to reach innovators across various countries and build additional partnerships. IDIN developed partnerships with several different local actors, including universities and NGOs, through its IDDS events and innovation centers. Through these partnerships, IDIN worked toward formalizing a global network of innovators and design summits. “It was a nice consortium of higher education institutions, two different USG agencies, and this advocacy network NGO all working together, collaborating on this activity.” —USAID Staff “We started getting other people who said, ‘We want to work with you, because the work that you do resonates with what we want. We want to connect the university with community work that we do.’” —HESN Development Lab Staff “Our network is a quite amazing group of people, so when wanting to do work in a region, we have a network of over a thousand people who can participate in projects.” —HESN Development Lab Leadership USAID.GOV PERFORMANCE EVALUATION OF HESN | 39 HESN provided innovators with technical support and resources that helped strengthen the local innovation ecosystems. First, HESN funding allowed IDIN to introduce IDDS to local communities around the world. IDDS events bring together diverse groups of people interested in learning about the co-creative design process and how to prototype low-cost solutions that can improve livelihoods. HESN funding allowed IDIN to run several summits a year and tap the support of local organizing teams. IDIN planned IDDS events in collaboration with at least two or three local institutions. Summit participants included students and faculty from local universities, as well as representatives of local NGOs and local communities where the summits took place. Participants became part of a global network of other summit participants. After the summit, they often continued exchanging lessons learned; in some cases, they also partnered in developing innovations. After the summits, some participants also went on to form local innovation centers with the support of IDIN. Centers often facilitated training, offered in-kind support to local innovators, and helped organize and gather resources for annual design summits. In Figure V.17 (see additional detail in Annex C), we detail one of IDIN’s most successful innovation centers. Figure V.17. C-Innova in Colombia • Objective: Launch a local innovation center, support innovators, and collaborate with university faculty and students • Funding sources: HESN, academic institutions, grants • Outputs: Organized themed design summits with innovators and universities; supported communities of innovators after summit participation; adapted and delivered the D-Lab Creative Capacity Building curriculum C-Innova is one of several local innovation centers that started with IDIN’s support and now supports local innovators. Its founders were inspired to set up the center in Colombia after participating in one of IDIN’s International Development Design Summits. The summits convene diverse groups of people interested in learning about the co-creative design process and how to prototype low-cost solutions that can improve livelihoods. C-Innova now supports the innovator ecosystem in Colombia through various activities, including annual design summits organized in collaboration with local universities and follow-on support for summit participants. It also adapted and continues to deliver MIT D-Lab’s Creative Capacity Building curriculum (see additional detail in Annex C). HESN Development Labs also supported local innovators with funding. IDIN offered grants of varying amounts to help them prototype, test, and launch innovations. RAN organized calls for innovators focused on resilience topics such as conflict resolution and climate change. DIL also supported innovators with grant funding through the Big Ideas annual competition held at the University of California Berkeley. These local innovators significantly benefited from the in-kind support offered by HESN Development Labs. IDIN team members trained partners in the methodologies needed for conducting design summits and building creative capacity. Local partners applied the training to develop their ability to reach more innovators. RAN convened a centrally located main advisory board and innovation advisory boards in each of its four RILabs. RAN selected board members for their ability to identify partners who could help surmount some of the hurdles and challenges that characterize the innovation process. RAN also hosts “It’s because of this funding that the network has been so visible that governments in different countries have now started supporting the innovation agenda, building on what USAID under HESN has been able to provide.” —Lab staff USAID.GOV PERFORMANCE EVALUATION OF HESN | 40 Tuesday walk-in pitches, a weekly event during which innovators can pitch their ideas to RAN staff and receive advice. RAN supports these innovators by offering guidance from a multidisciplinary pool of experts and providing a physical space where the innovators work on their prototypes with other innovators. Figure V.18. RootiO radio innovation supported by RAN • Objective: Enable rural communities to own and operate communication technology • Funding sources: HESN core funding • Uptake: Set up 10 FM radio stations in rural communities across Uganda The RAN Innovator Acceleration Program provided a local innovator in Uganda with grant funding to launch the RootIO community radio. RootIO, a small, low-cost FM radio technology for rural communities, uses affordable solar power and a smartphone to replace the traditional radio studio. Each community radio station serves a village (or a couple of villages) and can reach 10,000 listeners. RootIO enables communities to disseminate information, to provide local advertising, and to host community discussions. RootIO obtained the first community license for FM radio stations in Uganda with assistance from RAN’s advisory board. The grant funding allowed the innovator to work with communities to set up RootIO and explore how to mobilize communities to own and use radio technology. In each community, the innovator worked with local leaders to obtain their buy-in, which was vital for acquiring a second round of licenses from Uganda’s communication regulator. Support from local leaders also helped promote the uptake of RootIO locally. In some communities, leadership used RootIO to provide updates and share discussions from the local council meetings. The innovator is launching RootIO in other countries, with five to seven radio stations commissioned in Cabo Verde in 2017. HESN influence on HEIs (Research Question 5) This section highlights the results of how HESN contributed to changes within HEI networks as well as increased their engagement in international development. A core aim of the HESN program was to build the capacity of universities, faculty, and students to engage in international development research and innovation. Quantitative and qualitative data collected for this evaluation suggest that the program made progress in achieving that goal. This subsection discusses the changes in HEI capacity, the changes in opportunities for faculty and staff, and the likelihood of sustainability of those changes beyond HESN. In Figure V.19, we provide an overview of stakeholder perceptions of HESN influences on HEIs. More than half of survey respondents agreed (or strongly agreed) with each of the statements in Figure V.19, which suggests that HESN advanced the engagement and capacity of HEIs, faculty, and students in international development research and innovation. In the following subsections, we present qualitative evidence from the HESN Development Labs to illustrate how stakeholders experienced changes in HEI capacity and student and faculty opportunities. USAID.GOV PERFORMANCE EVALUATION OF HESN | 41 Figure V.19. HESN impacts on HEIs: Percentage of respondents who agree or strongly agree with key statements Source: HESN online survey. Notes: The number of respondents to each statement is indicated in parentheses. All statements were posed to USAID staff, HESN Development Lab staff, researchers, and HEI leaders, except for statement 4, which included innovators, and statement 1, which included only USAID staff. HESN allowed universities to connect with local communities. HESN brought together communities and universities involved in international development research and innovation. For example, through RAN, HESN funding increased the capacity of the West Africa RILab and helped it engage with Response to Resilience, Columbia University, and Ghana Public Health. The relationships allowed UDS to conduct applied research directly with communities in Ghana. RAN also used deliberative polling to source knowledge on community challenges by assessing the local population’s views in three stages: (1) levels of resilience challenges, (2) priorities for interventions that would help address those challenges, and (3) an assessment of the success of those interventions. In Ghana, IDIN worked with its local partner, Kwame Nkrumah University of Science and Technology, to conduct Creative Capacity Building training for artisans, farmers, and other community members. Participants spend three to five days in training workshops to learn the design process, practice basic skills with hand tools, and create technologies to solve local challenges. CITE and IDIN drew on HESN funding to accelerate the development of a “lean research” model. Lean research offers academics interested in conducting evaluations in the developing world codified guidance for carrying out research. Lean research is right-sized to community contexts and demonstrates respect for beneficiaries in the field. For example, it involves providing information to achieve authentic consent, which entails adapting “Previously, the university was seen as the ivory tower, but now it is involved in the community . . . The community started to get to know [the scholars and students].” —HESN Development Lab staff USAID.GOV PERFORMANCE EVALUATION OF HESN | 42 consent-seeking procedures to accommodate low literacy study participants who may not understand written (or even verbal) descriptions of the study and its implications. HESN helped its labs increase collaboration across university departments. Researchers and HESN Development Lab staff cited instances in which they stepped out of their comfort zones to work collaboratively with people in other departments. For example, at MIT, the CITE team drew on the expertise of research centers and departments across the institute in fields such as public service, mechanical engineering, supply chain management, computer science, business, and environmental engineering. Interviewees noted the continued existence of silos at MIT but reported that CITE has helped link disparate disciplines. In Africa, staff from East Africa RILab noted that HESN support through RAN helped break down researcher and student silos by setting a new norm, demonstrating how faculty from one department can link up with faculty from another to solve innovation problems collaboratively. HESN Development Labs developed and shared innovative methodologies. CITE and IDIN produced, deployed, and shared several useful research methodologies. For example, they developed lean research methodologies and a corresponding field guide to improve evaluation right-sizing and researchers’ ability to obtain authentic consent from study participants. Using follow-on funding from MIT, CITE also trained staff from UTEC in Peru in CITE’s methodologies for application to a cookstove problem. Additional examples from MIT include an online EdX course for students worldwide to access and learn MIT’s methodologies and an expansion of the IDIN-led IDDS as part of the Creative Capacity Building training program. AidData developed a geospatial impact evaluation methodology that became a high-demand tool commissioned by many stakeholders (including USAID missions) for assessing impacts in difficult-to-measure environments. HESN investments supported universities in developing new courses, majors, minors, and research centers related to international development research and innovation. Among the target HESN Development Labs, HESN supported 31 new classes and disciplines through AidData, 11 through CITE, 15 through IDIN, and 5 through RAN. For example, HESN Development Labs’ work led to new development-related courses for students in the United States and abroad. At the National University of Colombia, one faculty member adjusted the content of his classes following an IDIN IDDS in Ghana. Similarly, faculty at Universidad del Valle (in Colombia) developed a co-creation course based on IDDS Basura Cero (Zero Waste). In Kampala, the East Africa RILab promoted human-centered design courses in its networked universities. In the United States, the College of William and Mary developed two new classes taught by faculty hired through a HESN grant. At MIT, professors in the (1) development planning and implementation, (2) urban studies and planning, and (3) mechanical, civil, and environmental engineering areas integrated CITE methodologies into their curricula. HESN Development Labs fostered enough momentum in partner universities that the universities developed new degree programs. The private Universidad Sergio Arboleda (Colombia) added a new major option related to innovation, in concert with new courses and new opportunities for students to work embedded in communities. In Ghana, UDS became the anchor of the West Africa RILab and developed two new master’s degree programs: a master of implementation science and a master of science in community disaster resilience studies. These degree programs are in the process of accreditation. Finally, William and Mary launched a new data science major after students expressed strong interest in a minor with the same focus. The expertise behind the data science major comes mainly from AidData-supported hires. HESN influenced at least one university to form institutes for development research. UTEC (Peru) drew from its work with CITE to create a research center on development technologies. “Now, more and more, our students know that they are here to get an education to be empowered in order to be able to go and empower other people through creation of job opportunities, through their innovation.” —HEI leader USAID.GOV PERFORMANCE EVALUATION OF HESN | 43 HESN provided students with an opportunity to gain fieldwork experience. Most survey respondents and interviewees agreed that one of the HESN program’s essential contributions was increased opportunities for students, which included research assistantships, coursework/degrees, and fieldwork. For example, D-Lab and IDIN provided MIT students with opportunities to travel into the field to learn about co-design methodologies and the challenges faced by local communities. At IDIN partner universities such as Kwame Nkrumah University of Science and Technology, students brought to local communities their research and prototyping processes and their new training in collaborative design. The AidData Summer Fellows program made possible the direct engagement of students with in-country organizations around the world. The Global Research Institute, which now houses AidData, saw such value in the Summer Fellows program that it has continued to fund the fellowships beyond HESN 1.0. HESN produced a breadth of outputs that helped USAID support international development worldwide. In Figure V.20, we highlight the main results of the work conducted between 2012 and 2018, when most core-funded activities took place at the HESN Development Labs. HESN supported the creation of 96 new courses, programs, or disciplines, and influenced 160 institutions and generated more than 291 data sets or data-related technologies, tools, and approaches. The program also spurred 993 innovations and reached nearly 7.1 million direct beneficiaries over its life cycle. Figure V.20. HESN at a glance Source: HESN M&E data. Notes: * USAID devised the HESN program, or the policy change impact score, to assess the geographic reach and influence of a HESN Development Labs’ research outputs, with 6 being the highest score of policy and program influence (Fowle et al. 2020). HESN Development Labs submitted scores and justifications to USAID biannually. Figure V.11 and its associated findings above detail the program and policy change results. Inter-university collaboration Makerere University and RAN collaborated widely with other HESN Development Labs. For example, Makerere University worked with the University of California Berkeley to run the annual BigIdeas competition for five years (and beyond HESN 1.0), and RAN collaborated with AidData in efforts involving summer fellows, hackathons, and GIS trainings. USAID.GOV PERFORMANCE EVALUATION OF HESN | 44 D. HESN INFLUENCE ON STAKEHOLDER CAREERS The bibliometric analysis assessed the influence of HESN-associated researchers, and their publications, in their fields. The analysis also gauged those researchers’ career growth by examining the proliferation and citation of their articles.18 In this section, we highlight the results of the bibliometric study. Key findings for bibliometric analysis 1. Researchers whose work was supported by HESN funding varied widely in experience and influence in their fields. 2. Researchers in different stages of their careers experienced different publication trends during HESN. 3. The influence of sampled researchers in their fields increased substantially during the HESN program, particularly among mid-career scholars who earned their PhD’s between 2000 and 2009. 4. Sampled researchers who earned their PhD’s between 2000 and 2009 had the most significant average growth in citations of their work during and after the HESN period. HESN supported researchers and faculty in exploring new areas of development and expanding their expertise. Several universities supported by HESN adopted mandates for teaching, research, and community extension. In the East Africa RILab at RAN, HESN funding allowed a mindset change among faculty members who spent more time teaching by supporting research embedded in (and benefiting) local communities. Researchers whose work was supported by HESN funding varied widely in experience and influence in their fields. A review of the Google Scholar profiles for the 51 sampled researchers showed that seniority was associated most closely with publication productivity. Younger scholars (graduate students or recently graduated researchers) supported by HESN tended to publish fewer high-impact articles and generally counted fewer citations of their work than more senior HESN-supported scholars. As Table V.3 shows, 13 of the 51 sampled researchers had 100 or fewer citations of their work before 2015 (several with no citations).19 In contrast, established faculty (particularly those researchers who earned their PhD’s before 2000) had the highest citation levels; 14 of the 51 sampled researchers had more than 2,000 citations before 2015, and 3 of those researchers had more than 10,000 citations. In short, HESN funding supported researchers at all career stages.20 18 There are two limitations to this analysis. First, we cannot directly attribute the career growth of sampled researchers (or their level of engagement in international development scholarship) to HESN funding. Second, we cannot capture the full view of researchers’ careers, which may include work on data tools or innovations, by relying solely on metrics in the published academic literature. 19 We use 2015 as the threshold for grouping researchers because our review of HESN-supported publication dates suggests that research funded under HESN was published largely from 2015 onward (Figure V.3). 20 The pie chart in Annex D shows the number and share of citations among all researchers and demonstrates that just three of the sampled scholars carry more than half the total citations of the 300,000+ life-time citations represented by the sample. USAID.GOV PERFORMANCE EVALUATION OF HESN | 45 Table V.3. Average citation levels by pre-2015 researcher output group LEVEL OF RESEARCHER OUTPUT PRE-2015 ALL CITATIONS CITATIONS SINCE 2015 CITATIONS BEFORE 2015 LIFETIME H-INDEXA H-INDEX SINCE 2015 LIFETIME I10-INDEXB I10-INDEX SINCE 2015 Low: 0–100 citations (13) 191 180 10 6 6 4 4 Medium: 101–500 citations (14) 1,181 920 260 14 13 20 17 High: 501–2,000 citations (11) 2,197 1,264 933 22 17 38 28 Very high: 2,001+ citations (14) 19,761 9,634 10,127 54 39 178 128 Average of all groups 6,151 3,154 2,997 25 19 62 46 Source: HESN M&E data. a H-index is the number (h) of an author's studies cited h or more times. It gives a measure of an author’s impact in his or her field. b i10-index refers to the number of an author’s studies cited at least 10 times. It gives a measure of an author’s basic productivity in his or her field. During HESN, researchers in different stages of their careers experienced different publication trends. Grouping the same data by the year that each researcher earned their PhD,21 we can see a dramatic change from pre-2015 and post-2015 citations (Figure V.21). As expected, experienced researchers (those who received PhD’s before 2000) entered HESN with the most average citations of any group. However, once scholars had worked under HESN for several years (by 2019), established but younger researchers (those who received PhD’s between 2000 and 2009) more than doubled their research influence. They matched their more experienced colleagues in terms of average citations. Similarly, even younger researchers who earned their PhD’s in 2010 or later—or were still in graduate school—more than doubled their annual average sources from 2013 to 2019.22 21 We use PhD year as a proxy to group researchers by their career stage. Career stage is a useful categorization under the assumption that HESN-funded research opportunities (and other opportunities) may affect researchers’ scholarship differently at different points in their career. 22 Career stage in our analysis refers to the length of time since researchers acquired their PhD’s. We categorize sampled scholars into experienced researchers (PhD’s received 1999 or earlier), established researchers (PhD’s received 2000–2009), newer researchers (PhD’s received 2010–2014) and newest researchers (PhD’s 2015 or later, current graduate students). USAID.GOV PERFORMANCE EVALUATION OF HESN | 46 Figure V.21. Average citations early in HESN and after HESN support concluded (researchers grouped by PhD year) Source: HESN M&E data. Notes: We selected 2013 and 2019 because our review of HESN-supported publication dates suggests that most research under HESN was not yet published in 2013 (providing an idea of non-HESN output). By 2019, however, HESN-supported researchers had been writing with the program's support for several years. See Figure V.3, HESN Development Lab outputs produced by year, by lab. The influence of sampled researchers in their fields increased substantially during the HESN program, particularly among mid-career scholars who earned their PhD’s between 2000 and 2009. We assessed researchers’ influence by using the H-index, the number (h) of an author’s studies that have been cited h or more times, thus providing a measure of an author’s impact in her or his field. In Figure V.122, we display the average H-index of researchers grouped by PhD year. Before 2015, only researchers who earned their PhD’s before 2000 demonstrated prominence in their fields, with an average of 15 articles cited more than 15 times. However, the H-index of researchers since 2015 (when the publication of HESN-supported articles ramped up) is more robust across researchers in all career stages. Without a “control” group of similar researchers who did not receive HESN support, we cannot attribute the growth in publications, citations, or influence to HESN support. However, qualitative evidence from interviews and focus groups suggests that program funding generated opportunities for young scholars to expand their research and innovation work, gather new evidence, and publish studies to advance their fields and reputations. In Figure V.23, we show the citation levels of sampled scholars by PhD date group and demonstrate the differences in scholarship productivity growth during the HESN period. Each line represents one scholar’s citation metrics; the bold black line indicates the group average. Sampled researchers who earned PhD’s between 2000 and 2009 had the most significant average growth in citations during and after the HESN period. Younger researchers saw substantial citation growth during and after HESN, but experienced scholars generally maintained stable levels. The data support the theory that older researchers had already gained prominence in their fields before HESN but that younger researchers gained visibility in their fields during and after HESN—likely due in part to the program's research opportunities. USAID.GOV PERFORMANCE EVALUATION OF HESN | 47 Figure V.22. Annual citations of sampled scholars during and after the HESN period, by career stage Source: HESN M&E data. USAID.GOV PERFORMANCE EVALUATION OF HESN | 48 E. COST ANALYSIS OF HESN Donors worldwide are increasingly interested in the value-added that international development programs provide to their organizations and the communities that they serve. It is not enough to answer questions related to the contributions and impact of interventions; we need to know how effective interventions are relative to others. USAID has developed clear cost analysis guidance over the past several years based on best practices in education, economics, development economics, and related fields (Walls et al. 2020). USAID guidance details four main types of cost analyses: cost-economy, cost-efficiency, cost￾effectiveness, and cost-benefit. Mathematica conducted cost-economy and cost-efficiency analyses. 1. Cost-economy analysis. This examines the cost of delivering the interventions, ideally by task or component. It helps the government understand what it may cost to scale up the program or budget for a new intervention. These analyses require expenditures and contribution reports that are disaggregated by cost categories or ingredients, a method for allocating the shared costs across categories, and potentially a local price database for standard inputs. The analysis may also require government cost structures and output data (Walls et al. 2020). It also separates the recurrent and non-recurrent costs, which can help program designers understand what aspects of an intervention are locally sustainable. The cost-economy analysis is a basis for the other types of cost analyses and provides insights into unit cost variations among various project activities. 2. Cost-efficiency analysis. Reaching efficiency means that we maximize the outputs produced by a given project. Cost-efficiency analysis allows us to compare the costs of an intervention to the program's outputs (for example, cost per workshop or report or trainee). This analysis is useful when we want to understand whether different delivery models produce a given output more efficiently. This type of analysis also allows us to understand how the context of specific interventions and their characteristics drive the cost output (Walls et al. 2020). For example, it can help us understand why a workshop or innovation produced in one country is less costly than when produced or delivered in another location. To complete this type of analysis, we also need detailed cost data, including, at a minimum, expenditures disaggregated by cost categories, ingredients for each cost category, contributions data, and output data (Walls et al. 2020). 1. Cost data and assumptions The HESN program involved several interventions across the eight HESN Development Labs: workshops, research activities, evaluations, funding of innovations, and the creation of data sets. To assess the cost￾economy and cost-efficiency of the HESN interventions, we gathered cost data from the four labs in this evaluation. Unfortunately, the labs were not able to provide disaggregated funding for the core-funded activities. IDIN provided disaggregated cost by core component (such as researchers, M&E, innovation center support, travel). We were able to conduct a basic cost-economy analysis by estimating a percentage of the costs allocated to different types of deliverables. Based on the data's limitations (detailed in Chapter IV), we completed the following cost analyses. a. Cost per dollar leveraged: This analysis examines the amount of money leveraged by HESN per dollar invested. We assumed the total amount invested by USAID to be $115 million based on data provided by USAID. USAID also provided data related to the amount of funds that the HESN Development Labs leveraged. The HESN Development Labs leveraged $256 million as a result of the USAID investment. USAID.GOV PERFORMANCE EVALUATION OF HESN | 49 b. Cost per direct beneficiary: Based on our data analysis of the M&E output and outcomes data provided by the HESN Development Labs and USAID, we determined the total number of direct beneficiaries served by HESN between 2012 and 2018. We divided the total number of direct beneficiaries23 by the total USAID investment to determine the cost per direct beneficiary. c. Cost per direct beneficiary reached by HESN innovations, by lab. For this calculation, we used the M&E data to calculate the number of innovations across the HESN Development Labs. The cumulative number of beneficiaries comes from the reported M&E data. We used these data to calculate the average number of unique beneficiaries per innovation by dividing the cumulative number of beneficiaries by the number of innovations for each HESN Development Lab. We made the following assumptions for the cost data. – First, there was no way to determine which output types directly contributed to each innovation, because the HESN Development Labs could not tell us how much of the core and buy-in budgets went into the different output types (particularly core funding). To calculate the cost per direct beneficiary by innovation, we calculated the number of output types (for example, 22 workshops, 56 data sets) for each HESN Development Lab. We then estimated the percentage out of the total number of outputs (for example, 22 workshops of 225 total products for a HESN Development Lab equals 9.7% of the total products produced by a given HESN Development Lab). We made these calculations for each lab. – Once we had the percentage of total outputs for each innovation, we assumed that each HESN Development Lab used about the same percentage of the total budget to create the products (for example, if a lab innovation was 9.7% of the total outputs produced, then we assumed 9.7% of the overall HESN Development Lab budget). This calculation means that the lab used 9.7% of $24 million on workshops. We understand that this assumption is not rigorous, so the results presented in the findings section should be understood as loose estimates based on the data we had available. – To calculate the cost per average unique beneficiary, we used this: d. Cost of events and per-participant costs, by HESN Development Lab. Since workshops and events were a significant part of the HESN interventions, we also analyzed the cost per participant based on the number of events and participants in each event. We used the M&E data related to outputs to calculate these costs. To estimate the cost per event and cost per participant, we used the same assumptions as above (that is, percentage of workshops as a function of the overall products a HESN Development Lab produced and used that percentage to calculate the estimated percentage of budget used for the activity). We estimated the number of workshops and major events as a percentage of the HESN Development Lab's total outputs. We then used that percentage to estimate loosely how much of the total budget went into workshops and major events compared to other outputs. Table V.4 highlights the results of the analysis. e. Cost per output type: The final cost estimates relate to the cost per product by output type. To estimate this cost, we used M&E data to calculate the number of outputs that each HESN Development Lab completed during HESN. We then calculated the number of each output as a percentage of the total outputs for the specific HESN Development Lab (as detailed under #3 above). 23 Unique direct beneficiaries include households/individuals, local communities/governments, national and international policymakers, organizations/enterprises, researchers, and another category. USAID.GOV PERFORMANCE EVALUATION OF HESN | 50 We used that percentage and assumed that a similar percentage of the budget was used to produce the product. 2. Cost findings This section presents the findings for the HESN cost analysis. The HESN Development Labs were able to leverage a significant amount of funding relative to the amount that USAID invested in the program. The result of our cost analysis shows that for every dollar USAID invested in HESN, the HESN Development Labs leveraged $2.23 per dollar of investment. The HESN Development Labs provided more than double the initial USAID investment in in-kind contributions (for example, HESN Development Lab space, scientific equipment, teaching assistants, research assistants), including more than $200 million from academic, private sector, governmental and other stakeholders for development projects, and an additional $56 million in equity, debt, and philanthropic support for entrepreneurs. The cost of reaching each direct beneficiary through the HESN Development Lab activities was relatively low ($16.20 each). HESN served 7.1 million direct, unique beneficiaries. USAID invested $115 million across the eight HESN Development Labs. The result of the investment is a cost of $16.20 per direct, unique beneficiary. The cost per unique beneficiary of the HESN innovations varied widely across the different HESN Development Labs. RAN had the lowest cost per unique beneficiary at $2,723, and CITE had the highest at $81,564. The cost per beneficiary is a function of the total number of innovations, and both RAN and IDIN generated significantly more innovations than CITE. However, the lower cost for RAN and IDIN is also a function of the innovation workshops' location. RAN’s work took place exclusively in Africa, so running innovation workshops and grant programs is less than the costs for the other labs. Table V.4. Cost per unique beneficiary of HESN innovations, by HESN Development Lab LAB NUMBER OF INNOVATIONS CUMULATIVE NUMBER OF BENEFICIARIES REACHED OVER LIFE OF SUPPORT AVERAGE UNIQUE BENEFICIARIES PER INNOVATION COST PER AVERAGE UNIQUE BENEFICIARY RAN 637 4,700,362 7,379 $2,723.86 IDIN 880 1,996,297 2,269 $3,605.33 CITE 11 936 85 $81,564.08 Note: We were not able to include AidData in these analyses due to a lack of beneficiary data. USAID.GOV PERFORMANCE EVALUATION OF HESN | 51 The cost per participant24 in HESN Development Lab events also varied widely. IDIN had the lowest cost per participant at $377, and RAN had the second-lowest-cost at $717. The lower cost for these two HESN Development Labs is a function of several factors, including the events (mainly in LMICs), the proliferation of events, and the large number of participants attending each event. Table V.5. Cost per event and participants LAB EVENTS PARTICIPANTS PARTICIPANTS PER EVENT COST PER EVENT COST PER PARTICIPANT RAN 137 16,248 119 $ 85,398.85 $ 717.64 IDIN 472 16,324 35 $ 13,223.07 $ 377.80 AidData 26 748 39 $ 137,065.96 $ 3,514.51 CITE 10 1,355 136 $ 467,148.58 $ 3,434.92 The average cost of producing an output or deliverable under HESN is $39,263 based on USAID’s $115 million investment. The highlighted boxes in the table below show the product that was the primary output (highest percentage of the total outputs) for each HESN Development Lab (for example, AidData produced mainly data sets, and CITE produced mainly reports). Overall, RAN and IDIN’s products tended to be the most cost-efficient compared to CITE and AidData. The products produced by AidData were two to three times more expensive than IDIN and most of RAN’s products. However, the costs experienced by AidData and CITE are in line with other government contractors and NGOs working in the international development field. Table V.6. Cost per output OUTPUT TYPE AIDDATA PRODUCT CITE PRODUCT IDIN PRODUCT RAN PRODUCT Workshop/training/capacity building $ 132,534.85 $ 78,980.94 $ 14,634.50 $ 61,855.25 Publication or report $ 124,905.67 $ 78,911.59 $ 52,335.51 $ 61,862.52 Knowledge-sharing or collaborative platform – – – – Major event $ 161,987.04 $ 79,119.84 $ 14,627.71 $ 62,498.09 Data set $ 124,315.64 $ 78,866.86 $ 11,327.20 $ 61,536.58 Other $ 161,987.04 – – $ 61,840.22 Data-related tool $ 129,589.63 – – $ 61,795.86 Evaluation – $ 79,149.03 $ 11,327.20 $ 62,498.09 Data-related technology $ 125,989.92 – – – Data-related approach $ 125,989.92 – – $ 62,498.09 Hub – – $ 14,662.43 – Technical meeting – – $ 15,102.93 $ 62,498.09 24 An event participant is based on the number of reported event attendees. These are a subset of the overall unique, direct beneficiaries. PERFORMANCE EVALUATION OF HESN | 52 F. HESN SUSTAINABILITY In 2019, Uganda’s Makerere University assumed management of the government of Uganda’s Research and Innovations Fund (RAN), the first of its kind, dedicated to supporting high-impact research and innovations at the university. It includes $8.1 million annually in funding from the government for at least three years. The research and innovation work conducted under HESN was a catalyst that guided Makerere’s 10-year strategic plan to become a lead research university and provide leadership in innovation. The university is now working in the following ways to multiply what HESN helped create: ● RAN’s chief of party and Makerere professor was appointed as the acting chairman for the committee managing the country’s National Research and Innovation Program. ● RAN has continued to use and expand its network as agencies draw on the expertise of researchers in specific countries. ● RAN continues to support USAID MBIOs. ● RAN continues to seek new funding and to receive grants in response to the research capacity and faculty development made possible by HESN. “The number one of benefit is the development and maturing of the innovation ecosystem, which we didn’t have before. Testimony for that is that Makerere has since established the Research and Innovation Fund. And if you look at the grant management committee that runs the fund, you have key members from RAN who were solicited for their expertise.” —Lab Staff Sustainability case study: RAN Institutionalization is the process of embedding something in an institution. It takes place when behaviors become standard practice and are widely accepted within an organization. The production and dissemination of data to guide decision making play a vital role in the institutionalization process, as does the availability of financial resources to continue carrying out research and innovation activities.25 The transfer of knowledge among HESN stakeholders, past and present, is also a key element to consider in looking at the institutionalization and sustainability of development activities. Several HESN Development Labs made progress toward institutionalization in response to the HESN program (Figure V.23). In 2019, Uganda’s Makerere University assumed management of the government of Uganda’s Research and Innovations Fund (RIF), which includes about $8.1 million annually in funding from the government for at least three years (sidebar). AidData raised more than $27 million in funding through grants and contracts. The College of William and Mary incorporated AidData into the AidData Center for Development Policy (see spotlight in Figure V.25 below). Both CITE and IDIN are located within MIT and receive university funding to support continued work in both HESN Development Labs. Survey respondents generally agreed that HESN activities led to additional collaborations, and qualitative evidence collected for the evaluation suggests that collaborations will continue to function and evolve. Similarly, survey respondents generally envisioned the sustainability of the changes to last over time. Figure V.23. HESN Development Lab institutionalization Source: HESN online survey. Notes: Numbers in parentheses are the total number of survey respondents. a Respondents include 15 USAID staff, 37 lab staff, 3 researchers, and 31 innovators. b Respondents include 21 USAID staff and 46 lab staff. 25https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5745864/#:~:text=The%20routine%20first%20b ecomes%20the,%2C%20expansion%2C%20consolidation%20and%20maturity. USAID.GOV PERFORMANCE EVALUATION OF HESN | 53 Figure V.24. AidData sustainability case study Housed in William and Mary’s Global Research Institute, AidData was founded when William and Mary, Development Gateway, and Brigham Young University came together to provide the global community with more comprehensive data on foreign assistance projects. Founded by a team of eight, AidData focused its efforts initially on generating and publishing more sectoral and spatially precise data on bilateral and multilateral programs. AidData has grown as the demand for more precise data grew, and the organization began placing more emphasis on partnering with international development organizations, including USAID. Its successes include: • Work with more than 40 bilateral, multilateral, and foundation partners since 2004. • In-country partnerships with more than 90 civil society organizations, line ministries, think tanks, and universities in 21 countries. • AidData quadrupled in size by 2016, in part due to work completed under HESN. • It became institutionalized into William and Mary as the AidData Center for Development Policy only 10 years after its establishment. • AidData raised $27 million in donor and foundation grants for international projects in the past 8–9 years. • The establishment of AidData contributed to creating the Center for African Development, an undergraduate think tank. • AidData supported the development of a new “data science” major at the university. • The international fieldwork opportunities created by HESN provided new linkages and experiences for students to gain research skills. Several elements facilitated the institutionalization, and hence the sustainability of the HESN Development Labs, including: 1. Each lab's ability to create new coursework, degree programs, and disciplines within the host university allowed them to hire faculty and expand their student base. 2. The creation of opportunities for students to gain fieldwork experience during their studies. The opportunities drew more students into the programs, including international students; allowed the labs and universities to hire better, more qualified staff; and helped the respective programs expand. 3. The experience with HESN improved each lab’s ability to respond to donor requests for proposals and helped the labs win more international development work. HESN also helped several labs learn how to work with donor organizations. HESN Development Labs now use the increased capacity to respond to and implement international development programs and undertake research and innovation. The data-related outputs produced by each HESN Development Lab reflect the intensity of each lab’s efforts in a given area, which helps the labs establish a niche within both the universities and the donors. AidData focused principally on developing data sets and data-related approaches and thus excelled in that area. Meanwhile, IDIN tended to focus on building communities worldwide to co￾design innovations that solved development challenges. In Figure V.25, we show the number of new classes or academic disciplines generated with or by the HESN Development Lab through HESN activities or support. The figure demonstrates that all four target HESN Development Labs substantially contributed to students’ education opportunities at their respective institutions and beyond. On the left, we USAID.GOV PERFORMANCE EVALUATION OF HESN | 54 show in the pie chart the number of data sets or data-related technologies produced by each HESN Development Lab. The chart also reflects unique work that helped establish the importance of each lab within its university. Figure V.25. The number of new data sets, data-related technologies, classes, or disciplines created by HESN-funded Development Labs Source: HESN M&E data. Each of the four target HESN Development Labs created a vibrant global network that supports sustainability. The network includes new U.S.-based partners, including other universities, colleges, and NGOs; international universities and community-based organizations; and students who have completed their programs, participated in HESN-related activities, and returned to their home countries to share their experiences. The global network created by HESN is one of the program’s most important contributions. When we asked MBIOs, HESN Development Lab staff, HEI leaders, researchers, and innovators whether HESN activities resulted in additional collaborations beyond the original work scope, 79 percent of survey respondents responded in the affirmative (Figure V.19). USAID.GOV PERFORMANCE EVALUATION OF HESN | 55 VI. CONCLUSIONS The HESN program was successful in meeting its main objectives. The program helped improve access to analytics and quality data and contributed to testing and implementing innovations that supported international development. It created a vibrant ecosystem of researchers, innovators, and practitioners who can continue their respective countries' progress. Survey results also showed that more than half of respondents felt that HESN outputs were useful and timely for the MBIOs. The outputs helped inform mission planning and design, and also contributed to improved M&E systems. The following section summarizes the conclusions of this evaluation. Research Question 1. To what extent have mission partners applied learnings from HESN 1.0 research or outputs to their programs? Cross-sectoral, methods-focused HESN Development Labs were more successful at adapting their offerings to meet the diverse needs of MBIOs and developing as research labs than sector￾specific HESN Development Labs. HESN Development Labs with a cross-sectoral focus could pivot their methodologies and find different new sectoral niches in which to work. As highlighted in our findings, HESN Development Labs such as AidData, CITE, and IDIN, and RAN each participated in various activities, ranging from GIS mapping and evaluation to designing and implementing innovative training seminars engaging in cash transfer programs. Their ability to draw on methodological strengths and engage experts from various sectors made them more marketable to MBIOs, helping them gain work. For example, AidData took its geospatial impact evaluation work from an idea to a proof of concept, and eventually to buy-ins with missions. One result of this methodological advancement was that evaluators conducting rigorous research did not have to travel to the field to gather primary data, because they could use granular, geocoded satellite imagery to evaluate work retrospectively. It also allowed multiple missions to adapt the data to local needs for evidence-based decision-making. AidData has conducted geospatial evaluations in 10 locations around the world and a variety of sectors. It is one of the best examples of how the hybrid funding model (core plus buy-in) helped HESN Development Labs reach more beneficiaries. Sector-specific HESN Development Labs such as GCFSI and SEAD faced more significant challenges in finding opportunities to engage with USAID MBIOs. Although the cross-sector HESN Development Labs could explore research activities across many areas, the sector-focused labs had smaller areas to work. When they did obtain work, the HESN Development Lab universities had to partner with other institutions in-country. This requirement meant that staff at the partner institutions conducted the actual work, rather than HESN Development Lab staff. The sector-specific HESN Development Labs engaged researchers at their university because the faculty wanted to conduct, not supervise, the work. In several cases, MBIOs requested activities unrelated to the HESN Development Labs’ focus, which forced them to stretch to acquire the expertise for the work or turn down the buy-in altogether. The utility of core, buy-in, and hybrid funding models varied depending on the activity's purpose. Activities funded through core were more useful to a broad audience, such as researchers, because the products tended to be publicly accessible goods such as data sets. Buy-in-funded activities were more relevant and pertinent to MBIOs, as these products were often co-developed and linked directly to the country strategy. The hybrid model had more functionality because it allowed missions to build on existing core work while adapting work to the mission's needs. It was also the most cost efficient, because it benefited a larger group of people. USAID.GOV PERFORMANCE EVALUATION OF HESN | 56 Research Question 2. What was the partnership's perceived utility among stakeholders (HEIs, HESN Development Labs, and policymakers)? The HESN innovation grants allowed researchers and innovators to develop and test ideas in a way that had never been done before. Innovation grantees strongly benefited from the support offered by the innovation infrastructure set up by HESN Development Labs. This infrastructure included local innovation centers, networks, advisory boards, and other spaces for working on innovations with others. Support could include activities like (1) capacity building through summits, workshops, and additional training on topics ranging from how to understand the needs of communities to marketing innovations; (2) access to advice from networks of other innovators covering a variety of sectors and disciplines; and (3) help navigating the policy environment to allow for the rollout and scaling of innovations. Although the innovation grants supported the design and pilot testing, grant beneficiaries noted that it was not enough. Innovation is a learning process that often requires ongoing support to test and re-test technologies. HESN lacked flexibility and funding to support ongoing work by innovators. Several people noted that investing in innovation hubs or incubators that allowed for ongoing technical and financial support is critical to ensuring potentially transformative innovations. Policymakers also believed that HESN was useful to local development efforts but wanted more engagement in the program. HESN’s primary audience was not policymakers. However, this stakeholder group benefited from the work done by the HESN Development Labs. Buy-ins helped local communities solve development challenges, and data sets helped policymakers make data-driven decisions. Even so, policymakers would have liked more engagement in the co-creation of activities. YOUNG INNOVATORS WORK ON A PROTOTYPE AT IDDS ZERO WASTE (SOURCE: DIVERSA) USAID.GOV PERFORMANCE EVALUATION OF HESN | 57 Research Question 3. Which structural or institutional elements of the partnership contributed to different levels of usefulness to mission programming and decision making? Which elements contributed to utility? Research and innovation take time, and the missions needed to show quick results. The goals of the HESN Development Labs differed from those of the missions, which created challenges in including innovators in the new learning ecosystem. The most successful engagements between HESN Development Labs and missions were built on missions explicitly identifying their needs for evidence, the labs grasping those needs, and the labs having the capacity to address those needs. For example, upon seeing the quality of the AidData’s core work and early buy￾ins, missions frequently reached out to that lab to commission geospatial impact evaluations, including in the West Bank/Gaza and in sub-Saharan Africa. AidData understood the exact research goals of the missions, because the missions approached with clear examples of prior work. Similarly, Berkeley DIL broke new ground when they engaged with USAID Rwanda to conduct the cash benchmarking studies, which generated demand from other missions for similar research on their programs. MIT CITE’s research on the Feed the Future Market system provided the Uganda mission with useful information to examine the level of U.S. market subsidies, making U.S. tax dollars go further. Finally, USAID Guatemala commissioned CITE to assess digital financial services in the Western Highlands of Guatemala, which produced a highly relevant and useful report for adjusting mission strategy and agricultural programming in the region. Each of these engagements was successful because HESN Development Labs grasped the explicit MBIO needs and had aligned interests and capacities. HESN’s flexibility was a crucial factor in allowing HESN Development Labs to build the ecosystem. The USAID AORs played an essential role in working with HESN Development Labs to allow them flexibility in finding niches and areas of work in which they could grow, engaging new expertise through partnerships, and venturing into new work areas. The flexibility and support were critical factors that allowed the HESN Development Labs to build vibrant national and international networks with existing and new partners. Although the national and international networks grew, the inter-HESN Development Lab network faltered because of a lack of funding and lack of incentives to work together. Staff from both the HESN Development Labs and USAID had hoped that the labs would work closely together on research and innovation activities under HESN. However, factors associated with HESN structure and processes created barriers to building a robust inter-HESN Development Lab network. Apart from the TechCon events, HESN Development Labs had no funding to participate in events with other lab members and found few lasting incentives to work together. The challenge was worsened by HESN funding reductions that made the labs feel that they had to compete with their lab peers for buy-in funds, so they were less likely to work together. HESN Development Labs also expressed that they received little guidance from USAID on working in partnership with one another in core and buy-in activities. Other barriers to network￾building ranged from structural elements, such as contracting difficulties, to process elements, such as finding projects of mutual interest. “Building the innovation ecosystem took time. Many innovators expect that things will be done in a year. The process is particularly slow with health, as many innovations have to go through clinical trials. We're learning to manage the expectations of innovators and partners and stakeholders so that they know that innovation is a great thing, but it requires flexibility.” —HESN Development Lab Staff USAID.GOV PERFORMANCE EVALUATION OF HESN | 58 Research Question 4. Which process elements of the partnership contributed to different levels of usefulness to mission programming and decision making? Which elements contributed to utility? HESN Development Labs struggled to engage MBIOs because goals, objectives, and timelines were not fully aligned. The partnerships with MBIOs was another area that occasionally “broke down” in the ecosystem. From the mission and OU perspective, it was often challenging to engage with HESN Development Labs; even when there was interest in working together, the two-year planning cycle meant that missions had few resources. It was structurally complicated for missions to pivot and accommodate ideas from USAID/Washington and the HESN Development Labs, though several missions managed to do it. From the HESN Development Labs' perspective, mission timelines did not always match research timelines, which made it difficult for the labs to find mutually agreeable activities. As mentioned throughout this report, the AORs played an important role in facilitating and helping MBIOs and HESN Development Labs negotiate mutually agreed-on work. Activities that focused on concrete, action￾oriented things also tended to be more successful, such as when HESN Development Labs contributed reports or inputs into MBIO planning and programming; conducted evaluations; and developed data sets that MBIOs used for decision making. HESN Development Labs sometimes struggled to communicate the results of their findings. Researchers working with HESN Development Labs were accustomed to communicating results through long reports and academic publications. Under HESN, they had to learn how to package results in a way that was accessible to the MBIOs and other stakeholders. Missions sometimes found that studies or evaluations lacked conclusions suitable for immediate application to programming at the country level. Research products that were most useful were those such as the MIT CITE assessment of digital financial services in the Western Highlands of Guatemala, which provided clear, specific, and geographically sensitive conclusions for the mission to adjust its programming. Research Question 5. To what extent has HESN contributed to changes in HEIs or in HEI networks that increase their engagement in international development? To what extent would any changes be sustained? Why or why not? HESN Development Labs supported the creation of 96 new courses, programs, or disciplines within their universities under HESN 1.0. A critical part of donor investments is enabling the long-term sustainability of their investments. The four HESN Development Labs increased its international development engagement through work with USAID missions, OUs, policymakers, and local communities. The HESN Development Labs also created new courses, degree programs, and disciplines that support data analytics and their use in international development. The four case study HESN Development Labs are institutionalized into their respective universities in a way that will make them sustainable. For example, AidData now receives direct funding from the College of William and Mary to ensure the center can remain active and grow. As a result of HESN, AidData is now able to recruit more expert researchers, who in turn also teach at the college. With its integration into its host HEI, AidData now also has more international students supporting data processing, and HESN has in turn enabled those students to gain more international fieldwork experience. AidData’s growing leadership in geospatial data analytics (and well as in other areas, such as tracking underreported financial flows) is fed by its status as a research center integrated into the college. HESN Development Labs contributed to program or policy influences in 160 different institutions. These institutions included USAID, other U.S. government institutions, host country governments, bi- or multilateral institutions, NGOs, the private sector, and others. Program and policy changes included minister-level decisions to make aid project data publicly available going forward, NGO pivots in their priorities, and USAID mission adjustments to agriculture programming. DIL, CITE, and IDIN were the most productive HESN Development Labs in influencing outside institutions. USAID.GOV PERFORMANCE EVALUATION OF HESN | 59 Additional evaluation findings HESN accelerated the creation, testing, and scaling of transformative innovations, technologies, and approaches. HESN fostered an essential ecosystem for innovation, product design, testing, and evidence-building. The HESN Development Labs created about 2,000 innovations, including technology prototypes such as PedalTap, a hands-free foot-operated water dispensing system, and the Grass Fuel project, which is an alternative to wood fuel and helps reduce depletion of the forest reserves in the Kasena-Nankana District in the upper East Region of Ghana. HESN Development Labs also conducted numerous capacity-building workshops, including IDDS and lean research. RAN-SUPPORTED WINNOWING MAIZE THRESHER INNOVATION AT WORK (SOURCE: RAN) HESN catalyzed a global interdisciplinary ecosystem of individuals and institutions, one that shares knowledge, promotes learning, and builds mutual capacity. A “learning ecosystem” is a system of people, strategies, technologies, content, and cultures that exist within and outside organizations but work together to impact formal and informal learning that happens within an organization.26 Like living ecosystems, learning ecosystems have complex relationships that require people to play different roles, use other content, and develop complex relationships to ensure the system functions. HESN has successfully created its global interdisciplinary ecosystem. Through its work, HESN Development Labs have established more than 1,205 partners who worked or are working across 83 countries on various activities, including supporting research, student fellowships, and innovation summits and contests. It is truly a global system of learning and sharing knowledge, innovation, and capacity building. The challenge for the universities and USAID will be to keep the network healthy under HESN 2.0. Several HESN Development Labs noted during key informant interviews that, even beyond HESN, they have continued to partner with the new stakeholders when they bid on projects with other donors. HESN prioritized data-driven decision making is one of the critical drivers of international and economic development in countries worldwide. Hard data must be used ethically, and the data for organizations must inform decisions to adopt and use knowledge. Insufficient data can erode trust in analytics, increase concerns about transparency, and lead decision-makers to trust their “instincts” rather than the data. The UN estimates that 90 percent of “big data” has been created in the past two to three years, and they expect it to continue to increase by nearly 40 percent annually.27 HESN has contributed to this growth in data analytics. 26 See https://www.ej4.com/blog/what-is-a-learning-ecosystem. 27 See https://www.un.org/en/sections/issues-depth/big-data-sustainable-development/index.html. USAID.GOV PERFORMANCE EVALUATION OF HESN | 60 HESN Development Labs created 620 data sets or data-related technologies, tools, analyses, or approaches. AidData and DIL produced the bulk of the 271 data sets developed with HESN funding and specialized in geocoded data sets and data sets from surveys of beneficiaries in electricity-related projects. Data-related technologies included survey apps, cloud platforms, and sensors for precision measurement of heat and pollution. RAN developed the most data-related tools, including focus group discussion protocols, deliberative polling systems, and resilience assessment tools. Data analyses and approaches included geospatial impact evaluations, data visualizations, modeling, and mapping. These data-related outputs were often used directly by policymakers or NGOs to develop policy and programming. Researchers and universities nearly always used them to advance development-related sciences. HESN Development Labs created more than 100 innovation-related publications or other intellectual works. As noted in the bibliometric analysis, HESN activities provided researchers with opportunities to publish on a broader scale. Researchers whose work was supported by HESN funding varied widely in experience and influence in their fields. The influence of sampled researchers in their fields increased substantially during the HESN program. According to the H-index, scholars in the 2000– 2009 PhD group gained the most influence in their fields of any career stage group during HESN. HESN generated more than $256 million in leveraged funds. That investment was more than two dollars leveraged for every dollar of U.S. government funding provided to HESN. Capacity-building events and investments in innovations were more cost-efficient than other types of outputs produced under the program. The average cost per unique HESN beneficiary in the four target HESN Development Labs ranged from $2,273 to $229,913, depending on the number of outputs, products, or innovations the labs produced. Innovations reached over 19.5 million non-unique beneficiaries and 7.1 unique beneficiaries across about 35 countries. RAN, IDIN, and SEAD produced the most innovations of the HESN Development Labs. These innovations included water- and health-related technologies, novel digital systems, agricultural implements, and livelihood-supporting prototypes. Although most innovations benefited a few dozen individuals, many, such as RootIO (community radio system), the Ayzh clean birth kit, and the unbanked digital financial services in Uganda (known as AkelloBANKER), benefited tens or hundreds of thousands. USAID.GOV PERFORMANCE EVALUATION OF HESN | 61 VII. RECOMMENDATIONS The following section provides USAID with recommendations that can help facilitate and improve the use and utility of research programs. Improving how MBIOs apply research or outputs to their programs. Adhere to a co-design model to engage USAID missions, operating units, and local governments in developing relevant and applicable activities. Although HESN processes included a co-design model for activities, mission staff often expressed that they did not feel that activities were actually co￾designed with all relevant stakeholders. Missions highly encouraged their Washington colleagues to adhere to a co-design process with MBIOs on future research programs. The co-design process can ensure that missions include funding in their CDCS planning processes for new projects that meet country needs. The co-design process can be helpful at both the project-level and the individual research activity level. HESN 2.0 appears to use a co-design process in aspects of its work. However, that process should also include local government if USAID wants to have a policy impact and uptake of research to drive evidence-based decision making. Because local governments were not the primary audience for deliverables, they were left out of original HESN activities. Maintain the focus on “growth mindset” and flexibility. As discussed in the report and our conclusions, stakeholders repeatedly identified the focus on flexibility and a “growth mindset” as the elements that most facilitated the success of HESN activities. USAID should look for ways to maintain this flexibility on research-related programs, to allow implementers to explore new avenues, build new partnerships, and adjust as they learn along the way. Failure often leads to innovation, so being open to failure and supporting participants in their efforts to try again can facilitate learning. Improving the utility of research programs for stakeholders. Continue to focus on evidence, technology, and innovation. Many HESN activities contributed to development outcomes that are changing and affecting lives in LMICs—reaching nearly 35 million direct and indirect beneficiaries. USAID should continue to design and implement programs that focus on evidence, technology, and innovation, contributing to international development in both large and small ways. Continue to invest in capacity building. HESN successfully invested in different kinds of capacity￾building over time. USAID should continue to support local capacity building because it empowers participants at the local level to adapt and contribute to changes. The focus should be on “action without harm.” Assess the contributions of workshops and summits that identify local needs and gaps. The USAID-funded LASER program is using a model that engages MBIOs and local stakeholders to co0identify needs and gaps in development. The model allows the implementing university to then design interventions and activities that support local stakeholders. It is an interesting model and should be evaluated to determine the extent to which it contributes to MBIO future programming and the use of outcomes and products by stakeholders. Overcoming structural and institutional barriers. Monitor and reconcile diverse incentive systems. Differing incentive systems can present structural barriers to reaching goals. For example, universities are motivated to publish research findings, whereas USAID is motivated to serve local populations. USAID should consider these differing incentive systems under HESN 2.0 and work with universities to ensure stakeholders use research in practical ways. HESN’s work provides examples of ways to reconcile the incentives. Development Gateway with AidData developed and implemented geocoding work that met the mission’s and the government USAID.GOV PERFORMANCE EVALUATION OF HESN | 62 partners' needs. RAN worked hard to help researchers see the value of moving away from the “ivory tower” of research to working directly with communities. Although reconciling different incentive systems is not easy, lessons learned under HESN 1.0, including the importance of trust-building, open communication, and flexibility, can help USAID and its partners move toward workable solutions. As also mentioned above, workshops, summits, and events help reconcile different research agendas or identify local development needs. These activities also provide examples of successfully aligning incentive systems, and reconciling the different research agendas through the co-design process mentioned above. Find ways to align university research and USAID missions’ timelines to support the use of evidence. The differing timelines to complete deliverables between higher education institutions and donors were structural barriers that researchers faced in trying to complete activities. HEIs are accustomed to having two or three years to complete research products, but USAID MBIOs need quick evidence to take advantage of country policy opportunities. The HESN Development Labs suggested that the lack of “quick evidence” caused USAID leadership to lose interest and made it more difficult for USAID to get funding to continue some of the work. USAID staff felt intense pressure to show results to continue getting resources to implement HESN. The challenge noted by HESN Development Labs and USAID AORs related to the type of research conducted under HESN. They felt the research was not always oriented toward quick results, so HESN Development Labs had to balance the need for quick results with the time necessary for quality research. Although challenging and sometimes outside USAID’s control, trying to expand on the buy-in system might help meet USAID missions’ shorter-term research needs. Strengthening process elements to improve research program utility. Provide examples of communication products that MBIOs feel best help them communicate findings and results. USAID has numerous examples of concise, well-prepared policy briefs and infographics that it can identify and use as examples. When preparing scopes of work or buy-in activities, MBIOs could include examples of communication pieces that successfully reach their target audiences. These examples can help academic institutions understand how to convert research into evidence and results that are useable and understandable to missions and policymakers. The most successful communication pieces (1) synthesize and frame evidence in a way that policymakers demand and understand information; (2) is provided at the “right time,” meaning that MBIOs can deliver the results during a time that key decision-makers in the mission or government are planning and have a need for the information; and (3) demonstrate an understanding of the policy audience and their planning processes. Several HESN Development Lab and USAID staff also recommended hiring (or including) communication experts on research teams so they can facilitate the development of improved products that meet the needs of the client. Consider integrating stakeholder analysis as part of a new research activity. Stakeholder analyses can help implementers better understand the political and institutional context for research in a specific context, including the incentive systems for collaboration and uptake of research findings. These types of analyses can help ground findings and provide implementers and MBIOs with guidance on the types of products that will reach different audiences, as well as identify any potential roadblocks to the use of the evidence. Stakeholder analyses can also facilitate collaborations by helping all stakeholders understand the power dynamics among participating stakeholders. Sustaining changes at HEIs Engage directly with the government, universities, and the private sector to increase and sustain future funding. One of the biggest challenges in the innovations work was that, although the financing allowed innovators to work on new ideas, the budget rarely allowed them to pilot, test, and incubate the innovations at larger scales. Innovators noted that they needed time, funding, and space to properly “incubate” their ideas. A HESN Development Lab leader noted, “The original idea to improve products required a connection to the private sector…however, legal issues and the timing of research, whose idea USAID.GOV PERFORMANCE EVALUATION OF HESN | 63 you take, who is in charge….There is a mismatch between the time and pace it takes to do university research compared to the private sector and that limited their interest and engagement in HESN.” Future programs need to consider how to successfully engage and bring in different actors to support the innovation process. For example, in Colombia, innovators noted that universities often provided additional funding for innovators to work on their ideas. In Brazil, the government frequently releases funding calls that allow innovators to apply for grants to support their ideas. In other countries, both international and local companies sometimes create incubators or “hubs” that allow innovators to develop their ideas. USAID and its implementers need to consider what works in each country context, at the same time ensuring some support exists to help foster and sustain innovation. Build a more robust network among the HESN Development Labs. Consider eliminating the disincentives to inter-HESN Development Lab collaboration. New programs should build funding and activities for implementers, whether they are universities, NGOs, or private sector firms, to work together and bring their networks together. These types of connections can help grow the global learning ecosystem. They also help sustain research work as the networks continue to collaborate in the post-program cycle, including seeking new funding for other activities emanating from USAID collaborations. Other recommendations Improve collection of cost data to support more detailed cost-analysis. USAID is rolling out its cost guidance under new programs, which will improve their ability to collect and report cost information in the future. It will also help evaluators to conduct better cost analysis. In line with USAID’s guidance, we recommend that at a minimum, future research programs collect and report on the following data. ● Determine appropriate cost categories for research programs. These categories should include elements such as program management, researcher time, travel, and other direct costs. ● Implementers should collect and report the level of effort, associate travel, and ODCs for each product or deliverable completed under the research project. The detailed disaggregation will allow USAID to determine the cost per deliverable quickly. ● For programs that have both core- and buy-in-funded activities, implementers need to track the cost of deliverables by type of funding. ● If USAID wants to complete a cost-effectiveness analysis, ensure that the implementer is able to set up some type of impact assessment from the beginning of the program. Impact estimates are crucial to comparing cost effectiveness. ● Ensure that shared costs across activities are accounted for throughout the life of a project and reflected in the product costing and reporting. ● Ensure that implementers review USAID cost guidance and adapt the guidance for the project. The adaptation should be reflected in the monitoring and evaluation plan. “[To improve impact]...it would be good if there was broader participation and many more meetings to find the relevant questions yet unanswered that are most relevant to the mission strategies. It’s about identifying the challenges, so that information solutions are useful to them. Having more of a process for identification would be good, and more analysis of challenges before jumping in.” —Lab Staff This page has been left blank for double-sided copying. ANNEX A SOURCES OF INFORMATION This page has been left blank for double-sided copying. ANNEX A. SOURCES OF INFORMATION REFERENCES AidData Center for Development Policy (AidData). “Higher Education Solutions Network—Annual Report (FY 2017).” Report prepared for USAID. Williamsburg, Virginia: College of William and Mary, November 2017. Comprehensive Initiative on Technology Evaluation (MIT CITE). “Higher Education Solutions Network— Annual Report (FY 2017).” Report prepared for USAID. Cambridge, Massachusetts: Massachusetts Institute of Technology, October 2017. ej4. “What is a learning Ecosystem? And How does it support corporate strategy?” July 2018. Available at: https://www.ej4.com/blog/what-is-a-learning-ecosystem. Accessed on October 1, 2020. International Development Innovations Network (IDIN). “Higher Education Solutions Network—Annual Report (FY 2016).” Report prepared for USAID. Cambridge, Massachusetts: Massachusetts Institute of Technology, October 2016. McCowan, Tristan. “Universities and the Post-2015 Development Agenda: An Analytical Framework.” Higher Education, vol. 72, no. 4, 2016, pp. 505-523. McIntosh Craig, and Andrew Zeitlin. “Lessons from a cash benchmarking evaluation: Authors' version.” September 2018. Available at: https://blogs.worldbank.org/impactevaluations/lessons-cash￾benchmarking-evaluation-authors-version. Accessed October 2, 2020. Peters DH, Adam T, Alonge O, et al “Implementation research: what it is and how to do it.” British Journal of Sports Medicine, 2014;48:731-736. ResilientAfrica Network (RAN). “Higher Education Solutions Network—Annual Report (FY 2017).” Report prepared for USAID. Kampala, Uganda: Makerere University, April 2017. United Nations Office on Drugs and Crime. “Big Data for Sustainable Development.” Available at: https://www.un.org/en/sections/issues-depth/big-data-sustainable-development/index.html. Accessed October 2, 2020. United Nations Office on Drugs and Crime. “Post-2015 Development Agenda.” Available at: https://www.unodc.org/unodc/en/about-unodc/post-2015-development-agenda.html. Accessed October 2, 2020. A.3 | PERFORMANCE EVALUATION OF HESN USAID.GOV Table A.1. HESN Evaluation Interview List HESN STAKEHOLDER GROUP ASSOCIATED HESN LAB FULL NAME HEI leadership AidData Stephen Hanson HEI leadership GCFSI Joe Messina HEI leadership RAN Barnabas Nawangwe Innovator IDIN Alexander Freese Innovator IDIN Alexander Ortiz Innovator IDIN Asante Johnson Innovator IDIN Claudia Alejandra Villamil Innovator IDIN Diana Duarte Innovator IDIN Fabio Fajardo Innovator IDIN Hernán Pérez Innovator IDIN Jorge Appiah Innovator IDIN Juan David Reina Rozo Innovator IDIN Luis Fernando Marmolejo Innovator IDIN Pedro Reynolds Innovator IDIN Sneyder Neira Innovator RAN Grace Nakibaala Innovator RAN Jude Mukundane Innovator RAN Margaret Nanyombi Lab leadership AidData Ariel BenYishay Lab leadership CITE Bish Sanyal Lab leadership CITE Dan Frey Lab leadership IDIN Amy Smith Lab leadership RAN Nathan Tumuhamye Lab leadership RAN Roy William Mayega Lab leadership RAN William Bazeyo Lab staff AidData Alena Stern Lab staff AidData Jessica Wells Lab staff CITE Laura Budzyna Lab staff IDIN Kendra Leith Lab staff RAN Deborah Naatujuna Lab staff RAN Dennis Chirawura Lab staff RAN Dorothy Okello Lab staff RAN Harriet Adong Lab staff RAN Julius Ssentongo Lab staff RAN Natasha Kassami Lab staff RAN Ronald Kayiwa Partner (HEI) CITE Gordon Adomdza Partner (HEI) CITE Julien Noel A.4 | PERFORMANCE EVALUATION OF HESN USAID.GOV HESN STAKEHOLDER GROUP ASSOCIATED HESN LAB FULL NAME Partner (implementer) AidData Geoff Bergen Partner (NGO) AidData Carlos J. Neisa Partner (NGO) CITE Katherine Lucey Partner (private sector) AidData Alexandra Colmenares Policymaker AidData Dennis Akorlor Policymaker AidData Edward Chonia Policymaker AidData Shaffiq Mamudu Policymaker IDIN Laura Olave Researcher AidData Josh Powell Researcher AidData Mike Findley Researcher RAN Simon Kasasa Student (innovator) IDIN Juan Bedoya Student (innovator) IDIN July Carolina Rojas Gómez USAID Mission AidData Daniel Baako USAID Mission AidData Kevin Brown USAID Mission AidData Lucy Malo USAID Mission AidData Omar Lopez USAID Mission AidData Paulo Gomez USAID Mission CITE Gerson Morales USAID Mission CITE Martin Fowler USAID Mission ConDev Godefroid Mayala USAID Mission DIL Daniel Handel USAID Mission RAN Amber Lily Kenny USAID Mission RAN Laura Gonzalez USAID Operating Unit AidData, ConDev Kalene Resler USAID Operating Unit CITE, RAN Aubra Anthony USAID Operating Unit RAN Molly Dean USAID Operating Unit SEAD Grace Kim USAID/CDR AidData Brian Bingham USAID/CDR ConDev Michelle L'Archeveque Jones USAID/CDR Multiple Ticora Jones USAID/CDR RAN and MIT Maggie Linak A.5 | PERFORMANCE EVALUATION OF HESN USAID.GOV Table A.2. HESN Documents Reviewed PROGRAMMATIC DOCUMENTS RESEARCH DOCUMENTS HESN lab applications Evaluation proposals (buy-ins and other) Cooperative agreements and MOUs Monitoring and evaluation plans Annual and longer-term work plans Evaluation reports HESN annual reports Conference proceedings Quarterly reports Policy briefs Sustainability plans Other research products A.6 | PERFORMANCE EVALUATION OF HESN USAID.GOV ANNEX B DATA COLLECTION METHODS This page has been left blank for double-sided copying. ANNEX B. DATA COLLECTION METHODS A. DOCUMENT REVIEW Our evaluation team conducted an extensive document review to understand the work Labs completed under HESN. We synthesized the findings using a document review protocol. Figure B.1 shows the number of documents the research team reviewed. Figure B.1. Document review coverage B. ONLINE SURVEY The online survey included multiple-choice questions and Likert scales, which we converted into quantitative data. We conducted a separate online survey with each HESN stakeholder group so that we could triangulate the results. USAID staff reviewed the survey to ensure that wording and syntax were culturally appropriate. We piloted the online survey with Mathematica senior staff who have previously worked for the donor community, served as adjunct faculty at universities, or had project management experience. Upon completing the pilot survey, we followed up with a phone call to each person to discuss the survey and gather their input on the questions and format. This process helped us refine the questions for the final online survey. Following the pilot assessment, we finalized the surveys and distributed the surveys to the targeted stakeholder groups. We distributed 451 surveys and received 163 responses for a 36 percent response rate. Table B.1 shows the distribution of surveys and responses among stakeholder groups. Table B.1. Survey distribution and response rates RESPONDENT GROUP NUMBER SENT SURVEY NUMBER OF PARTIAL/FULL COMPLETERS PERCENTAGE COMPLETE USAID Missions 33 11 33% USAID Operating Units 50 26 52% HESN Labs 103 56 54% Researchers 22 6 27% Innovators 211 56 27% HEI leaders 32 8 25% TOTAL 451 163 36% C. KEY INFORMANT INTERVIEWS AND FOCUS GROUPS We recognize that in-person interviews are the ideal way to collect information from the Labs, Missions, and other local HESN stakeholders. In-person interviews allow us to interact with the staff and probe in more depth to understand critical issues. We initially proposed to travel to Uganda, Ghana, and Colombia to collect the qualitative data. We selected these countries because they demonstrated a range of B.3 | PERFORMANCE EVALUATION OF HESN USAID.GOV engagement and utilization of HESN by USAID Missions. We developed semi-structured interview protocols and refined them based on feedback from USAID. Our qualitative data collection involved interviews and focus groups with only four of the eight HESN labs: MIT IDIN, MIT CITE, AidData, and RAN. We conducted interviews and focus groups with 70 individuals, from a total pool of 82 individuals. Nine people did not respond to interview requests. In nearly all respondent groups, we achieved saturation. We were able to travel to Colombia but not to the other countries, due to the onset of COVID￾19. As a result, we conducted all interviews for Ghana and Uganda virtually, using the Webex platform. Table B.2 summarizes the number of interviews conducted during this evaluation and the sampling method used to select participants. Table B.2. Interviews conducted by stakeholder type and sampling method STAKEHOLDER GROUP TOTAL POOL OF POSSIBLE INTERVIEWEES NUMBER OF INTERVIEWS CONDUCTED SAMPLING METHOD USAID Mission staff and OUs 83 21 Maximum variation sampling using low, medium, and high usage benchmarks. HESN Lab leadership and staff 103 19 Purposeful sampling Policymakers 10 4 Convenience sampling HESN partner stakeholders 15 4 Purposeful/snowball sampling HEI leaders 16 3 Purposeful sampling HEI researchers 22 6 Purposeful sampling Innovators 93 25 Purposeful sampling Figure B.2. highlights the percentage coverage of the interviews per stakeholder group. Figure B.2. Key stakeholder interviews coverage Note: Numbers in parentheses are the total numbers of stakeholder entities or individuals available for data collection. The darker bars show coverage rates for entities and the lighter bars show rates for individuals. B.4 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. BIBLIOMETRIC ANALYSIS We exploited monitoring and evaluation (M&E) data provided by USAID on HESN-supported publications and Google Scholar metrics to conduct the bibliometric analysis. We focused our analysis on the authors of HESN-supported outputs rather than the publications themselves, because many of the items were not available in easily accessible online journals. We began the analysis by filtering the outputs in the HESN M&E data to include the following peer-reviewed items: • Publication or report • Evaluation • Conference proceeding or presentation • Evaluation of development programming • Innovation-relevant publication (market analysis, product evaluations, toolkits) • Journal article • Other intellectual works (e.g., technical reports, white papers) We randomly selected 15 percent of the filtered outputs and searched Google Scholar for those publications.1 Although most of the researchers had Google Scholar profiles, we excluded researchers who did not have a profile for this analysis. We also skimmed CVs to identify the year that each researcher received their Ph.D. This process allowed us to determine whether "publication pressure" or opportunities related to HESN contributed to changes in publication levels. This process produced a list of 51 HESN researchers. We collected the total number of times each researcher's work was cited in Google Scholar, their h-index, and their i10-index, as well as their citations, h-index, and i10-index from 2013 to 2019. We grouped researchers by pre-20152 research productivity levels, and by the year they received their Ph.D. The bibliometric analysis allows us to see how researchers' influence and impact changed during and after the work that HESN supported. This pie chart shows the number and share of all researchers' citations; it demonstrates that just three of the sampled scholars carry more than half of the total 300,000+ lifetime citations of the sample. 1 If the publication itself was not available on the internet, we directly searched for researchers mentioned in that publication's M&E details. 2 2015 was the peak of HESN publications so we used that year as the cut off to determine before and after publication growth. B.5 | PERFORMANCE EVALUATION OF HESN USAID.GOV Figure B.3. Distribution of lifetime citations among sampled researchers. It is important to note two limitations to this analysis. First, we cannot attribute the career growth of sampled researchers (or their level of engagement in international development scholarship) directly to HESN funding—we can only speculate a possible relationship. Second, we cannot capture the full view of researchers' careers, including work on data tools or innovations, with only metrics on published academic literature. E. POLICY CHANGE SCORECARD USAID developed a Program and Policy Change framework to track and quantify the influence of research on program and policy change in international development. The framework draws on existing conceptual frameworks of evidence uptake and the literature on policy change (Fowle et al. 2020). The framework is useful because it (1) can apply to all research sectors, (2) focuses on evidence-informed policy at various levels of geographical influence, and (3) includes a numeric scoring system for quantifying outcomes (Fowle et al. 2020). Mathematica analyzed the results provided by the policy and change scorecard and calculated an average score for institutional influence and implementation for each of the four target Labs. We present the results in the findings section. F. ADMINISTRATIVE M&E DATA We used relevant data collected for M&E reporting by Missions and all eight HESN Labs to help us answer the appropriate evaluation research questions. The M&E reporting data included the annual indicator data submitted by each Lab during the period of performance. The period of performance under the core HESN Cooperative Agreements with Labs began in fall 2012 and concluded in fall 2017; however, some Labs signed agreements with USAID Missions and Operating Units that extended their performance period. B.6 | PERFORMANCE EVALUATION OF HESN USAID.GOV ANNEX C CASE STUDIES This page has been left blank for double-sided copying. IDIN AND C-INNOVA IN COLOMBIA A CASE STUDY THIS PUBLICATION WAS PRODUCED AT THE REQUEST OF THE UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT. IT WAS PREPARED INDEPENDENTLY BY MATHEMATICA, AND THE AUTHORS INCLUDE AUDREY-MARIE MOORE, EDITH FELIX, AND JOSH MEUTH ALLDREDGE. Cover photos clockwise from top left: AidData trains enumerators for a ground-truthing exercise (AidData); RAN￾supported innovator helps a community farmer (ResilientAfrica Network); Participants at the IDIN Zero Waste summit (MIT IDIN/C-Innova); Women test a CITE-evaluated water filter (Sydney Beasley MIT CITE). IDIN AND C-INNOVA IN COLOMBIA: A CASE STUDY INTRODUCTION AND CONTEXT MIT IDIN conducted extensive technology innovation work in Colombia through the International Development Design Summits (IDDS). IDDS is an "intense, hands-on, community-based design training." It brings together a diverse group of people to teach them the co-creative design process and prototype low-cost technological solutions to improve people (IDIN 2020). IDDS began as an MIT D-Lab program before the university received funding to launch IDIN. In 2013, the summits expanded under the supporting structure of the HESN-funded IDIN. This case study highlights C-Innova – a permanent hub in Colombia created by HESN and IDON in Colombia to continue spurring the innovation process. Local participants and IDIN micro-grant recipients decided to establish C-Innova as a hub for ongoing innovation, workshop, and summit activities after a 2015 summit in Colombia that focused on improving waste pickers' livelihoods streamlining recycling processes (Zero Waste). C-Innova engaged in the following summits. IDDS WORKSHOP TOPIC LOCATION IDDS 2015 Zero Waste Cali, Colombia IDDS 2016 Education Bogata, Colombia IDDS 2017 Climate Change Action Fusagasugá, Colombia IDDS 2018 Building Peace Guaviare, Colombia Coastal Territories Northern coast, Colombia C-Innova was an integral partner in the IDDS events. C-Innova ran two of the IDDS summits with IDIN funding and three using their resources. Each workshop was customized based on location, participants, and themes related to local issues. The focus of the workshops ranged from waste management and climate change to social reconciliation. Over time, C-Innova gained more ownership over innovation and design activities. Specifically, the center provided follow-up support for innovator communities after summits to help them continue their prototyping and advance their technological solutions' potential impacts. For example, C-Innova maintained collaborative relationships with young innovators in Cali after designing an improved waste transport system and plastic recycling technologies. The center also built lasting relationships among community members, designers, and faculty and students at several universities in Colombia. Faculty members who formed part of C-Innova were inspired to create Distancia Cero, a university student program embedded in rural communities, and focused on developing co-design solutions to rural challenges alongside community members. IDIN/C-Innova. “In 2015, we realized we needed an innovation center that could respond to the need to follow up with or continue [IDDS] projects. The innovation center, C-Innova, is itself a product of an IDDS event.” – C-Innova leader C.5 | PERFORMANCE EVALUATION OF HESN USAID.GOV HESN IDIN initially funded C-Innova through core funding. C-Innova delivered programming on behalf of IDIN and produced a version of the D-Lab Creative Capacity Building (CCB) curriculum adapted to the local context. After core funding ran out, C-Innova sought funds through academic institutions and grants. C-Innova recently merged with Distancia Cero, forming a new body is called Diversa. C-Innova is an example of sustainable HESN activities. WHAT WORKED WELL? C-Innova was inherently interdisciplinary organization. Its membership included professionals from engineering, design, academic research, and community capacity-building fields. It also directly engaged (and valued the experiences of) community members in design summits. This structure produced the most significant benefit of the IDIN/C-Innova work in Colombia: a mindset change among participants in the summits and other activities, such as CCB. Specifically, participants (particularly innovators) stated that they experienced: • Increased confidence in their design abilities, • A stronger desire to collaborate with others on the design and development of innovations, • Stronger relationships with people outside of their typical sphere of work, • An appreciation of IDIN and C-Innova for providing space for such work. WHAT WERE THE MAIN CHALLENGES? C-Innova encountered two principal challenges that impeded their work. First, the organic growth of C-Innova produced organizational instability. As a rapidly-evolving organization in a complex network, C-Innova had to navigate differences of vision between itself and D￾Lab, IDIN, Distancia Cero, and allied Colombian universities. The differences led to a fissure within C￾Innova, causing several members to leave and form a new institution at UNAL, the national university. While this separation did not threaten either group's work, it did reflect the growing pains of C-Innova. Second, C-Innova experienced constraints in working with D-Lab. While the contractual relationship between D-Lab and C-Innova was generally functional, members of the Colombian-based center expressed frustration that D-Lab did not continue to update and customize program curricula to reflect lessons learned and best practices. C-Innova also voiced concern that D-Lab and IDIN let the IDDS websites, a key point of dissemination and engagement, become outdated. Finally, after their initial role as the implementer of IDDS, C-Innova felt that they took on much more creative responsibility in local innovation programming and noted they would have appreciated treatment as an equal to D-Lab, instead of as a subsidiary or support. “There are a lot of events that one goes to, especially as an academic, that only last for a moment.” But with IDDS, participants “felt a change that would last their whole lives. There was a deep learning.” – Innovator C.6 | PERFORMANCE EVALUATION OF HESN USAID.GOV LESSONS LEARNED The greatest lesson from this case study relates to the timeliness and utility of HESN partner efforts. The uptake and enthusiasm among summit participants demonstrated strong local support for the work, and both organizations capitalized on that to build local knowledge and capacity around innovation and collaboration. C-Innova and IDIN organized summits and workshops that impacted thousands of beneficiaries' mindsets and skills by seizing the momentum and reading the interests and needs of target communities. The second lesson is that engaging faculty and students with community members can produce long-term shifts in both communities' trust in academia and students' passion for co-designing innovations outside the classroom. As a spin-off of C-Innova, Distancia Cero placed university students in communities facing challenges in agriculture, health, infrastructure, and other areas. Host communities illuminated complex challenges and shared their experiences and constraints with students, who brought technical knowledge and methodologies to support innovation. Distancia Cero leadership noted that these interactions built trust, rapport, and long-term relationships among various parties. These parties were able to support the design and development of innovations using contextualized expertise. Now, the evolved form of C-Innova and Distancia Cero (Diversa) continues implementing this work. C.7 | PERFORMANCE EVALUATION OF HESN USAID.GOV This page has been left blank for double-sided copying. DIL CASH-BENCHMARKING STUDY IN RWANDA A CASE STUDY Cover photos clockwise from top left: AidData trains enumerators for a ground-truthing exercise (AidData); RAN￾supported innovator helps a community farmer (ResilientAfrica Network); Participants at the IDIN Zero Waste summit (MIT IDIN/C-Innova); Women test a CITE-evaluated water filter (Sydney Beasley MIT CITE). THIS PUBLICATION WAS PRODUCED AT THE REQUEST OF THE UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT. IT WAS PREPARED INDEPENDENTLY BY MATHEMATICA, AND THE AUTHORS INCLUDE AUDREY-MARIE MOORE, EDITH FELIX, AND JOSH MEUTH ALLDREDGE. DIL CASH-BENCHMARKING STUDY IN RWANDA: A CASE STUDY INTRODUCTION AND CONTEXT Cash-benchmarking seeks to assess structured programming's cost-effectiveness against the direct disbursement of funds to potential program beneficiaries. In 2013, USAID staff in Rwanda began exploring the potential for research under HESN using cash-benchmarking to assess their programs. Specifically, USAID staff worked with the University of California, Berkeley's Development Impact Lab (DIL), to 'benchmark' a major child malnutrition program against what would have occurred if USAID disbursed its cost directly to beneficiaries. Funding for the program itself came from a Global Development Alliance mechanism with Google, representing the private sector, contributing half and USAID/Rwanda contributing the other half. The NGO GiveDirectly administered this program and evaluation support came from the Global Development Lab within USAID and two additional Missions. “The broad research question was and still is, how does what we are currently doing compare, measured by impact per dollar, to what would happen if we as an Agency moved and gave money directly? How does our machine for contracting add value?” – USAID staff WHAT WORKED WELL? The expertise of DIL researchers and the support of USAID Washington and Mission staff produced a high-quality and relevant evaluation. The study1 garnered substantial attention within the USAID Rwanda Mission, USAID Washington, and in Congress. This attention resulted in the following impacts. • USAID began a series of productive internal discussions around the value of programming compared to cash. • Cash benchmarking is now on checklists for large USAID grants that move through the Senior Obligation Agreement Review (SOAR). • At least two Missions made concrete policy or program decisions using information from the cash benchmarking studies. • The study also prompted additional evaluations using cash benchmarking in Malawi, Liberia, and the Democratic Republic of the Congo. • The results influenced the establishment of a cost working group by USAID/PPL. 1 USAID released the results of the study. Neither the holistic intervention nor the smaller cash transfers moved the needle much on nutrition. The tailored program did increase savings, while the small cash transfer allowed individuals to repay debt and accumulate assets. Larger cash transfers (about $530 per household), however, had substantial effects. Households increased their productive assets by 76 percent, saved 60 percent more, and were able to consume 32 percent more than in the past. They were able to buy more varied food for their families. Children in these households were taller, weighed more, and were less likely to die early. WHAT WERE THE MAIN CHALLENGES? The cash-benchmarking work encountered three problems. First, with the original Rwanda study and subsequent cash-benchmarking evaluations, a lag in years between data collection and publication meant that the highly-anticipated results were absent during intervening policy discussions. Second, the study arm based in the DRC ran into problems when new DIL researchers assigned to the evaluation contested the previously agreed-upon study design. Months of negotiation dwindled researchers' time and resources. It also showed USAID staff the importance of including all parties early in the evaluation to agree on key design elements. Third, cash-benchmarking is a controversial research area because it contests the premise that bilateral contracting and programming add value to aid dollars. Given the long history of USAID programming and attitudes that support those programs, policy uptake has been slow to take hold. According to one USAID staff member, conversations around this topic require "a complicated dance and kind of diplomacy." Brooke Patterson / USAID. LESSONS LEARNED The work on cash-benchmarking demonstrated how vital it is that USAID staff and technical academic researchers communicate early and often with one another. With stronger communication throughout the study, the team may have been able to avoid delays in finalizing the study and producing results. In turn, earlier results might have allowed policymakers to adopt the lessons and implications of the cash-benchmarking study more quickly. Another lesson is that cash-benchmarking should not be viewed as threat to traditional aid approaches. Instead, advocates for the research suggest that the different means of spending program resources generate fundamentally different types of benefits (such as increases in household savings versus increases in productive assets). With a more nuanced understanding of these differences, USAID can design programs that are more likely to deliver specific desired impacts. “The first study got a huge amount of attention within the Mission and the outside, and support from the Hill as well…people refer to ‘the study’.” – USAID staff CITE AND THE GEORGE WASHINGTON UNIVERSITY FEED THE FUTURE MARKET SYSTEM MONITORING ACTIVITY A CASE STUDY THIS PUBLICATION WAS PRODUCED AT THE REQUEST OF THE UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT. IT WAS PREPARED INDEPENDENTLY BY MATHEMATICA, AND THE AUTHORS INCLUDE AUDREY-MARIE MOORE, EDITH FELIX, AND JOSH MEUTH ALLDREDGE. Cover photos clockwise from top left: AidData trains enumerators for a ground-truthing exercise (AidData); RAN￾supported innovator helps a community farmer (ResilientAfrica Network); Participants at the IDIN Zero Waste summit (MIT IDIN/C-Innova); Women test a CITE-evaluated water filter (Sydney Beasley MIT CITE). CITE AND THE GEORGE WASHINGTON UNIVERSITY FEED THE FUTURE MARKET SYSTEM MONITORING ACTIVITY: A CASE STUDY INTRODUCTION AND CONTEXT Throughout the implementation of HESN, CITE defined and refined an evaluation methodology for the complex systems that deploy technologies in low and middle-income countries (LMICs). In 2016, USAID/Uganda sought assistance from CITE to better understand Uganda's agricultural market systems and measure outcomes of its Feed the Future Value Chain (FTF-VC) project. The Mission aimed to strengthen the implementation of the FTF-VC project and inform policy and investment decisions. In January 2016, CITE and George Washington University (GW) secured a $3.5 million, four-year buy-in with USAID/Uganda. Under this buy-in, they are developing new approaches to assess market facilitation activities in the FTF-VC project and assist in measuring outcomes. The intent was to complement the monitoring and evaluation efforts of individual activities by using other methods to assess how the project portfolio's combination enabled systemic change in markets. The buy-in included a series of tasks such as reviewing literature related to market facilitation, measurement frameworks, systems thinking, and technological adaptation. Primary data collection, analysis, and modeling of market subsystems to invent and enhance market monitoring indicators and develop market system maps informed by analysis and modeling. Researchers hoped that the activity would contribute to FTF-VC learning and decision￾making via the evidence base and fill gaps in market systems measures. WHAT WORKED WELL? The CITE and GW research team brought methods, markets, and local expertise. The research team included technically competent and experienced people who advised USAID throughout the study. The team also had the support of an advisory board with methodological and market experts. Through this expertise, the team brought new ideas to the Mission and helped staff think outside the box and determine where to focus on for the next phase of the Feed the Future program. The partnership benefited from a mixed-method and flexible market systems approach. The team used a mixed-methods approach that included mapping, modeling, and qualitative and quantitative data collection and analyses to capture market systems' complexities. For example, the team followed the market system's mappings with small surveys to provide more details about the activity. Additionally, as USAID/Uganda's priorities evolved, the team also evolved its approach to respond to Mission needs. For example, the Mission had been doing work in resilience and was interested in mapping resilience, so the research team also created a resilience map for the Mission. The partnership has strong support from the USAID HESN AOR and USAID/Uganda activity manager. The USAID/Uganda team valued the time and support provided by the AOR throughout the partnership. The team recognized that the partnership succeeded because of the Mission's strong interest in the work and their ongoing support. Clear and consistent communication was a key component of what made the partnership work well. “Whatever was the magic formula of great IP and an AOR has been really instrumental. We’re able to make a very esoteric, hard-to-understand thing real and useable by practitioners in the Mission and then among our implementing partners.” – USAID staff C.15 | PERFORMANCE EVALUATION OF HESN USAID.GOV Overall, USAID/Uganda found CITE and GW's outputs pertinent and easy to interpret. The Mission used the information and knowledge documented in CITE's mapping and reports produced as part of the Market System Monitoring (MSM) activity to write the Feed the Future country plan, which runs from 2017 and 2022. The Mission also used the results of the study to design project activities. Results from the study contributed to how the Mission thought about monitoring evaluation and learning based on the activity toolkit, which emphasized identifying, measuring, and interpreting systems indicators. Workshop summaries and reports, as well as behaviors-relationships-conditions maps from the partnership, are available here. The Mission used the product because it was relevant to their needs and timely. WHAT WERE THE MAIN CHALLENGES? While all partners felt that the buy-in was a success, there were a few challenges. First, the research team sometimes struggled to meet the deadlines for submitting outputs. The delays meant that the Mission had to use early draft information for decision-making instead of the final results. Second, some Mission staff indicated that they struggled to interpret the more complicated graphics included in reports, such as radar charts. Adding clear, detailed explanations or thinking about different graphics to convey the information would have been helpful. Finally, Mission staff also felt that the research activity needed more day-to-day contact so it would have been helpful to have a Uganda-based point of contact for the Mission to easily call or meet with in person. LESSONS LEARNED Labs need to be flexible and responsive to the needs of USAID Missions. Flexibility is vital for adapting to changing Mission priorities, especially for more extended partnerships. To enable the full use of outputs by Missions for planning, programming, and decision-making, teams also need to be responsive to the deadlines communicated by Missions; otherwise, results may be set aside and not used because the window of opportunity passes. The AOR and Mission activity manager plays a crucial role in ensuring the success of a partnership. The AOR should help Labs understand the specific needs of a Mission and navigate the different USAID requirements. To ensure the activity manager's continued interest and engagement, the research team should adapt to their preferred communication method and frequency and facilitate their interpretation of output results and findings. C.16 | PERFORMANCE EVALUATION OF HESN USAID.GOV ANNEX D EVALUATION DESIGN REPORT This page has been left blank for double-sided copying. Anchor Final The Higher Education Solutions Network (HESN): Evaluation Design Report October 3, 2019 Audrey Moore, Edith Felix, and Josh Meuth Alldredge Submitted to: U.S. Global Development Lab Ronald Reagan Building and International Trade Center 1300 Pennsylvania Ave NW Washington, DC 20004 Attn: Karen Fowle M&E Specialist Contract number: AID-OAA-A-16-00025 Submitted by: Mathematica 1100 First Street, NE, 12th Floor Washington, DC 20002-4221 Phone: 202-484-9220 Fax: 202-863-1763 Project Director: Delia Welsh D.3 | PERFORMANCE EVALUATION OF HESN USAID.GOV This page has been left blank for double-sided copying. CONTENTS I. INTRODUCTION ................................................................................................................................... 1 II. CONTEXT ............................................................................................................................................. 2 III. MOTIVATION FOR LONG-TERM EVALUATION................................................................................. 4 IV. RESEARCH QUESTIONS .................................................................................................................... 5 V. DATA COLLECTION METHODOLOGY AND PLAN ............................................................................ 6 VI. ANALYSIS PLAN ................................................................................................................................ 17 VII. EVALUATION RISK AND LIMITATIONS ............................................................................................ 21 VIII. ADDITIONAL OUTPUTS..................................................................................................................... 22 IX. TIMELINE ............................................................................................................................................ 23 X. REFERENCES .................................................................................................................................... 24 APPENDIX A: OUTLINE FOR HESN INSTRUMENTS .............................................................................A.1 APPENDIX B: LIST OF DOCUMENTS TO REQUEST FROM HESN AORS ...........................................B.1 Tables V.1 Overview of evaluation questions, key outcomes, and data sources .............................................. 7 V.2 Types of documents to be reviewed .............................................................................................. 11 V.3. Plans for quantitative and qualitative data collection ..................................................................... 14 IX.1. HESN implementation schedule .................................................................................................... 23 Figures V.1. Overview of data sources................................................................................................................. 7 VI.1. Conceptual framework for understanding DOCs ........................................................................... 18 D.5 | PERFORMANCE EVALUATION OF HESN USAID.GOV This page has been left blank for double-sided copying. I. INTRODUCTION The Expanding the Reach of Impact Evaluation (ERIE) consortium has partnered with USAID’s Global Development Lab to assess the long-term impacts of the Higher Education Solutions Network (HESN) and the Partnerships for Enhanced Engagement in Research (PEER) program. 1 ERIE partners on this evaluation include staff from HESN-funded projects, researchers supported through PEER, USAID Missions, the Notre Dame Initiative for Global Development (NDIGD), and Mathematica. NDIGD will lead the evaluation of PEER and Mathematica will conduct the evaluation of HESN. The following evaluation design for HESN lays out the context, motivation, data collection plan, analytical methods, evaluation risks and challenges, additional outputs, and timeline for the evaluation. 1 USAID directs Partnerships for Enhanced Engagement in Research to support researchers and scientists in developing countries as they seek solutions to development challenges. Through PEER and partnerships with other USG agencies, scholars in more than 50 countries have gained access to $50 million for research projects in water management, agriculture, energy, and other sectors. D.7 | PERFORMANCE EVALUATION OF HESN USAID.GOV II. CONTEXT The Higher Education Solutions Network (HESN) is a cooperative agreement between USAID and seven top universities designed to channel the ingenuity of university students, researchers, and faculty towards global development through a set of core funds and buy-in activities from USAID Missions and Bureaus. The three goals of HESN are to (1) accelerate the use of innovative technologies and approaches to evaluate development, (2) catalyze an interdisciplinary research environment, and (3) support evidence-based development decision making. At the launch of the partnership in 2012, each university established a Development Lab— including two at the Massachusetts Institute of Technology (MIT)—and created a network of researchers with a mission to revolutionize development through science and innovation. The eight HESN Development Labs research, incubate, test, and accelerate solutions to development challenges, from drought and food security to data availability and financial inclusion. The Labs work in partnership with local universities, organizations, and communities to deliver the most useful research for development practitioners and the greatest impacts for development program beneficiaries. The network has created a vibrant framework of cooperation between USAID Missions, local actors, development professionals, and academics to tackle the complexities of modern-day development challenges. Soon after initiation of their awards, the HESN 1.0 Labs began working with USAID Missions to address Mission-relevant development challenges across the sectors. The HESN partnership funded these projects in three ways: through (1) core funding, (2) Mission funding (as a buy-in), or (3) co-funding by both HESN and Missions. A midterm evaluation of HESN showed that agency and Mission staff were interested in the easy access to academic research and evaluations that programs such as HESN provided to them (Amulya et al. 2016). The evaluation also found that the HESN Labs were performing well in the field across their three objectives. The results of the midterm evaluation are summarized below.  The Labs met or exceeded 79 percent of their 34 indicator targets in the first three years, with the most robust performance in the areas of “improving data quality, access, and analytics to advance evidence-based development decision making.” - For example, Labs produced 265 data sets and offered 478 analyses, maps, and consultations to USAID Missions and other HESN stakeholders. - Labs also improved year-over-year in 71 percent of key indicators.  Ninety percent of the stakeholders interviewed during the midterm evaluation confirmed the benefits that the Labs provided to them. Benefits included: - Dialogues with leading researchers on development challenges, and discussions on how to solve them. - Opportunities for researchers in developing countries to build networks and capacity with US research labs. D.8 | PERFORMANCE EVALUATION OF HESN USAID.GOV - Training of students in essential tools for development innovation, including cultural awareness, lean experimentation, and design thinking. - Improved data access for researchers to conduct analysis and provide development actors with results. The research expertise of HESN Labs, as well as the services Lab staff offer to Missions and partners, varies across partnerships. The following examples show the breadth of the work conducted by the HESN Labs.  At MIT, the Comprehensive Initiative on Technology Evaluation (MIT CITE) and the International Development Innovation Network (IDIN) offer multiple research products. CITE examines the impacts and potential of technology in terms of suitability, scalability, and sustainability. At IDIN, lab members host co-design summits to formulate solutions to challenges in communities, research innovation ecosystems, and provide support for practitioners implementing solutions to address poverty, health, and food security issues (MIT CITE 2017; IDIN 2016).  AidData, a research lab at the College of William and Mary, focuses on improving the availability and utility of data for development advocates, researchers, and decision makers. As a HESN Lab, AidData produces high quality, geo-coded subnational data sets on aid flows and projects. Through core funding and buy-ins, USAID Missions and government agencies in low- and middle-income countries (LMICs) like Niger, Uganda, and Nepal benefit from increased data access and even training on geospatial data management and analysis. For example, AidData geocoding allowed Nepal’s Ministry of Finance to see the universe of aid projects in the country and address regional imbalances in development finance (AidData 2017).  The ResilientAfrica Network (RAN) works through four HESN Resilience Innovation Labs (RILabs) and numerous universities across the continent to identify and support innovations that strengthen community resilience. Using an innovation challenge model, the Labs select, support, and fund innovations that address regional issues, such as vulnerability to drought or HIV/AIDS. These innovations include digital financial literacy trainings and E-Health for Refugees, a system for providing reproductive health information to refugees in Rwanda (RAN 2017). Recent interviews with staff at these Labs suggest the transition from the use of core funds to pay for activities to using buy-in agreements went smoothly. The buy-in agreements—in which labs provide research services for specific USAID Missions, non-governmental organizations (NGOs), philanthropic organizations, and multilateral institutions—started before the core funding expired and provided continuity for the HESN mechanism. For example, MIT D-Labs now offers research services to Food for Peace on critical storage and distribution issues and the RAN network provides a variety of evaluation and research products to philanthropic and multilateral organizations such as the Rockefeller Foundation and UN Women. These partnerships were facilitated mainly by Agreement Office Representatives (AORs), who communicate the capacities of the HESN Labs to potential partners and report back on the needs of Missions and practitioner organizations so that Labs can continue supporting development work in the field. D.9 | PERFORMANCE EVALUATION OF HESN USAID.GOV III. MOTIVATION FOR LONG-TERM EVALUATION As the current programs have shifted more heavily towards Mission engagement, the Center for Development Research (CDR) is interested in understanding the utility and impact of HESN 1.0 projects on USAID Missions, local LMIC partners, and researchers. USAID wants to understand the conditions and models that generate effective partnerships with USAID Missions, how the mechanism worked to improve the use of research findings, and its potential LMIC policy impacts. The new CDR programs, such as the Long-Term Assistance and Services for Research (LASER) and the Research Technical Assistance Center (RTAC), also engage researchers and universities in providing long-term, technical, and research assistance to USAID. The mechanisms are designed to allow Missions, bureaus, or independent offices rapid access to a network of international universities and associated researchers interested in refining and solving development challenges in partnership with USAID and local stakeholders. The new CDR programs are based, in part, on lessons learned through Mission engagement with HESN 1.0; CDR modeled elements of the new programs to facilitate increased USAID access to quality researchers and grants. However, it is unclear whether and how USAID operating units use the outputs generated by the research grants to help them with future programming and decision making. An improved understanding of the relationships between the various elements of the HESN partnerships (funding source, type of buy-in, and so on) and the utility of the outputs for the Missions could greatly benefit the new programs. D.10 | PERFORMANCE EVALUATION OF HESN USAID.GOV IV. RESEARCH QUESTIONS The long-term evaluation of HESN seeks to answer the following questions. 1. To what extent have Mission partners applied learnings from HESN 1.0 research or outputs to their programs? a. What was the usefulness of the buy-ins to their programs (that is, output was pertinent, relevant, and timely)? b. What was the utility of the buy-ins to their programs (that is, the degree to which policymakers and stakeholders used the outputs)? c. To what extent did the usefulness and utility of the buy-ins vary by funding source? 2. What was the perceived utility of the partnership among stakeholders (that is, USAID Mission, Labs, policymakers)? 3. What structural/institutional elements of the partnership contribute to different levels of usefulness to Mission programming and decision making? Utility? 4. What process elements of the partnership contribute to different levels of usefulness to Mission programming and decision making? Utility? 5. To what extent has HESN contributed to changes at higher education institutions (HEIs) or in HEI networks that increase their engagement in international development? a. To what extent would any changes be sustained? Why? Or why not? D.11 | PERFORMANCE EVALUATION OF HESN USAID.GOV V. DATA COLLECTION METHODOLOGY AND PLAN Mathematica will conduct a mixed-methods, ex-post process evaluation that (1) explores how HESN funds were used to generate research; (2) determines whether the research was used to inform policy; (3) examines whether the funding mechanism (for example, CDR funded, Mission funded, or co-funded) contributed to any differences in the type of research produced by HESN Labs and whether Missions and other relevant stakeholders used the research; (4) helps contextualize outcome results by describing the geographic, social, and policy environment in which several Labs work; and (5) provides a deeper understanding of how Mission engagement around the HESN partnerships contributed to differences in the uptake of research outputs by Missions and other key stakeholders. The ex-post process evaluation will conduct one round of data collection and draw on document review, administrative data (that is, monitoring and reporting indicators), online surveys, and interviews with key stakeholders to answer the key evaluation questions. As shown in Figure V.1, we will follow these steps to complete the data collection process: 1. Review documents and administrative data to better understand the work conducted by each HESN Lab. In Appendix B, we include a list of documents to request from HESN Labs. 2. Use the results of the document review to refine the content and list of respondents for the online survey. 3. Conduct an online survey to collect data from the following respondent groups: staff from the eight HESN Labs, all USAID Missions and Operating Units independent of whether they used the mechanism, university leadership, and principle investigators (we provide more details on each respondent group in the subsections that follow). We will develop different online surveys for each respondent group. 4. Use the survey data to inform the development of interview protocols and our selection of a subsample of survey participants from each respondent group for in-person and phone interviews. 5. Conduct qualitative fieldwork with projects from three Labs (to be chosen from AidData, MIT, Makerere University, and Berkeley). D.12 | PERFORMANCE EVALUATION OF HESN USAID.GOV Figure V.1. Overview of data sources In Table V.1, we summarize the key outcomes, themes, and data sources that we will use to answer each research question. Table V.1 Overview of evaluation questions, key outcomes, and data sources2 Evaluation question Key outcomes and themes Data sources 1. To what extent have Mission partners applied learnings from HESN 1.0 research or outputs to their programs?  What was the usefulness of the buy￾ins to their programs (i.e., output was pertinent, relevant, and timely)?  What was the utility of the buy-ins to their programs (i.e., the degree to which policymakers and other stakeholders used the outputs)?  How and to what extent HESN Lab research outputs were used in partner development of strategies, design, implementation, monitoring, and or evaluation of development projects  Partners request for additional research (number, funding source, and type) from HESN Development Labs  Number of stakeholders engaged in problem-solving with each HESN Development Lab*  How and to what extent HESN Lab research outputs were used by policymakers/other stakeholders for programming, policymaking, and/or decision making  Document review  Online survey3  Interviews (USAID Mission staff, USAID operating unit staff, policymakers, other HESN partnership stakeholders, HESN Lab staff)  Relevant administrative data (monitoring & evaluation [M&E] reporting) 2 All instruments for the HESN evaluation will be adapted for each respondent group. 3 We will adapt the online surveys to ensure they encompass the specific type of technical work and research focus of each Lab. Content and respondents informed by survey results In-person interviews with HESN stakeholders in three countries Phone interviews with a sample of each type of respondent, including staff from three Labs Interviews Content and respondents informed by document review Sent to Labs, Missions, Operating Units, university leadership, and principal investigators Different modules for each type of respondent Online survey Key program and research documents from the eight Labs M&E reporting data from the eight Labs Document review D.13 | PERFORMANCE EVALUATION OF HESN USAID.GOV Evaluation question Key outcomes and themes Data sources  To what extent did the usefulness and utility of the buy-ins vary by funding source?  How and to what extent specific policies/legislations were influenced by HESN Lab research outputs  Facilitators and barriers (including geographic, social, and policy environment) to use of HESN Lab outputs by Mission partners, policymakers, and other stakeholders  Incentives and disincentives to use of the mechanism 2. What was the perceived utility of the partnership among stakeholders (i.e., USAID Mission, labs, policymakers)?  Perceptions of HESN program and tools  Awareness and visibility of the role of science and engineering in poverty alleviation  How and to what extent HESN partnerships were useful  Aspects of HESN mechanisms contributing to strong partnerships  Facilitators and barriers to HESN partnerships  Lessons learned for improving future partnerships on similar mechanisms  Document review  Online survey  Interviews (USAID Mission staff, USAID Operating Unit staff, policymakers, other HESN partnership stakeholders, HESN Lab staff) 3. What structural/institutional elements of the partnership contribute to different levels of usefulness to Mission programming and decision making? Utility?  Partnership structural/procedural elements (e.g., requirements, roles and responsibilities, timeline, political will) facilitating usefulness/utility of HESN research/outputs to Mission programming and decision making  Incentive structure  Facilitators and barriers to use of the mechanism  Document review  Online survey  Interviews (USAID Mission staff, USAID operating unit staff, HESN Lab staff) 4. What process elements (technical aspects) of the partnership contribute to different levels of usefulness to Mission programming and decision making? Utility?  Partnership process elements (e.g., actions/steps, inputs, and outputs), facilitating usefulness/utility of HESN research/outputs to Mission programming and decision making  Partnership process elements inhibiting usefulness/utility of HESN research/outputs to Mission programming and decision making  Incentive structures  Document review  Online survey  Interviews (USAID Mission staff, USAID operating unit staff, HESN Lab staff) D.14 | PERFORMANCE EVALUATION OF HESN USAID.GOV Evaluation question Key outcomes and themes Data sources 5. To what extent has HESN contributed to changes4 at HEIs or in HEI networks that increase their engagement in international development?  To what extent would any changes be sustained? Why or Why not?  The number and characteristics of:  New development-related classes or disciplines created by departments with human, financial, or institutional resources contributed by HESN Development Labs*  Students embedded in development institutions abroad as summer fellows or research assistants through HESN Development Lab projects  International development degree programs (majors, minors, and certifications) and number of students in each  Students with training and experience in development research  Development professionals proficient in data management and use as a consequence of HESN Lab-funded technical assistance  Participants in hubs, summits, and other problem-solving institutions created with human, financial, or institutional resources contributed by HESN Development Labs  MOUs or other agreements signed with public sector, private sector, local, community partners, and one HESN Development Lab*  Stakeholders engaged in problem solving with one HESN Development Lab*  Development stakeholder institutions that use research output from one HESN Development Lab in their work, and how they use it*  Research-based mentorships and field￾based practica, fellowships, internship opportunities  Newly hired faculty interested in development challenges  Faculty/researcher consulting assignments related to international development  Development-focused collaborations and grants  Document review  Online survey  Interviews (with HESN lab staff, HEI leadership, researchers, and students), focus groups (with HEI students)  Relevant administrative data (M&E reporting) 4 If administrative data are available on the quantitative outcome indicators for the pre-project time period, we will collect the data and determine if we can establish any trends in associated changes. D.15 | PERFORMANCE EVALUATION OF HESN USAID.GOV Evaluation question Key outcomes and themes Data sources  Perceptions of changes in donor funding linked to HESN  The extent to which HESN Labs have sought external funding Notes: * HESN M&E indicator Document review. Our evaluation team will conduct a document review to help us understand the work Labs have completed to date. We will hold meetings with staff from each of the eight Labs to gather their materials. We will then develop a document review protocol to review the most critical documents related to HESN (for example, program documents, program plan, progress reports/quarterly reports, evaluation reports, sustainability plans, original application submitted for funding, publications from principal investigators, policy documents, and conference proceedings). We will synthesize the findings using a document review protocol and write a summary memo of the results. Our team will then code the materials into NVIVO. Two key tools will assist in this process: 1. Bibliometric approach. Where possible and relevant, we will use a bibliometric approach for the document review. This approach allows our team to understand the quantity and quality of literature produced by HESN Labs and partners. Specifically, we will use citation count and h-index5 tools to (1) quickly find seminal reports on the HESN mechanism, (2) gather granular data on the research output of each of the Labs, and (3) understand the impact of their research reports. We plan to use the bibliometric functionalities of Google Scholar to examine the published literature produced by HESN labs and researchers, and we will manually apply similar tools in our review of the grey literature. 6 2. Information extraction. We will develop and use a document preview protocol to extract essential information from each document. Key code items that we will extract include document type, target audience, project and Lab partnerships, any changes in research networks and student involvement, changes in faculty and researcher hiring, core evaluation results (if appropriate), partnership with local government, contribution to policy and/or regulation changes, lessons learned, sustainability plans, and challenges to the HESN mechanism’s operation. With these items extracted, we will synthesize key trends and lessons from HESN-related literature. During the document review process, we will also create a list of all the HESN-funded stakeholders with the contact information (email, phones, Skype) for future use in data collection. We will also link the research study/grant conducted at each Lab with the type of funding mechanism (CDR, Mission, co-funded) to explore any potential patterns in usage of the mechanism and type of research. We will develop summaries of the main documents and code 5 H-index is the number (h) of an author's studies that have been cited h or more times. It gives a measure of an author’s impact in his or her field. 6 Due to differences in implementation timelines, as well as cross-discipline variation in how long publication takes in different fields, we may not be able to find publications for some HESN projects. D.16 | PERFORMANCE EVALUATION OF HESN USAID.GOV the summaries for the qualitative analysis. In Table V.2, below, we list the key types of documents that we plan to review for the HESN evaluation. Table V.2 Types of documents to be reviewed Programmatic documents Research documents HESN Lab applications Evaluation proposals (buy-ins and other) Cooperative agreements and MOUs Monitoring and evaluation plans Annual and longer-term work plans Evaluation reports HESN annual reports Conference proceedings Quarterly reports Policy briefs Sustainability plans Other research products Administrative data. We will use relevant data collected for monitoring and evaluation (M&E) reporting by Missions and all eight HESN Labs to help us answer the relevant evaluation’s research questions. The M&E reporting data cover the annual indicator data submitted by each Lab during the period of performance. The period of performance under the core HESN Cooperative Agreements with Labs began in fall 2012 and concluded in fall 2017; however, some Labs signed agreements with USAID Missions and Operating Units that extended their period of performance. If data for the pre-project time period are available, we will collect the information and determine if we can conduct a longer trend analysis with the data. In Table V.1, we identify some of the key outcomes that we will measure through administrative data. Online survey. Results of our document review will help us finalize the online survey, which we will use to help us cost-effectively collect quantitative data from all eight HESN Labs and stakeholders involved in the HESN program. The online survey includes a series of multiple￾choice questions and Likert scales, which can be converted into quantitative data. We will include as many numeric versions of the variables as possible to ensure we have disaggregated measures of responses. Mathematica will conduct a separate online survey with each HESN stakeholder group so that we can triangulate the results. We will collect data on a variety of outcomes and themes, including stakeholder perceptions around the utility and usefulness of the HESN mechanisms, whether the research was used to inform policymaking, and why (or why not) the research was used by stakeholders. In Table V.1, we identify the key outcomes and themes that we will cover through the online survey. The online survey will be pilot tested to ensure that wording and syntax are culturally appropriate. We will work with the Labs to select a small group of diverse stakeholders across multiple countries to take the pilot survey. On completion of the pilot survey, we will follow up with a phone call to each person to discuss the survey and gather their input on the questions and format. This process will help us refine the questions for the final online survey. Following the pilot assessment, we will use existing platforms (for example, Survey Monkey, Google forms, Enterprise) to implement the online survey. Surveys will then be sent out to USAID key staff at the each of the eight Labs, Mission and Operating Unit staff, policymakers, university D.17 | PERFORMANCE EVALUATION OF HESN USAID.GOV leadership, students, researchers, and other key HESN stakeholders identified through our document review and conversations with Labs. We will download and clean the data on an ongoing basis during the data collection period. In-person interviews. We recognize that in-person interviews are the ideal way to collect information from the Labs, Missions, and other local HESN stakeholders. In-person interviews allow us to interact with the staff and probe in more depth to understand key issues. We propose to travel to Uganda, Ghana, and Colombia to collect the qualitative data. We have selected these countries because they demonstrate a range of engagement and utilization by USAID Missions. We propose Uganda because AidData, MIT CITE, and RAN identified it as a country where their HESN projects have had high uptake by USAID/Uganda and partners. We selected Colombia because AidData currently has one of the most substantial HESN buy-ins (valued at almost 1.5 million USD) with the USAID Mission in Colombia. The buy-in involves supporting USAID/Colombia with impact evaluation of the Colombia Regional Governance Activity (RGA), funded by USAID/Colombia in partnership with the government of Colombia. We propose Ghana as the third country because it also has a mix of core and buy-in activities; allows us to also gather data related to MIT CITE and IDIN work on the innovative technologies, which was part of the India core work; and allows us to collect the most amount of data linking to various HESN activities. We will travel to each country to meet with USAID staff, researchers, and other stakeholders such as local Mission and government counterparts to understand why Mission staff chose to use the mechanism, and the contributions of the research conducted by local researchers. . We will develop a semi-structured interview protocol with key questions and conduct in-depth interviews with the selected stakeholders. We will pilot test the university faculty, HESN Lab, and USAID staff protocols with Mathematica senior staff. We will select Mathematica staff who have previously worked for the donor community (i.e. USAID, MCC); serve as adjunct faculty at universities around the U.S.; and several of our senior researchers with project management experience. We are also exploring the possibility of working through Notre Dame to conduct some additional pilot-testing of the surveys with their university population. We will also gather examples of how the Missions have used USAID tools and approaches to increase the use of research and influence policy. We anticipate conducting 6 to 10 interviews in each country. The interviews will gather in-depth information on similar topics to the online survey (which will be used to develop the interview protocols). Interviews will focus on key themes and outcomes, including questions related to stakeholder perceptions of the utility of HESN partnerships and research outputs, as well as their influence on both USAID Mission programming and policy. In Table V.1, above, we present the key outcome and themes that we will cover through in-person interviews. Travel for this activity will be coordinated between PEER and HESN teams evaluation teams if the same countries are selected and schedule allows. Phone interviews. In conjunction with the in-person interview process, we will select a sample of additional USAID Mission staff who completed the online survey, and will conduct phone interviews based on the scripts for in-person interviews. We will work closely with USAID staff to locate any Mission or Operational staff who have relocated to other posts in the interim time D.18 | PERFORMANCE EVALUATION OF HESN USAID.GOV period. We estimate that we will conduct between 10 and 15 additional interviews,7 which accounts for nearly 25 percent of the grants allocated under the HESN mechanism. We will use maximum variation sampling to select participants, drawing from Missions that had high, medium, and low usage of the grants. This process allows us to learn more about factors that played into the decision-making process for selecting the mechanism. We will conduct those interviews by phone or video conference call. In Table V.1, above, we also identify the key outcomes and themes that we will cover through the phone interviews. We will record all in￾person and phone interviews as well as take detailed notes during the interview process. Based on the availability of resources, we will transcribe and code key interviews. If we are unable to transcribe all the interviews, we will write summaries of the interview findings and analyze the summaries in NVivo.  USAID Mission staff and operating units. We will interview key staff at USAID Missions and USAID Operating Units that were involved in a partnership with a HESN Lab. Staff will include those designated as the primary points of contacts (POCs) for the partnerships; those in charge of reviewing deliverables produced by the HESN Lab as part of the partnership; and those involved in securing buy-in for HESN projects or research outputs from government counterparts and other stakeholders. We will select interviewees based on the following steps:  We will use maximum variation sampling to select USAID Missions and operating units to interview. We will first create a sampling frame by categorizing each Mission and Operating unit into one of three categories: high, medium and low usage of the mechanism. The cut-off for each group will be determined based on the overall number of projects and to ensure that each group has similar numbers of projects in each one. The sampling frame will exclude the countries where we will conduct fieldwork (i.e. Uganda, Colombia, and Rwanda).  We will then randomly select 2-3 Missions/Operating Units from each category for telephone interviews. Each interview will take no more than 30-40 minutes each. The focus of the interviews will be to understand the usefulness and utility of HESN Lab research outputs and partnerships to their work, including their collaborations with governments and other key stakeholders. We will further probe about facilitators and barriers to the use of HESN Lab research outputs. Finally, we will ask them about the successes and challenges of the partnerships and lessons learned for future partnerships.  HESN Lab staff. We will interview HESN Lab leadership and core team members at each of the eight Labs that worked on the core, buy-in, and co-funded HESN projects. Staff will include those designated as project leads in charge of the planning and execution of all phases of their HESN projects and core team members supporting partner engagement and dissemination of findings. We will ask respondents about their perceptions of the usefulness of the HESN program, partnerships, and tools. We will also ask them about facilitators and barriers to the use of their outputs by partners, as well as their experience pursuing other funding opportunities. Similar to USAID respondents, we will ask them about the successes and challenges of the partnerships and recommendations for future partnerships. 7 We will adjust the total number of interviews based on the saturation point for this study. D.19 | PERFORMANCE EVALUATION OF HESN USAID.GOV  Policymakers and other HESN partnership stakeholders. We will interview key local policymakers and other partnership stakeholders in Colombia, Uganda and Rwanda to help us understand whether or not key stakeholders used research outputs produced by HESN Labs and how and why the results were used, as well as the social and policy environment. We will work with HESN Lab staff, the local university counterparts, and USAID Mission staff to identify the key local stakeholders to interview in each country. Policymaker respondents will include those involved with HESN partnerships and/or working in policy areas related to the research outputs produced by HESN Labs. We will also interview key representatives from the private sector, NGOs, and multilateral organizations that worked with HESN Labs. We will interview partnership stakeholders that bought into HESN projects or research outputs produced by HESN Labs. Stakeholder buy-in could include co-funding a HESN project, serving as an implementation partner, and/or utilizing HESN research outputs. We believe that these interviews will produce the most useful data when conducted in person so we will only carry out these interviews in the field.  Higher education institution leadership, students, and researchers. To assess the influence of the HESN mechanism on universities, we will conduct interviews with HEI leadership, graduate and undergraduate students, and faculty and researchers who worked in partnership with a HESN Lab or HEI international development opportunities impacted by a HESN Lab. Direct involvement with a HESN Lab could include participation in a HESN research project, training, or other relevant HESN Lab activity. HEI international development opportunities impacted by a HESN Lab could consist of development-related degree programs, courses, and partnership/funding opportunities. As discussed previously in this report, we will use the online survey to assess the reach and characteristics of HESN-linked activities in selected HEIs. After the survey, we will use preliminary results to inform our purposeful selection of participants for the interviews. We will conduct in-person interviews with these stakeholders in Uganda, Rwanda, and Colombia, including focus groups with students (if appropriate). We will use purposeful sampling to select and additional 6-8 HEIs for telephone interviews. These interviews (over the phone and in-person, as appropriate) will allow us to gather rich data from key leaders, students, and researchers on the quality, quantity, and sustainability of HESN-linked activities and programs. Table V.3. Plans for quantitative and qualitative data collection Data source Data collection method (approximate number) Evaluation questions Illustrative areas of focus USAID Mission staff  Online survey  In-person interviews (2–3 per country)  Phone interviewsa (5-6) 1, 2, 3, 4  Perceptions of the usefulness of HESN Lab research outputs  Use of HESN Lab research outputs in work  Additional research requested from HESN Development Labs  Use of HESN Lab research by partners  Facilitators and barriers to use of HESN Lab outputs USAID Operating Unit staff  Online survey  Phone interviews (5- 6) 1,2, 3, 4 D.20 | PERFORMANCE EVALUATION OF HESN USAID.GOV Data source Data collection method (approximate number) Evaluation questions Illustrative areas of focus  Perceptions of the usefulness of HESN program, partnerships, and tools  Successes and challenges of HESN partnerships, and lessons learned for future partnerships HESN Lab staff  Online survey (staff from all 8 Labs)  Phone interviews (8- 12 1,2, 3, 4, 5  Perceptions of the usefulness of HESN program, partnerships, and tools  Successes and challenges of HESN partnerships, and lessons learned for future partnerships  Facilitators and barriers to use of HESN Lab outputs  The pursuit of external funding opportunities  Awareness and perceptions of changes in HEI departments and programs with funding or support from HESN Policymakers  In-person interviews (2–3 per country) 1,2  Awareness and visibility of the role of science and engineering in poverty alleviation  Perceptions of usefulness of HESN research outputs and partnerships  Use of HESN Lab research outputs in work  Facilitators and barriers to use of HESN Lab outputs  Successes and challenges of HESN partnerships, and lessons learned for future partnerships Other HESN partnership stakeholders  In-person interviews (TBD) 1,2 University leadership  Online survey  Phone interviews  In-person interviews (2 per country) 5  Awareness of HESN program, partnerships, and tools  (Perceptions of) growth in university departments and programs affiliated with HESN  Changes in the engagement of students, faculty, and donors in support of the HESN mechanism  Goals and plans for the maintenance and expansion of international development partnerships, programs, and instruction Graduate and undergraduate students  In-person interviews (TBD)  Focus groupsb (TBD) 5  Perceptions of opportunities (mentorship, fellowships, internships, research assistantships) influenced by HESN programs and partnerships D.21 | PERFORMANCE EVALUATION OF HESN USAID.GOV Data source Data collection method (approximate number) Evaluation questions Illustrative areas of focus  Awareness and perceptions of changes in HEI departments and programs with funding or support from HESN  Career plans and perceptions of the influence of HESN Researchers  Online survey  Phone interviews (TBD)  In-person interviews (TBD) 5  Perceptions of opportunities (research collaborations, fellowships, and fieldwork) influenced by HESN partnerships  Perceptions of changes in HEI departments and programs with funding or support from HESN  Perceptions of changes in the quality and quantity of international development research publications from the HEI Notes: TBD = to be determined. a Phone interviews will be conducted with HESN participants who are located in countries outside of the three case studies or USAID staff who have rotated to other posts. b If local universities have sufficient students available, we will conduct focus groups or small group interviews with this population. D.22 | PERFORMANCE EVALUATION OF HESN USAID.GOV VI. ANALYSIS PLAN We will use both quantitative and qualitative software for data analysis and reporting. The analysis of qualitative information will be based on the variability and uniqueness of responses, using software for consolidating qualitative information. We will develop codes to guide the analysis. In addition to presenting qualitative analysis results and frequencies, we will use quotes in the report to support the quantitative results findings. We will share the HESN evaluation report we prepare with CDR for their review and feedback. In addition to these analyses, we will use USAID’s Program and Policy Change Score Card to examine how HESN research contributed to policy change in LMICs. The results would contribute to USAID’s indicator “SO #1: High impact program or policy changes made by the public sector, private sector, or other development actors that are influenced by Lab-funded research results or related scientific activities.” The Score Card identifies the following levels of program and policy change:  Level of influence. Low/small (1 point) if influenced local governments, a few villages, individual (smaller) NGOs/civil society organizations, and or universities. Medium (2 points) if influenced provincial governments or major urban centers (with a population of 1,000,000 or more), combinations of the same change over multiple individual organizations, or multiple districts across a region, and/or a USAID Mission program. High (3 points) if influenced national government policy or core business of a multinational institution or corporation.  Level of implementation. Scientific solution produced (1 point) if research activity completed and evidence disseminated to program/policy stakeholders with potential impact outlined. Stakeholder commitment (2 points) if research findings or outcomes integrated into policy, strategy document, or program design. Solution implemented (3 points) if policy/program change has been implemented. We will also explore the possibility of conducting a cost-effectiveness analysis of USAID investments against outcomes. For this analysis, we will use the overall funding by HESN Labs and several key HESN outcomes or a composite outcome. We are currently exploring options for conducting the cost-effectiveness analysis of HESN projects using budget/funding data and a series of criteria or indicators to measure the outcomes. We will provide a separate, one-page design document to USAID for review and approval prior to beginning this part of the evaluation. Once we have an opportunity to review the cost data, we will work closely with USAID and Notre Dame to determine whether a cost-effectiveness analysis can be completed as part of the evaluation. Furthermore, we will consider conducting a political economy analysis (PEA)8 in each of the three countries that we visit for the fieldwork to examine institutional changes related to HESN’s work. Political economy analysis. Political economy analysis allows us to delve into issues beyond efficiency and usage to look at power dynamics, willingness to change, and institutional 8 Practitioners and researchers use political economy analysis to determine the underlying reasons for a lack of progress on important social issues—such as alleviating poverty—as well as to identify potential strategies for social reform. D.23 | PERFORMANCE EVALUATION OF HESN USAID.GOV facilitators and blockages that can prevent the use of research and data for decision-making. PEA also supports risk analysis and adaptive management, and can help policymakers obtain a more comprehensive understanding of situations within their work environment (Fritz et al. 2010). To conduct the PEA, we would use the World Bank’s Problem-driven Governance and Political Analysis tool to understand specific issues and changes within HEI and the government related to the outcomes of the different HESN activities in developing countries. Political economy analysis involves using data from key information interviews (KII) along with administrative data to identify issues, facilitators of and barriers to change, and the political and institutional environment, to see how projects can effect change over time. Each KII protocol that we develop will include a PEA module that collects data on the agents (individuals), institutions, and enabling environments in each country we visit. Data analysis for PEA will use the Drivers of Change Framework (DOC) to code and map the qualitative data. In applying the DOC (Warrener 2004), we will assess how project design and implementation addressed contributed to changes in contextual factors such as structure, institutions, and agents. In Figure VI.1, we demonstrate the interrelated nature of the DOC analysis framework, as discussed in Warrener (2004). Figure VI.1. Conceptual framework for understanding DOCs Source: Warrener 2004. We have applied Warrener’s (2004) three conceptual areas to the HESN evaluation as follows: 1. Structure. The political structure/history of the university system, its linkages to policy and government decision-making; the political structure and history of government use of data and research for decision-making. 2. Institutions. The relevant legal framework, government policies (related to the research conducted in each country), formal administrative and financial processes, and informal rules that influence the behavior of agents. 3. Agents. Organizations and individuals who pursue their interests. In the given evaluation, agents include politicians and political appointees, public service staff employed by ministries, and university staff and students. The institutional modules to be added to the qualitative protocols will include questions related to the three conceptual areas. We will code the results and then map them to show the changing D.24 | PERFORMANCE EVALUATION OF HESN USAID.GOV political and economic relationships that occur over time. The mapping process will allow us to document how different institutions and agents can influence the decision-making process and hence the political economy. The analysis can further help us to understand the utility component of the HESN activities. DOC analysis provides insight into what, how, and why change takes root in a given sector, and examines the change process through interviews and document reviews. This analysis can contribute to helping policymakers and project managers interact more effectively with key stakeholders and to increase support for a given policy or program. Quantitative data analysis. We will import data from the online survey into Stata (a statistical software package) for review, cleaning, and analysis. We will use the online survey data and key indicators from the administrative data we collect to construct analysis variables that map to the five research questions. We plan to present descriptive statistics on these analysis variables in the evaluation report. If we have adequate sample sizes, we will also employ a regression framework or conduct correlational analyses to examine the relationship between different HESN partnership characteristics and specific outcomes of interest. 9 Qualitative data analysis. We will follow four steps to analyze the data (Creswell and Creswell 2009): 1. Raw data management. Raw data management is the process of organizing raw data into formats usable for analysis (that is, from audio files to transcripts). This implies using both notes and audio recordings to construct a full record of the interview or focus groups. Depending on the richness or sparseness of the data, we may also use a truncated process of transcribing shorthand notes to improve efficiency. During raw data management, we will review all data and eliminate any pieces that are incomplete or not useful to our analysis. 2. Chunking and initial coding. Often referred to as data reduction, chunking and initial coding will enable us to read through the interview transcripts and summaries of the primary documents reviewed several times and obtained a holistic view of the data. We will develop a detailed initial coding scheme, that is, a set of themes we might encounter in the transcripts and summaries, which maps to the research questions. We will document potential topics, linkages among results, and possible findings. 3. Detailed coding. Detailed coding will involve refinement of the coding scheme and the recoding of data as we examine them in greater depth. We will use NVivo software to review and code the transcripts and summaries based on the initial codes developed during the chunking process. Use of NVivo software to assign codes to the qualitative data will enable us to access data on a specific topic quickly and organize information in different ways to identify themes and compile evidence supporting the themes. We will expand and refine the codes during the coding exercise and during subsequent analysis of the coded transcripts and summaries in an iterative process as additional themes emerge. Further, the software will enable us to categorize respondents by salient characteristics to facilitate analysis by subgroup. 9 If our data allow, we will identify plausible hypotheses to be tested in the LASER evaluation. D.25 | PERFORMANCE EVALUATION OF HESN USAID.GOV 4. Data interpretation and writing. Data interpretation and writing will require the triangulation of findings across data sources to highlight mechanisms, contexts, and similarities and differences in perspectives. D.26 | PERFORMANCE EVALUATION OF HESN USAID.GOV VII. EVALUATION RISK AND LIMITATIONS Stakeholder availability for qualitative data collection. With Missions and partners around the world, it is essential to ensure that stakeholders we select for interviews are available to participate in data collection during field visits. As indicated by our data collection and analysis plans, we will conduct online surveys in September 2019. These surveys will include questions related to the availability of researchers and timing for the field visits. The results of the online survey will inform the timing of the interviews. Through that process, we will identify specific stakeholders for phone and in-person interviews. We will take steps to ensure smooth data collection through in-person interviews and focus groups by carefully arranging visits with university and government calendars in mind. Based on the results of the online survey, we will first contact the selected USAID Missions, university staff, national policymakers, and other stakeholders with ample lead time, propose field visit dates between January and April 2020, and confirm stakeholders’ willingness and availability for participation. Second, we will reach out to selected key informants regularly to remind them of the interview dates and confirm logistics. Finally, we will offer a customized interview agenda for national-level policymakers such as Ministry staff. These interview agendas will include the key themes related to the implementation of HESN activities that we would like to discuss during the interview so that they are comfortable meeting with us. Depending on the results of the online surveys, we may also share relevant findings in the interview agenda so that the interviewee can update or confirm the data. Recall bias. Recall bias occurs when participants in studies cannot accurately remember experiences or events in detail because their memories are affected by subsequent events. The HESN activities began in 2012 and have since concluded. Researchers, graduate students, and even government officials have left for other positions, engaged in new activities, or moved to new posts. These changes affect their recollection of the HESN activities, as well as the institutional memory of the organizations involved in the studies. To address these challenges, we will develop memory aids (such as a critical incidence technique10) with basic HESN timelines and major events. These tools will help respondents anchor memories and improve their recall. 10 “The critical incident technique consists of a set of procedures for collecting direct observations of human behavior in such a way as to facilitate their potential usefulness in solving practical problems and developing broad psychological principles. The critical incident technique outlines procedures for collecting observed incidents having special significance and meeting systematically defined criteria.” (Flanagan 1954). D.27 | PERFORMANCE EVALUATION OF HESN USAID.GOV VIII. ADDITIONAL OUTPUTS Preparation of case studies. We will also prepare between 2-4 case studies, which offer a more in-depth look at the most utilized HESN grants to highlight how the mechanism contributed to the use of HESN research outputs by USAID Missions and other key stakeholders, and the factors that most affected the extent to which research outputs were used. We will use data collected from our fieldwork to generate case studies to help us understand the policy-level effects of the program. These case studies will be included in the evaluation report, with each including a description of the context, what worked well, the main challenges, and a summary of lessons learned. Policy brief for LASER. The findings from the HESN and PEER evaluations should inform the evaluation design for the LASER program. LASER is a new engagement model for accessing and responding to the research needs of USAID. The cooperative agreement for LASER began in July 2018 and is expected to end in September 2013. We will develop a 5- to 7-page policy brief that brings together conclusions from both the HESN and PEER evaluations and identifies a credible hypothesis for the LASER evaluation to test on the key elements influencing the uptake of research by Missions. The policy brief will compare the recommended elements from the review of HESN and PEER with the current version of LASER, and identify correlations that can be further examined through LASER. The policy brief will also provide design options for the LASER evaluation. D.28 | PERFORMANCE EVALUATION OF HESN USAID.GOV IX. TIMELINE The timeline in Table IX.1, below, details the planned implementation schedule for the HESN evaluation. Table IX.1. HESN implementation schedule Activity/Deliverable Jun-19 Jul-19 Aug-19 Sep-19 Oct-19 Nov-19 Dec-19 Jan-20 Feb-20 Mar-20 Apr-20 May-20 Jun-20 Jul-20 Aug-20 Sep-20 Oct-20 Nov-20 Dec-20 Jan-21 Feb-21 Mar-21 Detailed Design Document Draft Complete Final Complete Data Collection: Quantitative Internet-based survey Analysis of survey data Data Collection: Qualitative Phone and In-person script revision Phone Interviews In-country field work Data Analysis Report Writing PEER/HESN Evaluation Draft PEER/HESN Evaluation Final Discussion with LASER Team LASER Recommendations Draft LASER Recommendations Final D.29 | PERFORMANCE EVALUATION OF HESN USAID.GOV X. REFERENCES AidData Center for Development Policy (AidData). “Higher Education Solutions Network— Annual Report (FY 2017).” Report prepared for USAID. Williamsburg, Virginia: College of William and Mary, November 2017. Amulya, Joy, James Chung, Stephanie Monschein, and Charu Vijayakumar. “Midterm Performance Evaluation of the Higher Education Solutions Network (HESN).” Report prepared for USAID. Washington, DC: Dexis Consulting Group, April 2016. Comprehensive Initiative on Technology Evaluation (MIT CITE). “Higher Education Solutions Network—Annual Report (FY 2017).” Report prepared for USAID. Cambridge, Massachusetts: Massachusetts Institute of Technology, October 2017. Creswell, John. W., and J. David Creswell. Research Design: Qualitative, Quantitative, and Mixed-Methods Approaches. Washington, DC: SAGE Publications, 2009. Fritz, Verena, Kai Kaiser, and Brian Levy. “Problem-Driven Governmance and Political Economic Analysis: Good Practice Framework.” Washington, DC: World Bank, 2009. Flanagan, John C. “The Critical Incident Technique.” Psychological Bulletin, vol. 51, no. 4, 1954, pp. 327. International Development Innovations Network (IDIN). “Higher Education Solutions Network—Annual Report (FY 2016).” Report prepared for USAID. Cambridge, Massachusetts: Massachusetts Institute of Technology, October 2016. ResilientAfrica Network (RAN). “Higher Education Solutions Network—Annual Report (FY 2017).” Report prepared for USAID. Kampala, Uganda: Makerere University, April 2017. Warrener, Debbie. “Synthesis Paper 3: The Drivers of Change Approach.” London: Overseas Development Institute, 2004. D.30 | PERFORMANCE EVALUATION OF HESN USAID.GOV This page has been left blank for double-sided copying. D.31 | PERFORMANCE EVALUATION OF HESN USAID.GOV APPENDIX A: OUTLINE FOR HESN INSTRUMENTS In this appendix, we outline the content of the online survey and interview protocols for the HESN evaluation. For the online survey, we will primarily use multiple choice and Likert scale questions. For interviews we will ask open ended questions. We plan to update the questions covered in the interview protocols based on results from the online survey. I. COVER SHEET A. Introduction text B. Consent II. GENERAL INFORMATION AND ROLES A. Respondent background information (first and last name, current position, updated contact information, which project(s)) B. HESN role(s) and responsibilities C. Most interesting recollection of HESN (warm-up question) III. USAID MISSION AND OPERATING UNIT STAFF A. Experience with HESN mechanisms 1. How became aware of HESN 2. Reasons why decided to get involved with HESN B. HESN partnership (if multiple partnerships, cover this section for each partnership) 1. Background information on partnership (including funding source) 2. Purpose and objectives of partnership 3. Perceptions of partnership Utility of partnership to the Mission/units and its partners Aspects that facilitate or inhibit setting up and moving forward with partnership a) Structural/procedural elements (contractual) b) Process elements (daily implementation of activities) c) Environmental elements (geographic, social, and policy) d) Incentives to partner 4. Lessons learned for improving future partnership on similar mechanisms D.32 | PERFORMANCE EVALUATION OF HESN USAID.GOV C. HESN partnership research outputs (if multiple partnerships, cover this section for each partnership) 1. Specific research outputs produced by partnership 2. How research outputs were used by the Mission/unit and its partners 3. Perceptions of research outputs Utility of research outputs to work of Mission/unit and its partners Aspects that facilitate or inhibit use of outputs by the Mission and its partners 4. Whether Mission/unit has/is planning any collaboration(s) beyond the original partnership, if yes then: Background information on collaborations(s) (including project title and funding source) Purpose and objectives of collaboration(s) 5. Lessons learned for improving future types of research outputs produced through similar partnerships (multiple choice plus “other category) IV. POLICYMAKERS AND OTHER HESN PARTNERSHIP STAKEHOLDERS A. Experience with HESN partnerships 1. Awareness and visibility of the role of science and engineering in poverty alleviation 2. How became aware of HESN 3. Reasons why decided to get involved with HESN B. HESN partnership (if multiple partnerships, cover this section for each partnership) 1. Name of partnership 2. Purpose and objectives of partnership 3. Perceptions of partnership Utility of partnership for work done by policymaker/partner Aspects facilitating/inhibiting engaging with HESN partners 4. Recommendations for improving future similar partnerships C. HESN partnership research outputs (if multiple partnerships, cover this section for each partnership) 1. Specific research outputs shared by HESN Lab 2. How research outputs were used in work D.33 | PERFORMANCE EVALUATION OF HESN USAID.GOV 3. Perceptions of research outputs Utility of partnership for work (for example, policy, programming) done by policymaker/partner Aspects facilitating or inhibiting use of outputs in work 4. Recommendations for improving future types of research outputs produced through similar partnerships V. HESN LAB STAFF A. Overall experience with HESN mechanisms 1. Perceptions of strengths and weaknesses of structural/procedural elements 2. Perceptions of strengths and weaknesses related to process elements 3. Aspects that facilitate or inhibit engagement with (such as role of AOR in connecting Labs with Missions/Units) USAID Mission Operating units Other HESN partners 4. Recommendations for improving future similar mechanisms B. HESN Lab projects (will cover this section for core-funded, co-funded, and buy-in projects separately) 1. Most successful projects in terms of partner engagement and use of research outputs (will cover this sub-section for up to 3 projects separately) Key project details (name, funding, timeline) Purpose and objectives of project Factors contributing to success of project Types of research outputs produced How research outputs were used by partners Any additional collaborations resulting from this project Key lessons learned from these projects for improving partner engagement and use of outputs by partners 2. Least successful projects in terms of partner engagement and use of research outputs (will cover this sub-section for up to 3 projects separately) Key project details (name, funding, timeline) Purpose and objective of project Factors inhibiting project success Types of research outputs produced D.34 | PERFORMANCE EVALUATION OF HESN USAID.GOV How research outputs were used by partners Any additional collaborations resulting from this project Key lessons learned from these projects for improving partner engagement and use of outputs by partners C. Degree to which undergraduate and graduate students engaged with the HESN lab 1. Involvement in the research products the lab produced 2. Career path changes and opportunities D. Degree to which HEI leadership and administrators supported the HESN lab 1. Strength of institutional financial backing 2. Donor recruitment, dissemination, and media support VI. HEI LEADERSHIP, RESEARCHER/FACULTY, STUDENT A. Respondent type 1. HEI leadership [TO B, then E] 2. Researcher/faculty [TO B, then D] 3. Undergraduate or graduate student [TO B and C] B. Experience with HESN 1. How they became aware of the HESN mechanism 2. Perception of the utility of HESN in terms of: Expanding HEI capacity in international development research Improving student engagement in international development C. Student-level HESN experience 1. Experience with opportunities through HESN: Research practica Mentorship with faculty and researchers Fellowships and internship opportunities 2. Experience working abroad in development institutions as summer fellows or RAs 3. Perceptions of HESN-related training opportunities in: Development issues Research methods 4. Perceptions of: D.35 | PERFORMANCE EVALUATION OF HESN USAID.GOV Utility of research products for USAID Missions and other stakeholders Partnerships with USAID Missions and other stakeholders 5. Level of interest in development careers D. Faculty and researcher-level HESN experience 1. General experience working on development challenges, including consulting and fellowships 2. Experience working with HESN Lab Degree to which HESN Lab provides opportunities for: a) Skills growth b) Career advancement c) Publication and dissemination d) Engagement with policymakers and other stakeholders 3. Perceptions of development research opportunities over the long term at the HEI E. HEI-level experience 1. Student opportunities influenced by HESN Labs: Development-related degree programs (majors, minors, and certifications) and number of students in each Development-related classes created by departments with human, financial, or institutional resources contributed by HESN Lab Quantity and quality of development-related: a. Research practica b. Mentorship with faculty and researchers c. Fellowships and internship opportunities d. Programs abroad 2. Faculty engagement and opportunities influenced by HESN Lab: Development-focused new hires Development-focused faculty consulting assignments and fellowships 3. HEI leadership perceptions of how: HESN influenced research funding and donor cultivation at the institutional level HESN has influenced partnership and HEI network opportunities in the development space 4. Perception of sustainability of HESN-related changes Student opportunities Faculty opportunities Lab research capacity D.36 | PERFORMANCE EVALUATION OF HESN USAID.GOV APPENDIX B: LIST OF DOCUMENTS TO REQUEST FROM HESN AORS Priority documents: 1. HESN lab applications a. Scope of work b. Budget (if we decide to conduct the CEA) 2. HESN cooperative agreements (AOR) 3. M&E plans (AOR) 4. M&E reports and data indicating progress on each HESN and Mission/Operating Unit indicator (M&E POC or AOR) 5. Project descriptions (with the scope of work, deliverables, etc.) prepared for co-funded and buy-in projects (AOR) 6. FY18-FY19 annual reports submitted to Mission/Operating Units (if continued preparing annual reports or similar type of reports beyond HESN core period of performance) (AOR) 7. Sustainability assessments/plans (AORs to ask Labs) Illustrative list of other documents: 1. Outreach/dissemination plans and reports 2. Reports on the impact of the HESN Lab’s work that are not publicly available (Mathematica can pull publicly available information such as promotional materials from each Lab’s website) D.37 | PERFORMANCE EVALUATION OF HESN USAID.GOV This page has been left blank for double sided copying. D.38 | PERFORMANCE EVALUATION OF HESN USAID.GOV Mathematica Princeton, NJ • Ann Arbor, MI • Cambridge, MA Chicago, IL • Oakland, CA • Seattle, WA Tucson, AZ • Woodlawn, MD • Washington, DC EDI Global, a Mathematica Company Bukoba, Tanzania • High Wycombe, United Kingdom mathematica-mpr.com D.39 | PERFORMANCE EVALUATION OF HESN USAID.GOV This page has been left blank for double-sided copying. ANNEX E SURVEY INSTRUMENTS AND INTERVIEW PROTOCOLS This page has been left blank for double-sided copying. Generated by edithgfelix, Apr 06, 2020 12:30 Questionnaire created by edithgfelix, Dec 11, 2019 18:05 Last modified by edithgfelix, Dec 17, 2019 18:58 Shared with: jmeuthalldredge (never edited) Sections: 8, Sub-sections: 9, Questions: 99. Questions with enabling conditions: 49 Questions with validation conditions:17 Rosters: 1 Variables: 0 HESN USAID Staff Survey SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION A. INTRODUCTION No sub-sections, No rosters,Questions: 5, Static texts: 2. B. GENERAL INFORMATION No sub-sections, No rosters,Questions: 11, Static texts: 2. C. HESN PROJECT INFORMATION No sub-sections, No rosters,Questions: 12, Static texts: 3. D. OVERALL EXPERIENCE WITH HESN Sub-sections: 4,No rosters,Questions: 10, Static texts: 11. E. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY Sub-sections: 5,No rosters,Questions: 24, Static texts: 13. F. HESN RESEARCH OUTPUTS AND DISSEMINATION No sub-sections,Rosters: 1,Questions: 23, Static texts: 2. G. RECOMMENDATIONS No sub-sections, No rosters,Questions: 6, Static texts: 1. H. FOLLOW-UP No sub-sections, No rosters,Questions: 8, Static texts: 3. LEGEND E.3 | PERFORMANCE EVALUATION OF HESN USAID.GOV 1 / 24 E.4 | PERFORMANCE EVALUATION OF HESN USAID.GOV SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION 2 / 24 SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION Basic information Title HESN USAID Staff Survey E.5 | PERFORMANCE EVALUATION OF HESN USAID.GOV A. INTRODUCTION 3 / 24 A. INTRODUCTION STATIC TEXT ● This survey gathers information about your perspectives and experience with the Higher Education Solutions Network (HESN). ● If you have participated in another USAID Global Development Lab program such as LASER, RTAC, STIP APS, or PEER, please only provide information related to your perspectives and experience with HESN. ● USAID’s Global Development Lab has contracted Mathematica to conduct an independent evaluation of the long-term impacts of HESN. ● USAID is interested in understanding the ultimate impact/utility of HESN projects implemented since 2015 and the types of partnerships thatmay produce policy impact. ● In this survey we define a HESN Lab, HESN project, and participation in HESN projects as follows: HESN Lab: The university-based umbrella partnership; for example, AidData at the College of William and Mary, the Development Impact Lab (DIL) at the University of California Berkeley, or the Center on Conflict and Development (ConDev) at Texas A&M. HESN project: Specific work undertaken by an HESN university Lab such as conducting a study, undertaking a country- level mapping/geo-referencing activity, holding an innovation competition, or hosting a fellowship program. Participation in an HESN project: This is meant to be viewed broadly and could include a range of ways of engaging such as providing technical advice about a project scope or its implementation, serving as a buy-in point of contact, collecting data, monitoring project progress, activity management, review of deliverables, communication about the project, review of projectmaterials to informnew programming etc. ● This survey should take about 30 minutes to complete. Any information you provide that can identify you will be kept strictly confidential. Information provided will be used for research purposes only. Your participation is voluntary and youmay choose not to answer any or all questions for any reason. ● You may contact Audrey-Marie Moore, the Mathematica Senior Researcher leading this evaluation at AMoore@mathematica-mpr.com, if you have study questions, concerns or complaints. Thank you for your time and help with this evaluation! Please click on the box below to record the survey start time. DATE: CURRENT TIME a_start_time Do you consent to participate in this survey? SINGLE-SELECT a_consent 01 Yes 00 No Name TEXT prefilled_name SCOPE: IDENTIFYING Email TEXT prefilled_email SCOPE: IDENTIFYING Section A end time DATE: CURRENT TIME a_end_time SCOPE: HIDDEN STATIC TEXT Please click on the box to proceed to the next section. E.6 | PERFORMANCE EVALUATION OF HESN USAID.GOV B. GENERAL INFORMATION 4 / 24 B. GENERAL INFORMATION a_consent==1 STATIC TEXT To begin, we would like to ask you some general information about the Higher Education Solutions Network (HESN). Are you familiar with the Higher Education Solutions Network (HESN)? SINGLE-SELECT b_familiar 01 Yes 00 No Please select the option that best describes your placement in USAID: $familiar SINGLE-SELECT b_usaid_place 01 Oversees Mission 02 US-based Bureau or Independent Office 96 Other (please specify) 99 Don't know 98 Choose not to answer Please select the option that best describes your placement in USAID: b_usaid_place==96 TEXT b_usaid_place_other Are you a member of the Center for Development Research (CDR) in USAID's Global Development Lab? $familiar SINGLE-SELECT b_cdr_mem 01 Yes 00 No 99 Don't know 98 Choose not to answer How did you first become aware of HESN? Please select all that apply. $familiar && b_cdr_mem!=1 (self.ContainsAny(1,2,3,4,5,6,7,8,96)&&!self.ContainsAny (99,98)) ||(self.ContainsAny(99,98)&&!self.ContainsAny(1 ,2,3,4,5,6,7,8,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT b_first_aware 01 Involved in the original design of the HESNmechanism 02 Introduced by HESN Lab's Agreement Officer's Representative (AOR) 03 Introduced by other USAID Global Development Lab staff 04 Introduced by other Mission staff 05 Introduced by other Operating Unit staff 06 Attended a HESN Lab event 07 HESN Lab communications 08 HESN Lab research output 96 Other (please specify) 99 Don't know 98 Choose not to answer How did you first become aware of HESN? Please specify other response. b_first_aware.Contains(96) TEXT b_first_aware_other Did you become involved with HESN? $familiar && b_cdr_mem!=1 SINGLE-SELECT b_involve 01 Yes 00 No E.7 | PERFORMANCE EVALUATION OF HESN USAID.GOV B. GENERAL INFORMATION 5 / 24 STATIC TEXT Please click on the blue box to proceed to the next section. Why did you decide not to get involved with HESN? b_involve==0 (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98 ) )||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4,9 6)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT b_not_involve_reason 01 Did not have any need for research outputs 02 Already involved with other researchers/evaluators 03 Budget constraints 04 Projects/programs did not align with work done by Labs 96 Other (please specify) 99 Don't know 98 Choose not to answer Why did you decide not to get involved with HESN? Please specify other response. b_not_involve_reason.Contains(96) TEXT b_not_invole_reason_other Why did you decide to get involved with HESN? Please select all that apply. b_involve==1 (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT b_involve_reason 01 Needed inputs for country development strategy or program design 02 Needed data (program monitoring data, demographic, geographic, sectoral, etc.) 03 Needed to implement a component of a development program or intervention 04 Needed an evaluator/evaluation 05 Needed to fill gaps in data 06 Needed USAID Mission/Operating Unit capacity building 07 Needed help with partner capacity building 08 Needed help with policy development 09 Needed access to new partners 96 Other (please specify) 99 Don't know 98 Choose not to answer Why did you decide to get involved with HESN? Please specify other response. b_involve_reason.Contains(96) TEXT b_involve_reason_other E.8 | PERFORMANCE EVALUATION OF HESN USAID.GOV C. HESN PROJECT INFORMATION 6 / 24 C. HESN PROJECT INFORMATION $involved || b_cdr_mem==1 STATIC TEXT Next, we would like to ask you several questions about your participation in specific HESN projects. Please see the below definitions: HESN Lab: The university-based umbrella partnership; for example, AidData at the College of William and Mary, the Development Impact Lab (DIL) at the University of California Berkeley, or the Center on Conflict and Development (ConDev) at Texas A&M. HESN project: Specific work undertaken by an HESN university Lab such as conducting a study, undertaking a country- level mapping/geo-referencing activity, holding an innovation competition, or hosting a fellowship program. Participation in an HESN project: This is meant to be viewed broadly and could include a range of ways of engaging such as providing technical advice about a project scope or its implementation, serving as a buy-in point of contact, collecting data, monitoring project progress, activity management, review of deliverables, communication about the project, review of projectmaterials to informnew programming etc. STATIC TEXT $projects Think about these projects (individual research activities/studies) that you considered to best represent HESN's goals. With these projects in mind, please answer the questions that follow: How many HESN projects (individual research activities/studies) have you been participated in since 2015? SINGLE-SELECT c_number_projects 01 1 to 2 projects 02 3 to 5 projects 03 6 to 10 projects 04 More than 10 projects 95 Did not participate in any individual projects 99 Don't know 98 Choose not to answer Of the HESN projects (individual research activities/studies) that you participated in since 2015, which projects (please list at least ONE project) best represent the HESN goals of: 1. Accelerating the use of innovative technologies and approaches to address global development challenges 2. Catalyzing an interdisciplinary research environment 3. Supporting evidence-based development decision making Please press enter to list each project name in a separate text box. $projects LIST c_main_project E.9 | PERFORMANCE EVALUATION OF HESN USAID.GOV C. HESN PROJECT INFORMATION 7 / 24 Which HESN Lab(s) were/are these projects (individual research activities/studies) linked to? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,7,8)&&!self.ContainsAny(99 ,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2, 3,4,5,6,7,8)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT c_project_lab 01 ResilientAfrica Network (RAN) 02 UC Berkeley – Development Impact Lab (DIL) 03 MIT – International Development Innovation Network (IDIN) 04 MIT – Comprehensive Initiative on Technology Evaluation (CITE) 05 MSU – Global Center for Food Systems Innovation (GCFSI) 06 Texas A&M – Center on Conflict and Development (ConDev) 07 Duke – Social Entrepreneurship Accelerator at Duke (SEAD) 08 William and Mary – AidData 99 Don't know 98 Choose not to answer What specific role(s) did/do you have under these projects? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98 ) )||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4,9 6)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT c_project_role 01 Overall project monitoring and oversight support 02 Coordinating partner communication 03 Activity management 04 Review of project deliverables 96 Other (please specify) 99 Don't know 98 Choose not to answer What specific role(s) did/do you have under these projects? Please specify other response. c_project_role.Contains(96) TEXT c_project_role_other What were/are the funding sources for these projects? Please select all that apply. $projects (self.ContainsAny(1,2,3,96)&&!self.ContainsAny(99,98 ))| |(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT c_project_fund 01 HESN (through Global Development Lab) 02 My USAID Mission 03 My USAID Operating Unit 96 Other (please specify) 99 Don't know 98 Choose not to answer What were/are the funding sources for these projects? Please specify other response. c_project_fund.Contains(96) TEXT c_project_fund_other E.10 | PERFORMANCE EVALUATION OF HESN USAID.GOV C. HESN PROJECT INFORMATION 8 / 24 STATIC TEXT Please click on the blue box to proceed to the next section. What sector(s) did/do these projects cover? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT c_project_sector 01 Agriculture, food distribution, food security 02 Democracy, human rights, and governance 03 Economic growth and trade (including infrastructure) 04 Education 05 Environment and climate change 06 Gender and women’s empowerment 07 Health and nutrition 08 Water and sanitation 09 Crisis and conflict 96 Other (please specify) 99 Don't know 98 Choose not to answer What sector(s) did/do these projects cover? Please specify other response. c_project_sector.Contains(96) TEXT c_project_sector_other In what country/countries were/are these projects being implemented? Ifmore than one country, please press enter to list each country in a s eparate text box. $projects LIST c_project_loc What were/are the main objectives of these projects? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,7,8,96)&&!self.ContainsAny (99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1 ,2,3,4,5,6,7,8,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT c_project_object 01 Inputs for country development strategy or program design 02 Collect data (program monitoring data, demographic, geographic, sectoral, etc.) 03 Impact evaluation of a program 04 Performance evaluation of a program 05 Fill gaps in data 06 Capacity building of USAID Mission/Operating Unit 07 Capacity building of partners 08 Policy development 96 Other (please specify) 99 Don't know 98 Choose not to answer What were/are the main objectives of these projects? Please specify other response. c_project_object.Contains(96) TEXT c_project_object_ot E.11 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. OVERALL EXPERIENCE WITH HESN 9 / 24 D. OVERALL EXPERIENCE WITH HESN $involved || b_cdr_mem==1 STATIC TEXT Please based your responses on your experience with the HESN projects (individual research activities/studies) that you identified as best representing HESN’s goals. STATIC TEXT Please click on each of the sub-section boxes to view and respond to statements. Each sub-section box will turn green once completed. D.OVERALL EXPERIENCE WITH HESN HESN HAS EXPANDED THE CAPACITY OF UNIVERSITIES WITH HESN LABS TO: STATIC TEXT HESN has expanded the capacity of universities with HESN Labs to: Please indicate how strongly you agree with each of the following statements. Design research and evaluations related to international development. SINGLE-SELECT d_design_eval 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Collect and analyze data related to international development. SINGLE-SELECT d_collect_data 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. D.OVERALL EXPERIENCE WITH HESN HESN HAS EXPANDED OPPORTUNITIES FOR USAID MISSIONS AND OPERATING UNITS TO: STATIC TEXT HESN has expanded opportunities for USAID Missions and Operating units to: Please indicate how strongly you agree with each of the following statements. E.12 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. OVERALL EXPERIENCE WITH HESN 10 / 24 Collaborate in the design and implementation of international development research. SINGLE-SELECT d_collaborate_design 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Create opportunities for students to work with university faculty to conduct international development research. SINGLE-SELECT d_opportunity_students 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. D.OVERALL EXPERIENCE WITH HESN PARTICIPATING IN HESN LAB ACTIVITIES HAS: STATIC TEXT Participating in HESN Lab activities has: Please indicate how strongly you agree with each of the following statements. Created new career opportunties for students. SINGLE-SELECT d_created_opportunities 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Motivated students to pursue work in international development. SINGLE-SELECT d_motivated_students 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Motivated university faculty/staff to engage students in their research opportunities. SINGLE-SELECT d_motivated_faculty 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E.13 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. OVERALL EXPERIENCE WITH HESN 11 / 24 D.OVERALL EXPERIENCE WITH HESN HESN HAS PROVIDED: STATIC TEXT HESN has provided: Please indicate how strongly you agree with each of the following statements. Additional financial resources to support research conducted by universities. SINGLE-SELECT d_financial_resources 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Added value to USAID Mission/Operating Unit policy and planning by providing evidence to support our decisions. SINGLE-SELECT d_added_value 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Motivation for me to participate in another HESN project if given the opportunity. SINGLE-SELECT d_participate_opportunity 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. STATIC TEXT Please click on the blue box to proceed to the next section. E.14 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY 12 / 24 E. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY $involved || b_cdr_mem==1 STATIC TEXT Please based your responses on your experience with the HESN projects (individual research activities/studies) that you identified as best representing HESN’s goals. STATIC TEXT Please click on each of the sub-section boxes to view and respond to statements. Each sub-section box will turn green once completed. E. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY DURING HESN PROJECT IMPLEMENTATION: STATIC TEXT During HESN project implementation: Please indicate how strongly you agree with each of the following statements. Communication with HESN Lab university staff was smooth throughout HESN. SINGLE-SELECT e_comm_staff 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know We experienced challenges in engaging with HESN Lab university staff. SINGLE-SELECT e_experience_challenge 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know We overcame the challenges we experienced in working with HESN Lab university staff. SINGLE-SELECT e_overcame_challenge 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Our original plans to work under the HESN core funding were implemented well and achieved the desired goals. SINGLE-SELECT e_orig_plan_core 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.15 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY 13 / 24 Our original plans to work under the HESN buy-ins were implemented well and achieved the desired goals. SINGLE-SELECT e_orig_buyin 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY OUTPUTS FROM HESN PROJECTS WERE: STATIC TEXT Outputs from HESN projects were: Please indicate how strongly you agree with each of the following statements. Used by USAID Mission and/or Operating Unit staff as inputs for their country development strategy or program design. SINGLE-SELECT e_used_usaid 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Relevant to USAID Missions/Operating Units for planning future programs. SINGLE-SELECT e_relevant_plan 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Delivered during a time period when USAID Missions/Operating Units needed the information for policy, planning, or learning. SINGLE-SELECT e_deliver_time 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Used by policymakers in the target developing country for planning and policy making. SINGLE-SELECT e_used_policy 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E.16 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY 14 / 24 E. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY HESN LED TO MORE OPPORTUNITES FOR MY USAID MISSION/OPERATING UNIT TO: STATIC TEXT HESN led to more opportunites for my USAID Mission/Operating Unit to: Please indicate how strongly you agree with each of the following statements. Broaden the evidence used for internal planning purposes. SINGLE-SELECT e_broaden_evidence 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Contribute to research/evidence in international development. SINGLE-SELECT e_contribute_research 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Provide funding to universities and other partner higher education institutions to conduct additional research work in the next year. SINGLE-SELECT e_provide_funding 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Engage with policymakers in developing countries around research findings. SINGLE-SELECT e_engage_policy_find 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Engage with policymakers in developing countries around key development issues facing their country. SINGLE-SELECT e_engage_policy_issue 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Engage with policymakers in developing countries to use evidence for the purpose of developing new policies, laws, or regulations. SINGLE-SELECT e_engage_policy_use_evidence 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.17 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY 15 / 24 Grow our international development research work. SINGLE-SELECT e_grow_intl_work 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Increase our focus on international development research. SINGLE-SELECT e_increase_focus 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Provide students and faculty with new opportunities to conduct research in international development work. SINGLE-SELECT e_provide_opportunity 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Contribute to changes in developing country universities that allow students and faculty to do more work in international development (e.g. led to new coursework, research opportunities, increased funding). SINGLE-SELECT e_change_dev_uni 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY MY EXPERIENCE ENGAGING WITH A HESN LAB HELPED ME: STATIC TEXT My experience engaging with a HESN Lab helped me: Please indicate how strongly you agree with each of the following statements. Understand how research can contribute to USAID planning and decision-making. SINGLE-SELECT e_exp_understand 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.18 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY 16 / 24 Use evidence to influence international development policy and practice. SINGLE-SELECT e_exp_use_evidence 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Develop new activities/projects to support international development. SINGLE-SELECT e_ex_new_act 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY HESN PROJECTS: STATIC TEXT HESN projects: Please indicate how strongly you agree with each of the following statements. Note: We define HESN projects as specific work undertaken by a HESN university Lab such as conducting a study, undertaking a country-level mapping/geo-referencing activity, holding an innovation competition, or hosting a fellowship program. Are a positive example of how different stakeholders can work together in partnership to achieve international development goals. SINGLE-SELECT e_positive_ex 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Brought changes that will be sustained in the next five years. SINGLE-SELECT e_changes_sustain 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. STATIC TEXT Please click on the blue box to proceed to the next section. E.19 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN RESEARCH OUTPUTS AND DISSEMINATION 17 / 24 F. HESN RESEARCH OUTPUTS AND DISSEMINATION ($involved || b_cdr_mem==1) && $projects STATIC TEXT Now we would like to ask you a few questions about the dissemination of results/research outputs from HESN project (individual research activities/studies). Once again, think about the HESN projects (individual research activities/studies) that you considered to best represent HESN's goals. With these projects inmind, please answer the questions that follow: What research results/outputs were/will be produced as part of these projects? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,7,96)&&!self.ContainsAny(9 9,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2 ,3,4,5,6,7,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT f_project_research 01 Report or publication 02 Policy brief 03 Toolkit 04 Data set/platform/mapping 05 Evaluation 06 Training 07 Event or conference 96 Other (please specify) 99 Don't know 98 Choose not to answer What research results/outputs were/will be produced as part of these projects? Please specify other response. f_project_research.Contains(96) TEXT f_project_research_other Did these projects already produce research SINGLE-SELECT f_project_produced results/outputs for sharing? 01 Yes 00 No 99 Don't know 98 Choose not to answer How satisfied were you with the quality of research outputs produced under these projects? SINGLE-SELECT f_project_satisfaction 01 Very satisfied 02 Satisfied $produced 03 Unsatisfied 04 Very unsatisfied 95 Does not apply 99 Don't know 98 Choose not to answer E.20 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN RESEARCH OUTPUTS AND DISSEMINATION 18 / 24 F. HESN RESEARCH OUTPUTS AND DISSEMINATION Roster: RESEARCH USE generated bymulti-select question f_project_use f_research_use f_project_use.ContainsAny(1,2,3,4,96) How did your Mission/Operating Unit use the research results/outputs under these projects? Please select all that apply. $produced (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98,5 ,6 ))||(self.ContainsAny(99,98,5,6)&&!self.ContainsAny(1 ,2,3,4,96)) Your answers cannot include both substantive answers and "not used yet butwill be used in the future"/"not used for anything"/"don't know"/ "choose not to answer". MULTI-SELECT f_project_use 01 Inputs for country development strategy or program design 02 Capacity building of USAID Mission/Operating Unit 03 Capacity building of government in target country 04 Policy development 05 Not used yet but will be used in the future 06 Not used for anything 96 Other specified use 99 Don't know 98 Choose not to answer What were the research results/outputs under these projects used for? Please specify other response. f_project_use.Contains(96) TEXT f_project_use_other Why were the research results/outputs produced under these projects not used for anything? f_project_use.Contains(6) TEXT f_project_no_use When do you expect that research results/outputs produced under these projects will be used? f_project_use.Contains(5) TEXT f_project_use_when What will the research results/outputs under these proejcts be used for? Please select all that apply. f_project_use.Contains(5) (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98 ) )||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4,9 6)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT f_project_exp_use 01 Inputs for country development strategy or program design 02 Capacity building of Mission/Operating unit 03 Capacity building of government in target country 04 Policy development 96 Other (please specify) 99 Don't know 98 Choose not to answer What will the research results/outputs under these projects be used for? Please specify other response. f_project_exp_use.Contains(96) TEXT f_project_use_exp_ot E.21 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN RESEARCH OUTPUTS AND DISSEMINATION 19 / 24 How useful were research results/outputs from these projects as an input for %f_research_use%? SINGLE-SELECT f_research_utility_level 01 Extremely useful 02 Very useful 03 Moderately useful 04 Slightly useful 05 Not at all useful 99 Don't know 98 Choose not to answer What facilitated the use of research results/outputs produced under these projects? Please select all that apply. $produced (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT f_project_facil 01 Timeline for completing the product 02 Daily implementation of activities 03 Support from policymakers due to relevance of topic 04 Geography in-country facilitates dissemination (i.e. easy access to internet, regional distribution point, easy access to urban and rural areas) 05 Local interest/desire for research product 06 Having financial resources to create products 07 Having financial resources to disseminate products 08 Having financial or in-kind incentives for partners 09 Expertise and reputation of HESN Lab 96 Other (please specify) 99 Don't know 98 Choose not to answer What facilitated the use of research results/outputs produced under these projects? Please specify other response. f_project_facil.Contains(96) TEXT f_project_facil_ot Why do you think the items you selected above facilitated the use of results/research outputs fromthese projects? f_project_facil.ContainsAny(1,2,3,4,5,6,7,8,9,96) TEXT f_project_facil_why E.22 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN RESEARCH OUTPUTS AND DISSEMINATION 20 / 24 What inhibited the use of research results/outputs produced under these projects? Please select all that apply. $produced (self.ContainsAny(1,2,3,4,5,6,7,8,9,10,11,96)&&!self.Con tainsAny(99,98 ))||(self.ContainsAny(99,98)&&!self.Conta insAny(1,2,3,4,5,6,7,8,9,10,11,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT f_project_inhib 01 Contractual requirements 02 Reporting requirements 03 Delays in completing research products 04 Lack of support from policymakers due to relevance of topic 05 Geography in-country inhibits dissemination (i.e. lack of consistent internet, no regional distribution points, difficult access to urban and rural areas) 06 Lack of local interest/desire for research product 07 Lack of financial resources to create products 08 Lack of financial resources to disseminate product 09 Lack of financial or in-kind incentives for partners 10 Limits on ability to publish/copyright issues 11 Political pressure 96 Other (please specify) 99 Don't know 98 Choose not to answer What inhibited the use of research results/outputs produced under these projects? Please specify other response. f_project_inhib.Contains(96) TEXT f_project_inhib_ot Why do you think the items you selected above inhibited the use of results/research outputs fromthese projects? f_project_inhib.ContainsAny(1,2,3,4,5,6,7,8,9,10,11,96) TEXT f_project_inhib_why Did any additional collaborations result beyond the original scope of these projects? SINGLE-SELECT f_project_additional 01 Yes 00 No 99 Don't know 98 Choose not to answer E.23 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN RESEARCH OUTPUTS AND DISSEMINATION 21 / 24 STATIC TEXT Please click on the blue box to proceed to the next section. What are the main objectives of the additional collaborations beyond these projects? Please select all that apply. $additional (self.ContainsAny(1,2,3,4,5,6,7,8,96)&&!self.ContainsAny (99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1 ,2,3,4,5,6,7,8,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT f_project_add_obj 01 Inputs for country development strategy or program design 02 Collect data (program monitoring data, demographic, geographic, sectoral, etc.) 03 Impact evaluation of a program 04 Performance evaluation of program 05 Fill gaps in data 06 Capacity building of USAID Mission/Operating Unit 07 Capacity building of partners 08 Policy development 96 Other (please specify) 99 Don't know 98 Choose not to answer What are the main objectives of the additional collaborations beyond the scope of these projects? Please specify other response. f_project_add_obj.Contains(96) TEXT f_project_add_obj_ot What are/will be the funding sources for the additional collaborations beyond these projects? Please select all that apply. $additional (self.ContainsAny(1,2,3,96)&&!self.ContainsAny(99,98 ))| |(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT f_project_add_fund 01 HESN (through Global Development Lab) 02 My USAID Mission 03 My USAID Operating Unit 96 Other (please specify) 99 Don't know 98 Choose not to answer What are/will be the funding sources for the additional collaborations beyond these projects? Please specify other response. f_project_add_fund.Contains(96) TEXT f_project_add_fund_ot In what country/countries will the additional collaborations beyond the scope of these projects be implemeneted? Ifmore than one country, please press enter to list each country in a s eparate text box. $additional LIST f_project_add_loc G. RECOMMENDATIONS E $involved || b_cdr_mem==1 E.24 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. RECOMMENDATIONS 22 / 24 STATIC TEXT Please click on the blue box to proceed to the next section. What recommendations do you have for improving uptake of future research results/outputs produced through HESN or similar mechanisms? Please select all that apply. (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98 ) )||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4,9 6)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_rec_uptake 01 Make content less technical 02 More dissemination activities 03 Better alignment with key stakeholder interests 04 Better alignment with Mission/Operating Unit development objectives 96 Other (please specify) 99 Don't know 98 Choose not to answer What recommendations do you have for improving uptake of future research results/outputs produced through HESN or similar mechanisms? Please specify other response. g_rec_uptake.Contains(96) TEXT g_rec_uptake_other What recommendation do you have for improving future mechanisms similar to HESN? Please select all that apply. (self.ContainsAny(1,2,3,96)&&!self.ContainsAny(99,98 ))| |(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_rec_mech 01 Fewer reporting requirements 02 More time flexibility 03 More funding 96 Other (please specify) 99 Don't know 98 Choose not to answer What recommendation do you have for improving future mechanisms similar to HESN? Please specify other response. g_rec_mech.Contains(96) TEXT g_rec_mech_other Do you have any other comments or suggestions related to your experience with HESN? SINGLE-SELECT g_has_comment 01 Yes (please specify) 00 No Do you have any other comments or suggestions related to your experience with HESN? Please specify any other comments/suggestions. g_has_comment==1 TEXT g_comment_specify H. FOLLOW-UP $involved || b_cdr_mem==1 E.25 | PERFORMANCE EVALUATION OF HESN USAID.GOV H. FOLLOW-UP 23 / 24 In order to learn more about the utility of the HESN mechanims and research outputs, we would like to conduct short in-person or phone interviews with a subsample of online survey respondents. Would you be willing to participate in a short follow-up interview? SINGLE-SELECT h_follow_part 01 Yes 00 No STATIC TEXT $follow Please provide the below information: STATIC TEXT Thank you for your time! Please click on the box below to record the survey end time. DATE: CURRENT TIME h_end_time STATIC TEXT Please click on the blue box to proceed to the next section. First name: $follow TEXT h_first_name Last name: $follow TEXT h_last_name Email: $follow TEXT h_email Phone number (with country code): $follow TEXT h_phone Current organization: $follow TEXT h_organ Current position: $follow TEXT h_position E.26 | PERFORMANCE EVALUATION OF HESN USAID.GOV LEGEND 24 / 24 LEGEND Legend and structure of information in this file This page has been left blank for double-sided copying. Generated by edithgfelix, Apr 06, 2020 12:30 Questionnaire created by edithgfelix, Dec 13, 2019 18:52 Last modified by edithgfelix, Dec 23, 2019 23:15 Shared with: jmeuthalldredge (never edited) Sections: 9, Sub-sections: 11, Questions: 120. Questions with enabling conditions: 54 Questions with validation conditions:26 Rosters: 1 Variables: 0 HESN Lab Staff Survey SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION A. INTRODUCTION No sub-sections, No rosters,Questions: 4, Static texts: 2. B. GENERAL INFORMATION No sub-sections, No rosters,Questions: 6, Static texts: 2. C. RESEARCH EXPERIENCE No sub-sections, No rosters,Questions: 10, Static texts: 2. D. HESN PROJECT INFORMATION No sub-sections, No rosters,Questions: 15, Static texts: 3. E. OVERALL EXPERIENCE WITH HESN Sub-sections: 5,No rosters,Questions: 15, Static texts: 13. F. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY Sub-sections: 6,No rosters,Questions: 26, Static texts: 16. G. HESN RESEARCH OUTPUTS AND DISSEMINATION No sub-sections,Rosters: 1,Questions: 32, Static texts: 2. H. RECOMMENDATIONS No sub-sections, No rosters,Questions: 4, Static texts: 1. I. FOLLOW-UP No sub-sections, No rosters,Questions: 8, Static texts: 3. LEGEND E.28 | PERFORMANCE EVALUATION OF HESN USAID.GOV 1 / 31 E.29 | PERFORMANCE EVALUATION OF HESN USAID.GOV SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION 2 / 31 SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION Basic information Title HESN Lab Staff Survey E.30 | PERFORMANCE EVALUATION OF HESN USAID.GOV A. INTRODUCTION 3 / 31 A. INTRODUCTION STATIC TEXT ● This survey gathers information about your perspectives and experience with the Higher Education Solutions Network (HESN). ● If you have participated in another USAID Global Development Lab program such as LASER, RTAC, STIP APS, or PEER, please only provide information related to your perspectives and experience with HESN. ● USAID’s Global Development Lab has contracted Mathematica to conduct an independent evaluation of the long-term impacts of HESN. ● USAID is interested in understanding the ultimate impact/utility of HESN projects implemented since 2015 and the types of partnerships thatmay produce policy impact. ● In this survey we define a HESN Lab, HESN project, and participation in HESN projects as follows: HESN Lab: The university-based umbrella partnership; for example, AidData at the College of William and Mary, the Center on Conflict and Development (ConDev) at Texas A&M, or the Development Impact Lab (DIL) at the University of California Berkeley. HESN project: Specific work undertaken by an HESN university Lab such as conducting a study, undertaking a country- level mapping/geo-referencing activity, holding an innovation competition, or hosting a fellowship program. Participation in an HESN project: This is meant to be viewed broadly and could include a range of ways of engaging such as providing technical advice about a project scope or its implementation, serving as a buy-in point of contact, collecting data, monitoring project progress, activity management, review of deliverables, communication about the project, review of projectmaterials to informnew programming etc. ● This survey should take about 30 minutes to complete. Any information you provide that can identify you will be kept strictly confidential. Information provided will be used for research purposes only. Your participation is voluntary and youmay choose not to answer any or all questions for any reason. ● You may contact Audrey-Marie Moore, the Mathematica Senior Researcher leading this evaluation at AMoore@mathematica-mpr.com, if you have study questions, concerns or complaints. Thank you for your time and help with this evaluation! Please click on the box below to record the survey start time. DATE: CURRENT TIME a_start_time Do you consent to participate in this survey? SINGLE-SELECT a_consent 01 Yes 00 No Name TEXT prefilled_name SCOPE: IDENTIFYING Email TEXT prefilled_email SCOPE: IDENTIFYING STATIC TEXT Please click on the box to proceed to the next section. E.31 | PERFORMANCE EVALUATION OF HESN USAID.GOV B. GENERAL INFORMATION 4 / 31 B. GENERAL INFORMATION a_consent==1 STATIC TEXT To begin, we would like to ask you some general information about the Higher Education Solutions Network (HESN). Are you familiar with the the Higher Education Solutions Network (HESN)? SINGLE-SELECT b_familiar 01 Yes 00 No How did you first become aware of HESN? Please select all that apply. $familiar (self.ContainsAny(1,2,3,4,5,6,7,8,9,10,96)&&!self.Contai nsAny(99,98)) ||(self.ContainsAny(99,98)&&!self.Contains Any(1,2,3,4,5,6,7,8,9,10,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT b_first_aware 01 USAID request for proposal (RFP/RFTOP) 02 Part of the HESN Lab team 03 Faculty/staff from my university’s department 04 Peers at my university 05 Peers at other institutions 06 University administration 07 USAID communications 08 HESN Lab event 09 HESN Lab communications 10 HESN Lab research output 96 Other (please specify) 99 Don't know 98 Choose not to answer How did you first become aware of HESN? Please specify other response. b_first_aware.Contains(96) TEXT b_first_aware_other Which HESN Lab(s) were/are you part of? Please select all that apply. $familiar (self.ContainsAny(1,2,3,4,5,6,7,8)&&!self.ContainsAny(99 ,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2, 3,4,5,6,7,8)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT b_part_lab 01 ResilientAfrica Network (RAN) 02 UC Berkeley – Development Impact Lab (DIL) 03 MIT – International Development Innovation Network (IDIN) 04 MIT – Comprehensive Initiative on Technology Evaluation (CITE) 05 MSU – Global Center for Food Systems Innovation (GCFSI) 06 Texas A&M – Center on Conflict and Development (ConDev) 07 Duke – Social Entrepreneurship Accelerator at Duke (SEAD) 08 William and Mary – AidData 99 Don't know 98 Choose not to answer E.32 | PERFORMANCE EVALUATION OF HESN USAID.GOV B. GENERAL INFORMATION 5 / 31 STATIC TEXT Please click on the blue box to proceed to the next section. What role(s) did you play under HESN? Please select all that apply. $familiar (self.ContainsAny(1,2,3,4,5,6,7,8,9,10,96)&&!self.Contai nsAny(99,98)) ||(self.ContainsAny(99,98)&&!self.Contains Any(1,2,3,4,5,6,7,8,9,10,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT b_role 01 HESN Lab director 02 HESN Lab manager 03 Research manager/coordinator 04 Research assistant/analyst 05 Project coordinator/manager 06 Monitoring and evaluation manager 07 Partner outreach coordinator/manager 08 Principal investigator 09 Researcher supporting principal investigator 10 Research fellow 96 Other (please specify) 99 Don't know 98 Choose not to answer What role(s) did you play under HESN? Please specify other response. b_role.Contains(96) TEXT b_role_other C. RESEARCH EXPERIENCE 6 / 31 C. RESEARCH EXPERIENCE $familiar STATIC TEXT Now, we would like to ask you about your individual research experience. E.33 | PERFORMANCE EVALUATION OF HESN USAID.GOV How many years have you been working with your HESN Lab? self < 30 && self>=0 This number does not seemcorrect, please double check. NUMERIC: INTEGER c_uni_yrs What are your primary research sectors? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT c_primary_sec 01 Agriculture, food distribution, food security 02 Democracy, human rights, and governance 03 Economic growth and trade (including infrastructure) 04 Education 05 Environment and climate change 06 Gender and women’s empowerment 07 Health and nutrition 08 Water and sanitation 09 Crisis and conflict 96 Other (please specify) 99 Don't know 98 Choose not to answer What are your primary research sectors? Please specify other response. c_primary_sec.Contains(96) TEXT c_primary_sec_other How many years have you worked in international development research? self < 70 && self>=0 This number does not seemcorrect, please double check. NUMERIC: INTEGER c_res_dev_yrs What is your total number of published peer- reviewed journal articles? self < 1000 && self>=0 This number does not seemcorrect, please double check. NUMERIC: INTEGER c_pub_num C. RESEARCH EXPERIENCE 7 / 31 Did you collaborate with any of the following entities on a research project prior to your participation in the HESN? Please select all that apply. (self.ContainsAny(1,2,3,4,5)&&!self.ContainsAny(99,98,6 ))||(self.ContainsAny(99,98,6)&&!self.ContainsAny(1,2,3, 4,5)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer"/"did not collaborate with any of these entiti es". MULTI-SELECT c_prior_collab_ent 01 USAID Operating Unit (ex. Bureau for Global Health, Bureau for Food Security) 02 USAID Mission 03 HESN Global Development Lab university 04 Other international development aid agency 05 Policymakers in developing countries 06 Did not collaborate with any of these entities 99 Don't know 98 Choose not to answer Besides your HESN Lab position(s), did you hold any other positions while conducting research under HESN? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,7,8,9,10,11,96)&&!self.Con tainsAny(99,98,12 ))||(self.ContainsAny(99,98,12)&&!self .ContainsAny(1,2,3,4,5,6,7,8,9,10,11,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer"/"no other position". MULTI-SELECT c_pos 01 Senior/full professor 02 Junior/associate/assistant professor 03 Professor/university lecturer (non-tenure track, non- adjunct) 04 Visiting or adjunct faculty 05 Research scientist 06 Post-doctoral student 07 PhD student 08 Master's student 09 College dean/department chair/ college rector/ or equivalent 11 Extension agent/faculty 12 No other position 96 Other (please specify) 99 Don't know 98 Choose not to answer Besides your HESN Lab position(s), did you hold any other positions while conducting research under HESN? Please specify other response. c_pos.Contains(96) TEXT c_pos_other What sectors of research were you involved with during your participation with HESN? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT c_dur_sec 01 Agriculture, food distribution, food security 02 Democracy, human rights, and governance 03 Economic growth and trade (including infrastructure) 04 Education 05 Environment and climate change 06 Gender and women’s empowerment 07 Health and nutrition 08 Water and sanitation 09 Crisis and conflict 96 Other (please specify) 99 Don't know 98 Choose not to answer C. RESEARCH EXPERIENCE 8 / 31 STATIC TEXT Please click on the blue box to proceed to the next section. E.35 | PERFORMANCE EVALUATION OF HESN USAID.GOV What sectors of research were you involved with during your participation with HESN? Please specify other response. c_dur_sec.Contains(96) TEXT c_dur_sec_other E.36 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. HESN PROJECT INFORMATION 9 / 31 D. HESN PROJECT INFORMATION $familiar STATIC TEXT Next, we would like to ask you several questions about your participation in specific HESN projects. Please see the below definitions: HESN Lab: The university-based umbrella partnership; for example, AidData at the College of William and Mary, the Development Impact Lab (DIL) at the University of California Berkeley, or the Center on Conflict and Development (ConDev) at Texas A&M. HESN project: Specific work undertaken by an HESN university Lab such as conducting a study, undertaking a country- level mapping/geo-referencing activity, holding an innovation competition, or hosting a fellowship program. Participation in an HESN project: This is meant to be viewed broadly and could include a range of ways of engaging such as providing technical advice about a project scope or its implementation, serving as a buy-in point of contact, collecting data, monitoring project progress, activity management, review of deliverables, communication about the project, review of projectmaterials to informnew programming etc. What was the primary reason you chose to work on research projects under HESN? Please select your primary reason. SINGLE-SELECT d_work_reason 01 Supporting the work of USAID Operating Units and Missions 02 Advancing international development research at your university 03 Gaining research experience 96 Other (please specify) 99 Don't know 98 Choose not to answer What was the primary reason you chose to work on research projects under HESN? Please specify other response. d_work_reason==96 TEXT d_work_reason_ot How many HESN projects (individual research activities/studies) have you been participated in since 2015? SINGLE-SELECT d_number_projects 01 1 to 2 projects 02 3 to 5 projects 03 6 to 10 projects 04 More than 10 projects 95 Did not participate in any individual projects 99 Don't know 98 Choose not to answer E.37 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. HESN PROJECT INFORMATION 10 / 31 STATIC TEXT $projects Think about these projects (individual research activities/studies) that you considered to best represent HESN's goals. With these projects in mind, please answer the questions that follow: Of the HESN projects (individual research activities/studies) that you participated in since 2015, which projects (please list at least ONE project) best represent the HESN goals of: 1. Accelerating the use of innovative technologies and approaches to address global development challenges 2. Catalyzing an interdisciplinary research environment 3. Supporting evidence-based development decision making Please press enter to list each project name in a separate text box. $projects LIST d_main_project Which HESN Lab(s) were/are these projects (individual research activities/studies) linked to? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,7,8)&&!self.ContainsAny(99 ,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2, 3,4,5,6,7,8)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_lab 01 ResilientAfrica Network (RAN) 02 UC Berkeley – Development Impact Lab (DIL) 03 MIT – International Development Innovation Network (IDIN) 04 MIT – Comprehensive Initiative on Technology Evaluation (CITE) 05 MSU – Global Center for Food Systems Innovation (GCFSI) 06 Texas A&M – Center on Conflict and Development (ConDev) 07 Duke – Social Entrepreneurship Accelerator at Duke (SEAD) 08 William and Mary – AidData 99 Don't know 98 Choose not to answer What specific role(s) did/do you have under these projects? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,7,8,9,10,96)&&!self.Contai nsAny(99,98)) ||(self.ContainsAny(99,98)&&!self.Contains Any(1,2,3,4,5,6,7,8,9,10,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_role 01 HESN Lab director 02 HESN Lab manager 03 Research manager/coordinator 04 Research assistant/analyst 05 Project coordinator/manager 06 Monitoring and evaluation manager 07 Partner outreach coordinator/manager 08 Principal investigator 09 Researcher supporting principal investigator 10 Research fellow 96 Other (please specify) 99 Don't know 98 Choose not to answer E.38 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. HESN PROJECT INFORMATION 11 / 31 What specific role(s) did/do you have under these projects? Please specify other response. d_project_role.Contains(96) TEXT d_project_role_other What are the funding sources for these projects? Please select all that apply. $projects (self.ContainsAny(1,2,3,96)&&!self.ContainsAny(99,98 ))| |(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_fund 01 HESN (through Global Development Lab) 02 USAID Mission 03 External USAID Operating Unit (outside the Global Development Lab) 96 Other (please specify) 99 Don't know 98 Choose not to answer What were/are the funding sources for these projects? Please specify other response. d_project_fund.Contains(96) TEXT d_project_fund_other What sector(s) did/do these projects cover? MULTI-SELECT d_project_sector Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". 01 Agriculture, food distribution, food security 02 Democracy, human rights, and governance 03 Economic growth and trade (including infrastructure) 04 Education 05 Environment and climate change 06 Gender and women’s empowerment 07 Health and nutrition 08 Water and sanitation 09 Crisis and conflict 96 Other (please specify) 99 Don't know 98 Choose not to answer What sector(s) did/do these projects cover? Please specify other response. d_project_sector.Contains(96) TEXT d_project_sector_other In what country/countries were/are these projects being implemented? Ifmore than one country, please press enter to list each country in a s eparate text box. $projects LIST d_project_loc E.39 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. HESN PROJECT INFORMATION 12 / 31 STATIC TEXT Please click on the blue box to proceed to the next section. What were/are the main objectives of these projects? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,7,8,96)&&!self.ContainsAny (99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1 ,2,3,4,5,6,7,8,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_object 01 Inputs for country development strategy or program design 02 Collect data (program monitoring data, demographic, geographic, sectoral, etc.) 03 Impact evaluation of a program 04 Performance evaluation of a program 05 Fill gaps in data 06 Capacity building of USAID Mission/Operating Unit 07 Capacity building of partners 08 Policy development 96 Other (please specify) 99 Don't know 98 Choose not to answer What were/are the main objectives of these projects? Please specify other response. d_project_object.Contains(96) TEXT d_project_object_ot During the life of these projects, how much of your professional time did you spend supporting their activities, on average? $projects SINGLE-SELECT d_project_time_spent 01 Less than 1/4 of time 02 1/4 to 1/2 of time 03 More than 1/2, but less than 3/4 of time 04 3/4 to full time 99 Don't know 98 Choose not to answer E. OVERALL EXPERIENCE WITH HESN 13 / 31 E.OVERALL EXPERIENCE WITH HESN E $familiar STATIC TEXT Based on your experience with HESN, please indicate how strongly you agree with each of the following statements. STATIC TEXT Please click on each of the sub-section boxes to view and respond to statements. Each sub-section box will turn green once completed. E. OVERALL EXPERIENCE WITH HESN HESN HAS EXPANDED THE CAPACITY OF MY LAB'S STAFF TO: STATIC TEXT HESN has expanded the capacity of my Lab's staff to: Please indicate how strongly you agree with each of the following statements. Design research evaluations related to international development (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT e_design_eval 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Collect and analyze data related to international development (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT e_collect_data 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E. OVERALL EXPERIENCE WITH HESN HESN HAS EXPANDED OPPORTUNITIES WITHIN MY LAB FOR: STATIC TEXT HESN has expanded opportunities within my Lab for: Please indicate how strongly you agree with each of the following statements. Faculty/staff to collaborate in the design and implementation of international development research (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT e_collaborate_design 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.41 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. OVERALL EXPERIENCE WITH HESN 14 / 31 Students to work with faculty to conduct international development research (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT e_opportunity_students 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E. OVERALL EXPERIENCE WITH HESN PARTICIPATING IN HESN LAB ACTIVITIES HAS: STATIC TEXT Participating in HESN Lab activities has: Please indicate how strongly you agree with each of the following statements. Provided opportunities for Lab staff to present research at the university. SINGLE-SELECT e_present_uni 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Provided opportunities for Lab staff to present research within the country (but outside the university). SINGLE-SELECT e_present_country 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Provided opportunities for Lab staff to present research abroad. SINGLE-SELECT e_present_abroad 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Created new career opportunties for students. SINGLE-SELECT e_created_opportunities 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.42 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. OVERALL EXPERIENCE WITH HESN 15 / 31 Motivated students to pursue work in international development (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT e_motivated_students 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E. OVERALL EXPERIENCE WITH HESN HESN HAS PROVIDED: STATIC TEXT HESN has provided: Please indicate how strongly you agree with each of the following statements. Additional financial resources to support my Lab. SINGLE-SELECT e_financial_resources 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Sufficient media support to publicize my Lab's research. SINGLE-SELECT e_hes_media_support 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Added value to my Lab and its university. SINGLE-SELECT e_added_value 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Motivation for me to participate in another HESN project if given the opportunity. SINGLE-SELECT e_participate_opportunity 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E.43 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. OVERALL EXPERIENCE WITH HESN 16 / 31 E. OVERALL EXPERIENCE WITH HESN MY LAB'S UNIVERSITY HAS PROVIDED: STATIC TEXT My Lab's university has provided: Please indicate how strongly you agree with each of the following statements. Sufficient institutional financial backing to my Lab. SINGLE-SELECT e_uni_fin 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Sufficient media suppport to publicize my Lab's research. SINGLE-SELECT e_uni_media_support 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. STATIC TEXT Please click on the blue box to proceed to the next section. E.44 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY 17 / 31 F. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY $familiar STATIC TEXT $familiar && $number<=2 Please base your responses on your experience with the HESN projects you worked on. STATIC TEXT $number>2 Please based your responses on your experience with the two HESN projects that you identified as best representing HESN’s goals. STATIC TEXT Please click on each of the sub-section boxes to view and respond to statements. Each sub-section box will turn green once completed. F. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY DURING HESN PROJECT IMPLEMENTATION: STATIC TEXT During HESN project implementation: Please indicate how strongly you agree with each of the following statements. Communication with our AOR in the Center for Development Research was smooth. SINGLE-SELECT f_comm_aor 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Communication with USAID Missions/Operating Units was smooth throughout HESN. SINGLE-SELECT f_comm_usaid 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know We experienced major challenges in engaging with our AOR in the Center for Development Research. SINGLE-SELECT f_exp_challenge_aor 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.45 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY 18 / 31 We overcame the challenges we experienced in working with our AOR in the Center for Developmetn Research. SINGLE-SELECT f_over_challenge_aor 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know We experienced major challenges in engaging with USAID Missions/Operating Units. SINGLE-SELECT f_exp_challenge_usaid 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know We overcame the challenges we experienced in working with USAID Missions/Operating Units. SINGLE-SELECT f_over_challenge_usaid 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Our original plans to work under the HESN core funding were implemented well and achieved the desired goals. SINGLE-SELECT f_orig_plan_core 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Our original plans to work under the HESN buy-in were implemented well and achieved the desired goals. SINGLE-SELECT f_orig_buyin 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. F. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY OUTPUTS FROM HESN PROJECTS WERE: STATIC TEXT Outputs from HESN projects were: Please indicate how strongly you agree with each of the following statements. E.46 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY 19 / 31 Used by USAID Mission and/or Operating Unit staff as inputs for their country development strategy or program design. SINGLE-SELECT f_used_usaid 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Relevant to USAID Missions/Operating Units for planning future programs. SINGLE-SELECT f_relevant_plan 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Delivered during a time period when USAID Missions/Operating Units needed the information for policy, planning, or learning. SINGLE-SELECT f_deliver_time 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Used by policymakers in the target developing country for planning and policy making. SINGLE-SELECT f_used_policy 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. F. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY HESN LED TO MORE OPPORTUNITES FOR MY LAB TO: STATIC TEXT HESN led to more opportunites for my Lab to: Please indicate how strongly you agree with each of the following statements. Broaden the research conducted. SINGLE-SELECT f_lab_broaden_research 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.47 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY 20 / 31 Conduct consulting in international development. SINGLE-SELECT f_lab_opp_lab_consult 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Conduct additional research work within the next year with external funding. SINGLE-SELECT f_lab_provide_lab_fund 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Engage with policymakers in developing countries around our research findings. SINGLE-SELECT f_lab_engage_policy_find 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. F. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY HESN LED TO MORE OPPORTUNITES FOR MY LAB'S UNIVERSITY TO: STATIC TEXT HESN led to more opportunites for my Lab's university to: Please indicate how strongly you agree with each of the following statements. Increase its contact with international development programming. SINGLE-SELECT f_uni_increase_contact 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Grow its international development research work. SINGLE-SELECT f_uni_grow_intl_work 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.48 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY 21 / 31 Increase its focus on international development research. SINGLE-SELECT f_uni_increase_focus 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Provide students and faculty with new opportunities to conduct research in international development work. SINGLE-SELECT f_uni_provide_opportunity 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. F. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY MY EXPERIENCE ENGAGING WITH A HESN LAB HELPED ME: STATIC TEXT My experience engaging with a HESN Lab helped me: Please indicate how strongly you agree with each of the following statements. Develop additional administrative and managerial skills. SINGLE-SELECT f_exp_adm_skills 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Develop additional research skills. SINGLE-SELECT f_exp_skills 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Advance my career. SINGLE-SELECT f_exp_career 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E.49 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY 22 / 31 F. HESN IMPLEMENTATION, USEFULNESS, AND SUSTAINABILITY HESN PROJECTS: STATIC TEXT HESN projects: Please indicate how strongly you agree with each of the following statements. Note: We define HESN projects as specific work undertaken by a HESN university Lab such as conducting a study, undertaking a country-level mapping/geo-referencing activity, holding an innovation competition, or hosting a fellowship program. Contributed to changes in my Lab's university that allow students and faculty to do more work in international development (e.g. led to new coursework, research opportunities, increased funding). SINGLE-SELECT f_change_dev_uni 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Are a positive example of how different stakeholders can work together in partnership to achieve international development goals. SINGLE-SELECT f_positive_ex 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Brought changes that will be sustained in the next five years. SINGLE-SELECT f_changes_sustain 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. STATIC TEXT Please click on the blue box to proceed to the next section. E.50 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 23 / 31 G. HESN RESEARCH OUTPUTS AND DISSEMINATION $familiar && $projects STATIC TEXT Now we would like to ask you a few questions about the dissemination of results/research outputs from HESN project (individual research activities/studies). Once again, think about the HESN projects (individual research activities/studies) that you considered to best represent HESN's goals. With these projects inmind, please answer the questions that follow: What research results/outputs were/will be produced as part of these projects? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,7,96)&&!self.ContainsAny(9 9,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2 ,3,4,5,6,7,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_research 01 Report or publication 02 Policy brief 03 Toolkit 04 Data set/platform/mapping 05 Evaluation 06 Training 07 Event or conference 96 Other (please specify) 99 Don't know 98 Choose not to answer What research results/outputs were/will be produced as part of these projects? Please specify other response. g_project_research.Contains(96) TEXT g_project_research_other Did these projects already produce research SINGLE-SELECT g_project_produced results/outputs for sharing? 01 Yes 00 No 99 Don't know What were the research results/outputs under these projects used for? Please select all that apply. $produced (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98,5 ,6 ))||(self.ContainsAny(99,98,5,6)&&!self.ContainsAny(1 ,2,3,4,96)) Your answers cannot include both substantive answers and "not used yet butwill be used in the future"/"not used for anything"/"don't know"/ "choose not to answer". MULTI-SELECT g_project_use 01 Inputs for country development strategy or program design 02 Capacity building of USAID Mission/Operating Unit 03 Capacity building of government in target country 04 Policy development 05 Not used yet but will be used in the future 06 Not used for anything 96 Other specified use 99 Don't know 98 Choose not to answer What were the research results/outputs under these projects used for? Please specify other response. g_project_use.Contains(96) TEXT g_project_use_other E.51 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 24 / 31 Why were the research results/outputs produced under these projects not used for anything? g_project_use.Contains(6) TEXT g_project_no_use When do you expect that research results/outputs produced under these projects will be used? g_project_use.Contains(5) TEXT g_project_use_when What will the research results/outputs under these projects be used for? Please select all that apply. g_project_use.Contains(5) (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98 ) )||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4,9 6)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_exp_use 01 Inputs for country development strategy or program design 02 Capacity building of Mission/Operating unit 03 Capacity building of government in target country 04 Policy development 96 Other (please specify) 99 Don't know 98 Choose not to answer What will the research results/outputs under these projects be used for? Please specify other response. g_project_exp_use.Contains(96) TEXT g_project_use_exp_ot How did your Lab disseminate research results/outputs from these projects? Please select all that apply. $produced (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98,10,11 ))||(self.ContainsAny(99,98,10,11)&&!self .ContainsAny(1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "Not disse minated yet but will be in the future"/"not disseminated"/"don't know"/ "choose not to answer". MULTI-SELECT g_project_dissem 01 Email 02 Social media 03 Podcast 04 Blog 05 Policy/issue brief 06 Virtual results presentation 07 In-person results presentation 08 Dissemination workshop 09 In the middle of dissemination 10 Not disseminated yet but will be in the future 11 Not disseminated 96 Other (please specify) 99 Don't know 98 Choose not to answer How did your Lab disseminate research results/outputs from these projects? Please specify other response. g_project_dissem.Contains(96) TEXT g_project_dissem_ot When do you expect that research results/outputs produced under these projects will be disseminated? g_project_dissem.Contains(10) TEXT g_project_dissem_when E.52 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 25 / 31 G. HESN RESEARCH OUTPUTS AND DISSEMINATION Roster: AUDIENCE generated bymulti-select question g_project_dissem_aud g_audience g_project_dissem_aud.ContainsAny(1,2,3,4,5,6,7,96) Why were the research results/outputs produced under these projects not disseminated? g_project_dissem.Contains(11) TEXT g_project_no_dissem To what audience did your Lab directly disseminate research results/outputs from these projects? Please select all that apply. g_project_dissem.ContainsAny(1,2,3,4,5,6,7,8,9,96) (self.ContainsAny(1,2,3,4,5,6,7,96)&&!self.ContainsAny(9 9,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2 ,3,4,5,6,7,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_dissem_aud 01 USAID Mission(s) 02 External USAID Operating Unit(s) (outside the Global Development Lab) 03 Policymakers in the country where research took place 04 Researchers in the country where research took place 05 Other stakeholder organizations where the research took place 06 Policymakers in other countries 07 Researchers in other countries 96 Other audience 99 Don't know 98 Choose not to answer To what audience did your Lab directly disseminate research results/outputs from these projects? Please specify other audience. g_project_dissem_aud.Contains(96) TEXT g_project_dissem_aud_ot How many times did you speak with the %g_audience% about research results/outputs fromthese projects? self < 500 && self>=0 This number does not seemcorrect, please double check. NUMERIC: INTEGER g_aud_spoke_num Do you know the position of key representatives from the %g_audience% with whom you spoke to about results/outputs fromthese projects? SINGLE-SELECT g_aud_spoke_know 01 Yes 02 No Who were the key representatives (organization and position) from the %g_audience% with whom you spoke about research results/outputs from these projects? For each key person please just list their position and organization. g_aud_spoke_know==1 LIST g_aud_spoke_ppl E.53 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 26 / 31 Please indicate your level of agreement with: the %g_audience% showed strong interest in research results from these projects? SINGLE-SELECT g_aud_interest 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Doesn't know Please indicate your level of agreement with: the %g_audience% indicated that research results/outputs from these projects were useful to their planning or programs? SINGLE-SELECT g_aud_use 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Doesn't know What facilitated the use of research results/outputs produced under these projects? Please select all that apply. $produced (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_facil 01 Timeline for completing the product 02 Daily implementation of activities 03 Support from policymakers due to relevance of topic 04 Geography in-country facilitates dissemination (i.e. easy access to internet, regional distribution point, easy access to urban and rural areas) 05 Local interest/desire for research product 06 Having financial resources to create products 07 Having financial resources to disseminate products 08 Having financial or in-kind incentives for partners 09 Expertise and reputation of HESN Lab 96 Other (please specify) 99 Don't know 98 Choose not to answer What facilitated the use of research results/outputs produced under these projects? Please specify other response. g_project_facil.Contains(96) TEXT g_project_facil_ot Why do you think the items you selected above facilitated the use of results/research outputs fromthese projects? g_project_facil.ContainsAny(1,2,3,4,5,6,7,8,9,96) TEXT g_project_facil_why E.54 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 27 / 31 What inhibited the use of research results/outputs produced under these projects? Please select all that apply. $produced (self.ContainsAny(1,2,3,4,5,6,7,8,9,10,11,96)&&!self.Con tainsAny(99,98 ))||(self.ContainsAny(99,98)&&!self.Conta insAny(1,2,3,4,5,6,7,8,9,10,11,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_inhib 01 Contractual requirements 02 Reporting requirements 03 Delays in completing research products 04 Lack of support from policymakers due to irrelevance of topic 05 Geography in-country inhibits dissemination (i.e. lack of consistent internet, no regional distribution points, difficult access to urban and rural areas) 06 Lack of local interest/desire for research product 07 Lack of financial resources to create products 08 Lack of financial resources to disseminate product 09 Lack of financial or in-kind incentives for partners 10 Limits on ability to publish/copyright issues 11 Political pressure 96 Other (please specify) 99 Don't know 98 Choose not to answer What inhibited the use of research results/outputs produced under the these projects? Please specify other response. g_project_inhib.Contains(96) TEXT g_project_inhib_ot Why do you think the items you selected above inhibited the use of results/research outputs fromthese projects? g_project_inhib.ContainsAny(1,2,3,4,5,6,7,8,9,10,11,96) TEXT g_project_inhib_why Did any additional collaborations result beyond the original scope of these projects? SINGLE-SELECT g_project_additional 01 Yes 00 No 99 Don't know E.55 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 28 / 31 STATIC TEXT Please click on the blue box to proceed to the next section. What are the main objectives of the additional collaborations beyond these projects? Please select all that apply. $additional (self.ContainsAny(1,2,3,4,5,6,7,8,96)&&!self.ContainsAny (99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1 ,2,3,4,5,6,7,8,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_add_obj 01 Inputs for country development strategy or program design 02 Collect data (program monitoring data, demographic, geographic, sectoral, etc.) 03 Impact evaluation of a program 04 Performance evaluation of program 05 Fill gaps in data 06 Capacity building of USAID Mission/Operating Unit 07 Capacity building of partners 08 Policy development 96 Other (please specify) 99 Don't know 98 Choose not to answer What are the main objectives of the additional collaborations beyond these projects? Please specify other response. g_project_add_obj.Contains(96) TEXT g_project_add_obj_ot What are/will be the funding sources for the additional collaborations beyond these projects? Please select all that apply. $additional (self.ContainsAny(1,2,3,96)&&!self.ContainsAny(99,98 ))| |(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_add_fund 01 Global Development Lab 02 USAID Mission 03 External USAID Operating Unit (outside the Global Development Lab) 96 Other (please specify) 99 Don't know 98 Choose not to answer What are/will be the funding sources for the additional collaborations beyond these projects? Please specify other response. g_project_add_fund.Contains(96) TEXT g_project_add_fund_ot In what country/countries will the additional collaborations beyond these projects be implemeneted? Ifmore than one country, please press enter to list each country in a s eparate text box. $additional LIST g_project_add_loc H. RECOMMENDATIONS $familiar E.56 | PERFORMANCE EVALUATION OF HESN USAID.GOV H. RECOMMENDATIONS 29 / 31 STATIC TEXT Please click on the blue box to proceed to the next section. What would make a mechanism such as HESN more beneficial for your Lab? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,96)&&!self.ContainsAny(99, 98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3 ,4,5,6,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT h_beneficial 01 More opportunities for faculty to join research projects 02 More robust funding for the HESN Lab itself 03 Stronger emphasis on student engagement in research 04 Stronger emphasis on student engagement in fieldwork 05 Better alignment with key stakeholder interests 06 Greater continuity from HESN and USAID for sustainability of research capacity 96 Other (please specify) 99 Don't know 98 Choose not to answer What would make a mechanism such as HESN more beneficial for your Lab? Please specify other response. h_beneficial.Contains(96) TEXT h_beneficial_ot Do you have any other comments or suggestions related to your experience with HESN? SINGLE-SELECT h_has_comment 01 Yes (please specify) 00 No Do you have any other comments or suggestions related to your experience with HESN? Please specify any other comments. h_has_comment==1 TEXT h_comment_specify I. FOLLOW-UP $familiar E.57 | PERFORMANCE EVALUATION OF HESN USAID.GOV I. FOLLOW-UP 30 / 31 In order to learn more about the utility of the HESN mechanism and research outputs, we would like to conduct short in-person or phone interviews with a subsample of online survey respondents. Would you be willing to participate in a short follow-up interview? SINGLE-SELECT i_follow_part 01 Yes 00 No STATIC TEXT $follow Please provide the below information: STATIC TEXT Thank you for your time! Please click on the box below to record the survey end time. DATE: CURRENT TIME i_end_time STATIC TEXT Please click on the blue box to proceed to the next section. First name: $follow TEXT i_first_name Last name: $follow TEXT i_last_name Email: $follow TEXT i_email Phone number (with country code): $follow TEXT i_phone Current organization: $follow TEXT i_organ Current position: $follow TEXT i_position E.58 | PERFORMANCE EVALUATION OF HESN USAID.GOV LEGEND 31 / 31 LEGEND Legend and structure of information in this file This page has been left blank for double-sided copying. Generated by edithgfelix, Apr 06, 2020 12:30 Questionnaire created by edithgfelix, Dec 02, 2019 15:39 Last modified by edithgfelix, Dec 24, 2019 14:43 Not shared with anyone Sections: 6, Sub-sections: 7, Questions: 73. Questions with enabling conditions: 21 Questions with validation conditions:5 Rosters: 1 Variables: 0 HESN Higher Education Institution Survey SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION A. INTRODUCTION No sub-sections, No rosters,Questions: 4, Static texts: 2. B. GENERAL INFORMATION No sub-sections,Rosters: 1,Questions: 18, Static texts: 2. C. OVERALL EXPERIENCE WITH HESN Sub-sections: 4,No rosters,Questions: 14, Static texts: 11. D. HESN USEFULNESS AND SUSTAINABILITY Sub-sections: 3,No rosters,Questions: 24, Static texts: 9. E. RECOMMENDATIONS No sub-sections, No rosters,Questions: 5, Static texts: 1. F. FOLLOW-UP No sub-sections, No rosters,Questions: 8, Static texts: 3. LEGEND E.60 | PERFORMANCE EVALUATION OF HESN USAID.GOV 1 / 19 E.61 | PERFORMANCE EVALUATION OF HESN USAID.GOV SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION 2 / 19 SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION Basic information Title HESN Higher Education Institution Survey E.62 | PERFORMANCE EVALUATION OF HESN USAID.GOV A. INTRODUCTION 3 / 19 A. INTRODUCTION STATIC TEXT ● This survey gathers information about your perspectives and experience with the Higher Education Solutions Network (HESN). ● If you have participated in another USAID Global Development Lab program such as LASER, RTAC, STIP APS, or PEER, please only provide information related to your perspectives and experience with HESN. ● USAID’s Global Development Lab has contracted Mathematica to conduct an independent evaluation of the long-term impacts of HESN. ● USAID is interested in understanding the ultimate impact/utility of HESN projects implemented since 2015 and the types of partnerships thatmay produce policy impact. ● This survey should take about 30 minutes to complete. ● Any information you provide that can identify you will be kept strictly confidential. Information provided will be used for research purposes only. Your participation is voluntary and you may choose not to answer any or all questions for any reason. ● You may contact Audrey-Marie Moore, the Mathematica Senior Researcher leading this evaluation at AMoore@mathematica-mpr.com, if you have study questions, concerns or complaints. Thank you for your time and help with this evaluation! Please click on the box below to record the survey start time. DATE: CURRENT TIME a_start_time Do you consent to participate in this survey? SINGLE-SELECT a_consent 01 Yes 00 No Name TEXT prefilled_name SCOPE: IDENTIFYING Email TEXT prefilled_email SCOPE: IDENTIFYING STATIC TEXT Please click on the box to proceed to the next section. E.63 | PERFORMANCE EVALUATION OF HESN USAID.GOV B. GENERAL INFORMATION 4 / 19 B. GENERAL INFORMATION a_consent==1 STATIC TEXT To begin, we would like to ask you some general information about the Higher Education Solutions Network (HESN). Please see the below definitions: HESN: A cooperative agreement between USAID and eight university-based development labs The eight labs use HESN core funds and buy-in funding from USAID Missions/Operating Units to incubate, test, and accelerate solutions to development challenges. HESN Lab: Includes any of the following Labs: CITE at MIT, IDIN at MIT, DIL at UC Berkeley, GCFSI at MSU, ConDev at Texas A&M, SEAD at Duke, AidData at William and Mary, and the ResilientAfrica Network (RAN). HESN project: Specific work undertaken by an HESN university Lab such as conducting a study, undertaking a country- level mapping/geo-referencing activity, holding an innovation competition, or hosting a fellowship program. What is the name of the university that you work under? SINGLE-SELECT b_hei_name 01 Massachusetts Institute of Technology 02 University of California, Berkeley 03 College of William and Mary 04 Michigan State University 05 Duke University 06 Texas A&M University 07 Makerere University 96 Other (please specify) 99 Don't know 98 Choose not to answer What is the name of the university that you work under? Please specify other response. b_hei_name==96 TEXT b_hei_name_ot What is your title/role within your university? Please select all that apply. (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98)) ||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4,96 )) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT b_uni_role 01 Dean of a department 02 Associate dean of a department 03 Director of a department 04 Associate director of a department 96 Other (please specify) 99 Don't know 98 Choose not to answer What is your title/role within your university? Please specify other response. b_uni_role.Contains(96) TEXT b_uni_role_ot E.64 | PERFORMANCE EVALUATION OF HESN USAID.GOV B. GENERAL INFORMATION 5 / 19 What sector(s) are the focus of your department? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,7,96)&&!self.ContainsAny(9 9,98)) ||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2 ,3,4,5,6,7,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT b_dep_sec 01 Education 02 Economics 03 Agriculture 04 Information, Communication and Technology 05 Physical Sciences 06 Natural Sciences 07 Social Sciences 96 Other (please specify) 99 Don't know 98 Choose not to answer What sector(s) are the focus of your department? Please select all that apply. b_dep_sec.Contains(96) TEXT b_dep_sec_ot Are you familiar with the the Higher Education Solutions Network (HESN)? SINGLE-SELECT b_familiar 01 Yes 00 No How did you first become aware of HESN? Please select all that apply. $familiar (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98)) ||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4,96 )) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT b_first_aware 01 HESN Lab staff 02 My university's administration 03 Other higher education institution 04 USAID communications 96 Other (please specify) 99 Don't know 98 Choose not to answer How did you first become aware of HESN? Please specify other response. b_first_aware.Contains(96) TEXT b_first_aware_other Is any HESN Lab housed within your university? $familiar SINGLE-SELECT b_has_lab 01 Yes 02 No 99 Don't know 98 Choose not to answer B. GENERAL INFORMATION 6 / 19 B. GENERAL INFORMATION Roster: WORK WITH HESN LAB generated bymulti-select question b_lab_partner b_lab_work b_lab_partner.ContainsAny(1,2,3,4,5,6,7,8) Which HESN Lab is housed within your university? b_has_lab==1 SINGLE-SELECT b_lab_name 01 ResilientAfrica Network (RAN) 02 UC Berkeley – Development Impact Lab (DIL) 03 MIT – International Development Innovation Network (IDIN) 04 MIT – Comprehensive Initiative on Technology Evaluation (CITE) 05 MSU – Global Center for Food Systems Innovation (GCFSI) 06 Texas A&M – Center on Conflict and Development (ConDev) 07 Duke – Social Entrepreneurship Accelerator at Duke (SEAD) 08 William and Mary – AidData 99 Don't know 98 Choose not to answer How closely did you work with the HESN Lab housed within your university? b_has_lab==1 SINGLE-SELECT b_lab_close 01 Very closely 02 Somewhat closely 03 Not closely 04 Did not work with the HESN Lab 99 Don't know 98 Choose not to answer Did your university partner with any HESN Lab? $familiar SINGLE-SELECT b_has_lab_partner 01 Yes 02 No 99 Don't know 98 Choose not to answer With which HESN Lab(s) did your university partner? Please select all that apply. b_has_lab_partner==1 (self.ContainsAny(1,2,3,4,5,6,7,8)&&!self.ContainsAny(99 ,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2, 3,4,5,6,7,8)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT b_lab_partner 01 ResilientAfrica Network (RAN) 02 UC Berkeley – Development Impact Lab (DIL) 03 MIT – International Development Innovation Network (IDIN) 04 MIT – Comprehensive Initiative on Technology Evaluation (CITE) 05 MSU – Global Center for Food Systems Innovation (GCFSI) 06 Texas A&M – Center on Conflict and Development (ConDev) 07 Duke – Social Entrepreneurship Accelerator at Duke (SEAD) 08 William and Mary – AidData 99 Don't know 98 Choose not to answer B. GENERAL INFORMATION 7 / 19 STATIC TEXT Please click on the blue box to proceed to the next section. E.66 | PERFORMANCE EVALUATION OF HESN USAID.GOV How closely did you work with %b_lab_work%? SINGLE-SELECT b_lab_partner_close 01 Very closely 02 Somewhat closely 03 Not closely 04 Did not work with the HESN Lab 99 Don't know 98 Choose not to answer Did your department participate in any HESN projects (individual research activities/studies) undertaken by a HESN Lab? $familiar SINGLE-SELECT b_dept_part_hesn 01 Yes 02 No 99 Don't know 98 Choose not to answer How many HESN projects (individual research activities/studies) has your department participated in since 2015? b_dept_part_hesn==1 SINGLE-SELECT b_number_projects 01 1 to 2 projects 02 3 to 5 projects 03 6 to 10 projects 04 More than 10 projects 95 Did not participate in any individual projects 99 Don't know 98 Choose not to answer Of the HESN projects (individual research activities/studies) that your department participated in since 2015, which projects (please list at least ONE project) best represent the HESN goals of: 1. Accelerating the use of innovative technologies and approaches to address global development challenges 2. Catalyzing an interdisciplinary research environment 3. Supporting evidence-based development decision making Please press enter to list each project name in a separate text box. b_number_projects==1|| b_number_projects==2||b_number_pr ojects==3||b_number_projects==4 LIST b_main_project C. OVERALL EXPERIENCE WITH HESN 8 / 19 C. OVERALL EXPERIENCE WITH HESN $familiar STATIC TEXT Based on your experience with HESN, please indicate how strongly you agree with each of the following statements. STATIC TEXT Please click on each of the sub-section boxes to view and respond to statements. Each sub-section box will turn green once completed. C. OVERALL EXPERIENCE WITH HESN HESN HAS EXPANDED THE CAPACITY IN MY DEPARTMENT TO: STATIC TEXT HESN has expanded the capacity in my department to: Please indicate how strongly you agree with each of the following statements. Design research evaluations related to international development (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT c_design_eval 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Collect and analyze data related to international development (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT c_collect_data 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. C. OVERALL EXPERIENCE WITH HESN HESN HAS EXPANDED OPPORTUNITIES WITHIN MY DEPARTMENT FOR: STATIC TEXT HESN has expanded opportunities within my department for: Please indicate how strongly you agree with each of the following statements. Faculty/staff to collaborate in the design and implementation of international development research (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT c_collaborate_design 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know C. OVERALL EXPERIENCE WITH HESN 9 / 19 Students to work with faculty to conduct international development research (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT c_opportunity_students 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. C. OVERALL EXPERIENCE WITH HESN PARTICIPATING IN HESN LAB ACTIVITIES HAS: STATIC TEXT Participating in HESN Lab activities has: Please indicate how strongly you agree with each of the following statements. Provided opportunities for faculty to present research at the university. SINGLE-SELECT c_present_uni 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Provided opportunities for faculty to present research within the country (but outside the university). SINGLE-SELECT c_present_country 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Provided opportunities for faculty to present research abroad. SINGLE-SELECT c_present_abroad 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Provided opportunities for faculty and students to engage with universities outside of the country. SINGLE-SELECT c_engage_abroad 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.68 | PERFORMANCE EVALUATION OF HESN USAID.GOV E.69 | PERFORMANCE EVALUATION OF HESN USAID.GOV C. OVERALL EXPERIENCE WITH HESN 10 / 19 Created new career opportunties for students. SINGLE-SELECT c_created_opportunities 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Motivated students to pursue work in international development (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT c_motivated_students 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. C. OVERALL EXPERIENCE WITH HESN HESN HAS PROVIDED: STATIC TEXT HESN has provided: Please indicate how strongly you agree with each of the following statements. Additional financial resources to support research within my university. SINGLE-SELECT c_financial_resources 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Technical resources to support faculty in my university as they improve their research and evaluations. SINGLE-SELECT c_tech_resources 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Added value to my department and university. SINGLE-SELECT c_added_value 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.70 | PERFORMANCE EVALUATION OF HESN USAID.GOV C. OVERALL EXPERIENCE WITH HESN 11 / 19 Opportunities to extend my university's research network. SINGLE-SELECT c_extend_network 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. STATIC TEXT Please click on the blue box to proceed to the next section. D. HESN USEFULNESS AND SUSTAINABILITY 12 / 19 D. HESN USEFULNESS AND SUSTAINABILITY $familiar STATIC TEXT Based on your experience with HESN, please indicate how strongly you agree with each of the following statements. STATIC TEXT Please click on each of the sub-section boxes to view and respond to statements. Each sub-section box will turn green once completed. D. HESN USEFULNESS AND SUSTAINABILITY RESULTS/OUTPUTS OF HESN-SUPPORTED ACTIVITIES: STATIC TEXT Results/outputs of HESN-supported activities: Please indicate how strongly you agree with each of the following statements. Have contributed to policy-making in the target country or countries at the local level. SINGLE-SELECT d_contribute_policy_loc 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Have contributed to policy-making in the target country or countries at the national level. SINGLE-SELECT d_contribute_policy_nat 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Have been disseminated across the university. SINGLE-SELECT d_dissem_uni 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Have been disseminated outside the university system. SINGLE-SELECT d_dissem_outside 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. D. HESN USEFULNESS AND SUSTAINABILITY 13 / 19 D. HESN USEFULNESS AND SUSTAINABILITY THE HESN WORK PERFORMED WITHIN MY DEPARTMENT: STATIC TEXT The HESN work performed within my department: Please indicate how strongly you agree with each of the following statements. Produced useful research products for USAID Missions and other stakeholders. SINGLE-SELECT d_useful_research 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Was relevant to development (poverty alleviation, social innovation, public health, etc.) issues that faculty, staff, and students in my department work on. SINGLE-SELECT d_rel_issues_dept 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Was relevant to development (poverty alleviation, social innovation, public health, etc.) issues in the target country or countries. SINGLE-SELECT d_rel_issues_country 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Was timely for the expansion of my department's international development (poverty alleviation, social innovation, public health, etc.) research. SINGLE-SELECT d_timely_expan 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Gave my university an opportunity to engage in pertinent research that contributed to international development issues in my country (if applicable). SINGLE-SELECT d_engage_research 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Was timely in terms of the international development issues facing policy-makers in my country (if applicable). SINGLE-SELECT d_timely_issues_policy 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know D. HESN USEFULNESS AND SUSTAINABILITY 14 / 19 Contributed to new programs or coursework in my department that focus on development issues (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT d_new_programs 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Helped expand my department’s international network. SINGLE-SELECT d_expand_network 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Helped broaden the research conducted within my department. SINGLE-SELECT d_broad_research 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Led to partnerships between my department, USAID Missions and other stakeholders that are beneficial to all parties. SINGLE-SELECT d_partner_ben 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. D. HESN USEFULNESS AND SUSTAINABILITY HESN HAS: STATIC TEXT HESN has: Please indicate how strongly you agree with each of the following statements. Increased opportunities for faculty to participate in international development (poverty alleviation, social innovation, public health, etc.) projects. SINGLE-SELECT d_num_opp_fac 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.73 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. HESN USEFULNESS AND SUSTAINABILITY 15 / 19 Improved the quality of international development (poverty alleviation, social innovation, public health, etc.) project opportunities for faculty. SINGLE-SELECT d_qual_opp_fac 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Increased opportunities for students to participate in international development (poverty alleviation, social innovation, public health, etc.) projects. SINGLE-SELECT d_num_opp_stud 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Improved the quality of international development (poverty alleviation, social innovation, public health, etc.) project opportunities for students. SINGLE-SELECT d_qual_opp_stud 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Generated faculty opportunities that will remain over the next 12-18 months. SINGLE-SELECT d_remain_opp_fac 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Generated faculty opportunities that will expand over the next 12-18 months. SINGLE-SELECT d_remain_expand_fac 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Generated student opportunities that will remain over the next 12-18 months. SINGLE-SELECT d_remain_opp_stud 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Generated student opportunities that will expand over the next 12-18 months. SINGLE-SELECT d_expand_opp_stud 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know D. HESN USEFULNESS AND SUSTAINABILITY 16 / 19 Improved research funding at the institutional level for the future. SINGLE-SELECT d_fund_inst 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Improved the ability of my department to cultivate donors for research centers at the institutional level for the future. SINGLE-SELECT d_cult_donors 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. STATIC TEXT Please click on the blue box to proceed to the next section. E.75 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. RECOMMENDATIONS $familiar E.76 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. RECOMMENDATIONS 17 / 19 STATIC TEXT Please click on the blue box to proceed to the next section. Please indicate your level of agreement with: Our university would likely participate in or support another HESN research or innovation activity if given the opportunity. SINGLE-SELECT e_another_opp 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know What would make a mechanism such as HESN more beneficial for your department? Please select all that apply. (self.ContainsAny(1,2,3,4,96)&&!self.Contains(99 ))||(se lf.Contains(99)&&!self.ContainsAny(1,2,3,4,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT e_rec_ben 01 More opportunities for faculty to join research projects 02 More robust funding for the HESN Lab itself 03 Stronger emphasis on student engagement in research and fieldwork 04 Greater continuity and support from HESN for sustainability of research capacity 96 Other (please specify) 99 Don't know 98 Choose not to answer What would make a mechanism such as HESN more beneficial for your department? Please specify other response. e_rec_ben.Contains(96) TEXT e_rec_ben_ot Do you have any other comments or suggestions related to your experience with HESN? SINGLE-SELECT e_has_comment 01 Yes (please specify) 00 No Do you have any other comments or suggestions related to your experience with HESN? Please specify any other comments. e_has_comment==1 TEXT e_comment_specify F. FOLLOW-UP $familiar E.77 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. FOLLOW-UP 18 / 19 In order to learn more about the utility of the HESN mechanism and research outputs, we would like to conduct short in-person or phone interviews with a subsample of online survey respondents. Would you be willing to participate in a short follow-up interview? SINGLE-SELECT f_follow_part 01 Yes 00 No STATIC TEXT $follow Please provide the below information: STATIC TEXT Thank you for your time! Please click on the box below to record the survey end time. DATE: CURRENT TIME f_end_time STATIC TEXT Please click on the blue box to proceed to the next section. First name: $follow TEXT f_first_name Last name: $follow TEXT f_last_name Email: $follow TEXT f_email Phone number (with country code): $follow TEXT f_phone Current organization: $follow TEXT f_organ Current position: $follow TEXT f_position E.78 | PERFORMANCE EVALUATION OF HESN USAID.GOV LEGEND 19 / 19 LEGEND Legend and structure of information in this file This page has been left blank for double-sided copying. Generated by edithgfelix, Apr 06, 2020 12:30 Questionnaire created by edithgfelix, Dec 19, 2019 21:40 Last modified by edithgfelix, Dec 20, 2019 00:19 Not shared with anyone Sections: 9, Sub-sections: 7, Questions: 112. Questions with enabling conditions: 54 Questions with validation conditions:25 Rosters: 1 Variables: 0 HESN Researcher Survey SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION A. INTRODUCTION No sub-sections, No rosters,Questions: 4, Static texts: 2. B. GENERAL INFORMATION No sub-sections, No rosters,Questions: 5, Static texts: 2. C. RESEARCH EXPERIENCE No sub-sections, No rosters,Questions: 12, Static texts: 2. D. HESN PROJECT INFORMATION No sub-sections, No rosters,Questions: 15, Static texts: 3. E. DEPARTMENT-LEVEL EXPERIENCE WITH HESN Sub-sections: 4,No rosters,Questions: 13, Static texts: 11. F. HESN USEFULNESS AND SUSTAINABILITY Sub-sections: 3,No rosters,Questions: 19, Static texts: 9. G. HESN RESEARCH OUTPUTS AND DISSEMINATION No sub-sections,Rosters: 1,Questions: 32, Static texts: 2. H. RECOMMENDATIONS No sub-sections, No rosters,Questions: 4, Static texts: 1. I. FOLLOW-UP No sub-sections, No rosters,Questions: 8, Static texts: 3. LEGEND E.80 | PERFORMANCE EVALUATION OF HESN USAID.GOV 1 / 28 E.81 | PERFORMANCE EVALUATION OF HESN USAID.GOV SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION 2 / 28 SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION Basic information Title HESN Researcher Survey E.82 | PERFORMANCE EVALUATION OF HESN USAID.GOV A. INTRODUCTION 3 / 28 A. INTRODUCTION STATIC TEXT ● This survey gathers information about your perspectives and experience with the Higher Education Solutions Network (HESN). ● If you have participated in another USAID Global Development Lab program such as LASER, RTAC, STIP APS, or PEER, please only provide information related to your perspectives and experience with HESN. ● USAID’s Global Development Lab has contracted Mathematica to conduct an independent evaluation of the long-term impacts of HESN. ● USAID is interested in understanding the ultimate impact/utility of HESN projects implemented since 2015 and the types of partnerships thatmay produce policy impact. ● In this survey we define a HESN Lab, HESN project, and participation in HESN projects as follows: HESN Lab: The university-based umbrella partnership; for example, AidData at the College of William and Mary, the Development Impact Lab (DIL) at the University of California Berkeley, or the Center on Conflict and Development (ConDev) at Texas A&M. HESN project: Specific work undertaken by an HESN university Lab such as conducting a study, undertaking a country- level mapping/geo-referencing activity, holding an innovation competition, or hosting a fellowship program. Participation in an HESN project: This is meant to be viewed broadly and could include a range of ways of engaging such as providing technical advice about a project scope or its implementation, serving as a buy-in point of contact, collecting data, monitoring project progress, activity management, review of deliverables, communication about the project, review of projectmaterials to informnew programming etc. ● This survey should take about 30 minutes to complete. Any information you provide that can identify you will be kept strictly confidential. Information provided will be used for research purposes only. Your participation is voluntary and youmay choose not to answer any or all questions for any reason. ● You may contact Audrey-Marie Moore, the Mathematica Senior Researcher leading this evaluation at AMoore@mathematica-mpr.com, if you have study questions, concerns or complaints. Thank you for your time and help with this evaluation! Please click on the box below to record the survey start time. DATE: CURRENT TIME a_start_time Do you consent to participate in this survey? SINGLE-SELECT a_consent 01 Yes 00 No Name TEXT prefilled_name SCOPE: IDENTIFYING Email TEXT prefilled_email SCOPE: IDENTIFYING STATIC TEXT Please click on the box to proceed to the next section. B. GENERAL INFORMATION a_consent==1 E.83 | PERFORMANCE EVALUATION OF HESN USAID.GOV B. GENERAL INFORMATION 4 / 28 STATIC TEXT To begin, we would like to ask you some general information about the Higher Education Solutions Network (HESN). STATIC TEXT Please click on the blue box to proceed to the next section. Are you familiar with the the Higher Education Solutions Network (HESN)? SINGLE-SELECT b_familiar 01 Yes 00 No How did you first become aware of HESN? Please select all that apply. $familiar (self.ContainsAny(1,2,3,4,5,6,7,8,96)&&!self.ContainsAny (99,98)) ||(self.ContainsAny(99,98)&&!self.ContainsAny(1 ,2,3,4,5,6,7,8,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT b_first_aware 01 Faculty from my university’s department 02 Peers from my university but outside my department 03 Peers at other institutions 04 University administration 05 USAID communications 06 HESN Lab event 07 HESN Lab communications 08 HESN Lab research output 96 Other (please specify) 99 Don't know 98 Choose not to answer How did you first become aware of HESN? Please specify other response. b_first_aware.Contains(96) TEXT b_first_aware_other What role(s) did you play under HESN? Please select all that apply. $familiar (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98)) ||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4,96 )) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT b_role 01 Principal investigator 02 Researcher supporting principal investigator 03 Research fellow 04 Researcher at a local university in a developing country 96 Other (please specify) 99 Don't know 98 Choose not to answer What role(s) did you play under HESN? Please specify other response. b_role.Contains(96) TEXT b_role_other C. RESEARCH EXPERIENCE $familiar E.84 | PERFORMANCE EVALUATION OF HESN USAID.GOV C. RESEARCH EXPERIENCE 5 / 28 STATIC TEXT Now, we would like to ask you about your individual research experience. How many years have you been working with your current university? self < 70 && self>=0 This number does not seemcorrect, please double check. NUMERIC: INTEGER c_uni_yrs What is your status as a researcher at your university? SINGLE-SELECT c_uni_status 01 Senior/full professor 02 Junior/associate/assistant professor 03 Professor/university lecturer (non-tenure track, non- adjunct) 04 Visiting or adjunct faculty 05 Research scientist 06 Post-doctoral student 07 PhD student 08 Master's student 09 College dean/department chair/ college rector/ or equivalent 11 Extension agent/faculty 96 Other (please specify) 99 Don't know 98 Choose not to answer What is your status as a researcher at your university? Please specify other response. c_uni_status==96 TEXT c_uni_status_other What are your primary research sectors? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT c_primary_sec 01 Agriculture, food distribution, food security 02 Democracy, human rights, and governance 03 Economic growth and trade (including infrastructure) 04 Education 05 Environment and climate change 06 Gender and women’s empowerment 07 Health and nutrition 08 Water and sanitation 09 Crisis and conflict 96 Other (please specify) 99 Don't know 98 Choose not to answer E.85 | PERFORMANCE EVALUATION OF HESN USAID.GOV C. RESEARCH EXPERIENCE 6 / 28 What are your primary research sectors? Please specify other response. c_primary_sec.Contains(96) TEXT c_primary_sec_other How many years have you worked in international development research? self < 70 && self>=0 This number does not seemcorrect, please double check. NUMERIC: INTEGER c_res_dev_yrs What is your total number of published peer- reviewed journal articles? self < 2000 && self>=0 This number does not seemcorrect, please double check. NUMERIC: INTEGER c_pub_num Did you collaborate with any of the following entities on a research project prior to your participation in the HESN? Please select all that apply. (self.ContainsAny(1,2,3,4,5)&&!self.ContainsAny(99,98,6 ))||(self.ContainsAny(99,98,6)&&!self.ContainsAny(1,2,3, 4,5)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer"/"did not collaborate with any of these entiti es". MULTI-SELECT c_prior_collab_ent 01 USAID Operating Unit (ex. Bureau for Global Health, Bureau for Food Security) 02 USAID Mission 03 HESN Global Development Lab university 04 Other international development aid agency 05 Policymakers in developing countries 06 Did not collaborate with any of these entities 99 Don't know 98 Choose not to answer What positions did you hold while conducting research under HESN? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,7,8,9,10,11,96)&&!self.Con tainsAny(99,98 ))||(self.ContainsAny(99,98)&&!self.Conta insAny(1,2,3,4,5,6,7,8,9,10,11,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT c_pos 01 Senior/full professor 02 Junior/associate/assistant professor 03 Professor/university lecturer (non-tenure track, non- adjunct) 04 Visiting or adjunct faculty 05 Research scientist 06 Post-doctoral student 07 PhD student 08 Master's student 09 College dean/department chair/ college rector/ or equivalent 11 Extension agent/faculty 96 Other (please specify) 99 Don't know 98 Choose not to answer What positions did you hold while conducting research under HESN? Please specify other response. c_pos.Contains(96) TEXT c_pos_other E.86 | PERFORMANCE EVALUATION OF HESN USAID.GOV C. RESEARCH EXPERIENCE 7 / 28 STATIC TEXT Please click on the blue box to proceed to the next section. What sectors of research were you involved with during your participation with HESN? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT c_dur_sec 01 Agriculture, food distribution, food security 02 Democracy, human rights, and governance 03 Economic growth and trade (including infrastructure) 04 Education 05 Environment and climate change 06 Gender and women’s empowerment 07 Health and nutrition 08 Water and sanitation 09 Crisis and conflict 96 Other (please specify) 99 Don't know 98 Choose not to answer What sectors of research were you involved with during your participation with HESN? Please specify other response. c_dur_sec.Contains(96) TEXT c_dur_sec_other E.87 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. HESN PROJECT INFORMATION 8 / 28 D. HESN PROJECT INFORMATION $familiar STATIC TEXT Next, we would like to ask you several questions about your participation in specific HESN projects. Please see the below definitions: HESN Lab: The university-based umbrella partnership; for example, AidData at the College of William and Mary, the Development Impact Lab (DIL) at the University of California Berkeley, or the Center on Conflict and Development (ConDev) at Texas A&M. HESN project: Specific work undertaken by an HESN university Lab such as conducting a study, undertaking a country- level mapping/geo-referencing activity, holding an innovation competition, or hosting a fellowship program. Participation in an HESN project: This is meant to be viewed broadly and could include a range of ways of engaging such as providing technical advice about a project scope or its implementation, serving as a buy-in point of contact, collecting data, monitoring project progress, activity management, review of deliverables, communication about the project, review of projectmaterials to informnew programming etc. What was the primary reason you chose to work on research projects under HESN? Please select your primary reason. SINGLE-SELECT d_work_reason 01 Supporting the work of USAID Operating Units and Missions 02 Advancing international development research at your university 03 Gaining research experience 96 Other (please specify) 99 Don't know 98 Choose not to answer What was the primary reason you chose to work on research projects under HESN? Please specify other response. d_work_reason==96 TEXT d_work_reason_ot How many HESN projects (individual research activities/studies) have you been participated in since 2015? SINGLE-SELECT d_number_projects 01 1 to 2 projects 02 3 to 5 projects 03 6 to 10 projects 04 More than 10 projects 95 Did not participate in any individual projects 99 Don't know 98 Choose not to answer E.88 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. HESN PROJECT INFORMATION 9 / 28 STATIC TEXT $projects Think about these projects (individual research activities/studies) that you considered to best represent HESN's goals. With these projects in mind, please answer the questions that follow: Of the HESN projects (individual research activities/studies) that you participated in since 2015, which projects (please list at least ONE project) best represent the HESN goals of: 1. Accelerating the use of innovative technologies and approaches to address global development challenges 2. Catalyzing an interdisciplinary research environment 3. Supporting evidence-based development decision making Please press enter to list each project name in a separate text box. $projects LIST d_main_project Which HESN Lab(s) were/are these projects (individual research activities/studies) linked to? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,7,8)&&!self.ContainsAny(99 ,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2, 3,4,5,6,7,8)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_lab 01 ResilientAfrica Network (RAN) 02 UC Berkeley – Development Impact Lab (DIL) 03 MIT – International Development Innovation Network (IDIN) 04 MIT – Comprehensive Initiative on Technology Evaluation (CITE) 05 MSU – Global Center for Food Systems Innovation (GCFSI) 06 Texas A&M – Center on Conflict and Development (ConDev) 07 Duke – Social Entrepreneurship Accelerator at Duke (SEAD) 08 William and Mary – AidData 99 Don't know 98 Choose not to answer What specific role(s) did/do you have under these projects? Please select all that apply. $projects (self.ContainsAny(1,2,3,96)&&!self.ContainsAny(99,98 ))| |(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_role 01 Principal investigator 02 Researcher supporting principal investigator 03 Research fellow 96 Other (please specify) 99 Don't know 98 Choose not to answer What specific role(s) did/do you have under these projects? Please specify other response. d_project_role.Contains(96) TEXT d_project_role_other E.89 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. HESN PROJECT INFORMATION 10 / 28 What are the funding sources for these projects? Please select all that apply. $projects (self.ContainsAny(1,2,3,96)&&!self.ContainsAny(99,98 ))| |(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_fund 01 HESN (through Global Development Lab) 02 USAID Mission 03 External USAID Operating Unit (outside the Global Development Lab) 96 Other (please specify) 99 Don't know 98 Choose not to answer What were/are the funding sources for these projects? Please specify other response. d_project_fund.Contains(96) TEXT d_project_fund_other What sector(s) did/do these projects cover? MULTI-SELECT d_project_sector Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". 01 Agriculture, food distribution, food security 02 Democracy, human rights, and governance 03 Economic growth and trade (including infrastructure) 04 Education 05 Environment and climate change 06 Gender and women’s empowerment 07 Health and nutrition 08 Water and sanitation 09 Crisis and conflict 96 Other (please specify) 99 Don't know 98 Choose not to answer What sector(s) did/do these projects cover? Please specify other response. d_project_sector.Contains(96) TEXT d_project_sector_other In what country/countries were/are these projects being implemented? Ifmore than one country, please press enter to list each country in a s eparate text box. $projects LIST d_project_loc E.90 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. HESN PROJECT INFORMATION 11 / 28 STATIC TEXT Please click on the blue box to proceed to the next section. What were/are the main objectives of these projects? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,7,8,96)&&!self.ContainsAny (99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1 ,2,3,4,5,6,7,8,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_object 01 Inputs for country development strategy or program design 02 Collect data (program monitoring data, demographic, geographic, sectoral, etc.) 03 Impact evaluation of a program 04 Performance evaluation of a program 05 Fill gaps in data 06 Capacity building of USAID Mission/Operating Unit 07 Capacity building of partners 08 Policy development 96 Other (please specify) 99 Don't know 98 Choose not to answer What were/are the main objectives of these projects? Please specify other response. d_project_object.Contains(96) TEXT d_project_object_ot During the life of these projects, how much of your professional time did you spend supporting their activities, on average? $projects SINGLE-SELECT d_project_time_spent 01 Less than 1/4 of time 02 1/4 to 1/2 of time 03 More than 1/2, but less than 3/4 of time 04 3/4 to full time 99 Don't know 98 Choose not to answer E. DEPARTMENT-LEVEL EXPERIENCE WITH HESN 12 / 28 E. DEPARTMENT-LEVEL EXPERIENCE WITH HESN $familiar STATIC TEXT Based on your experience with HESN, please indicate how strongly you agree with each of the following statements. STATIC TEXT Please click on each of the sub-section boxes to view and respond to statements. Each sub-section box will turn green once completed. E. DEPARTMENT-LEVEL EXPERIENCE WITH HESN HESN HAS EXPANDED THE CAPACITY IN MY DEPARTMENT TO: STATIC TEXT HESN has expanded the capacity in my department to: Please indicate how strongly you agree with each of the following statements. Design research evaluations related to international development (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT e_design_eval 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Collect and analyze data related to international development (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT e_collect_data 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E. DEPARTMENT-LEVEL EXPERIENCE WITH HESN HESN HAS EXPANDED OPPORTUNITIES WITHIN MY DEPARTMENT FOR: STATIC TEXT HESN has expanded opportunities within my department for: Please indicate how strongly you agree with each of the following statements. Faculty/staff to collaborate in the design and implementation of international development research (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT e_collaborate_design 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E. DEPARTMENT-LEVEL EXPERIENCE WITH HESN 13 / 28 Students to work with faculty to conduct international development research (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT e_opportunity_students 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E. DEPARTMENT-LEVEL EXPERIENCE WITH HESN PARTICIPATING IN HESN LAB ACTIVITIES HAS: STATIC TEXT Participating in HESN Lab activities has: Please indicate how strongly you agree with each of the following statements. Provided opportunities for faculty to present research at the university. SINGLE-SELECT e_present_uni 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Provided opportunities for faculty to present research within the country (but outside the university). SINGLE-SELECT e_present_country 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Provided opportunities for faculty to present research abroad. SINGLE-SELECT e_present_abroad 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Created new career opportunties for students. SINGLE-SELECT e_created_opportunities 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.92 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. DEPARTMENT-LEVEL EXPERIENCE WITH HESN 14 / 28 Motivated students to pursue work in international development (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT e_motivated_students 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E. DEPARTMENT-LEVEL EXPERIENCE WITH HESN HESN HAS PROVIDED: STATIC TEXT HESN has provided: Please indicate how strongly you agree with each of the following statements. Additional financial resources to support research within my department. SINGLE-SELECT e_financial_resources 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Technical resources to support faculty in my department as they improve their research and evaluations. SINGLE-SELECT e_tech_resources 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Added value to my department and university. SINGLE-SELECT e_added_value 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Motivation for me to participate in another HESN project if given the opportunity. SINGLE-SELECT e_participate_opportunity 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. STATIC TEXT E. DEPARTMENT-LEVEL EXPERIENCE WITH HESN 15 / 28 Please click on the blue box to proceed to the next section. E.94 | PERFORMANCE EVALUATION OF HESN USAID.GOV E.95 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN USEFULNESS AND SUSTAINABILITY 16 / 28 F. HESN USEFULNESS AND SUSTAINABILITY $familiar STATIC TEXT $familiar Based on your experience with HESN, please indicate how strongly you agree with each of the following statements. STATIC TEXT Please click on each of the sub-section boxes to view and respond to statements. Each sub-section box will turn green once completed. F. HESN USEFULNESS AND SUSTAINABILITY THE HESN WORK PERFORMED WITHIN MY DEPARTMENT: STATIC TEXT The HESN work performed within my department: Please indicate how strongly you agree with each of the following statements. Produced useful research products for USAID Missions and other stakeholders. SINGLE-SELECT f_useful_research 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Was relevant to development (poverty alleviation, social innovation, public health, etc.) issues in the target country or countries. SINGLE-SELECT f_rel_dev_issues 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Was timely for the expansion of my department’s international development research. SINGLE-SELECT f_timely_expan 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Helped expand our department’s international network. SINGLE-SELECT f_expand_network 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.96 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN USEFULNESS AND SUSTAINABILITY 17 / 28 Helped broaden the research conducted within my department. SINGLE-SELECT f_broad_research 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Led to partnerships between my department, USAID Missions and other stakeholders that are beneficial to all parties. SINGLE-SELECT f_partner_ben 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. F. HESN USEFULNESS AND SUSTAINABILITY MY EXPERIENCE WORKING WITH HESN LABS: STATIC TEXT My experience working with HESN Labs: Please indicate how strongly you agree with each of the following statements. Has been positive. SINGLE-SELECT f_exp_pos 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has increased my interest in international development (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT f_inc_interest 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has helped me develop additional research or evaluation design skills. SINGLE-SELECT f_dev_skills 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know F. HESN USEFULNESS AND SUSTAINABILITY 18 / 28 Has provided me with opportunities to develop other research skills. SINGLE-SELECT f_opp_skill_grow 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has provided me with opportunities for career advancement. SINGLE-SELECT f_opp_career 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has led to opportunities for working as a research consultant. SINGLE-SELECT f_opp_consult 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has led to opportunities for research fellowships. SINGLE-SELECT f_opp_fellow 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has led to opportunities to travel to other countries and present my work. SINGLE-SELECT f_opp_travel 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has provided me with opportunities for dissemination of my work. SINGLE-SELECT f_opp_pub 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has provided me with opportunities to engage policymakers around my research findings. SINGLE-SELECT f_opp_policy 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know F. HESN USEFULNESS AND SUSTAINABILITY 19 / 28 STATIC TEXT Please click on the box below to continue. F. HESN USEFULNESS AND SUSTAINABILITY OPPORTUNITIES FOR INTERNATIONAL DEVELOPMENT RESEARCH (POVERTY ALLEVIATION, SOCIAL INNOVATION, PUBLIC HEALTH, ETC.): STATIC TEXT Opportunities for international development research (poverty alleviation, social innovation, public health, etc.): Please indicate how strongly you agree with each of the following statements. Became more of a focus for my department due to HESN. SINGLE-SELECT f_dept_int_foc 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Are plentiful within my department. SINGLE-SELECT f_uni_dev_opp 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Will grow within my department in the next five years. SINGLE-SELECT f_uni_dev_opp_grow 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. STATIC TEXT Please click on the blue box to proceed to the next section. E.98 | PERFORMANCE EVALUATION OF HESN USAID.GOV E.99 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 20 / 28 G. HESN RESEARCH OUTPUTS AND DISSEMINATION $familiar && $projects STATIC TEXT Now we would like to ask you a few questions about the dissemination of results/research outputs from HESN project (individual research activities/studies). Once again, think about the HESN projects (individual research activities/studies) that you considered to best represent HESN's goals. With these projects inmind, please answer the questions that follow: What research results/outputs were/will be produced as part of these projects? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,7,96)&&!self.ContainsAny(9 9,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2 ,3,4,5,6,7,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_research 01 Report or publication 02 Policy brief 03 Toolkit 04 Data set/platform/mapping 05 Evaluation 06 Training 07 Event or conference 96 Other (please specify) 99 Don't know 98 Choose not to answer What research results/outputs were/will be produced as part of these projects? Please specify other response. g_project_research.Contains(96) TEXT g_project_research_other Did these projects already produce research SINGLE-SELECT g_project_produced results/outputs for sharing? 01 Yes 00 No 99 Don't know What were the research results/outputs under these projects used for? Please select all that apply. $produced (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98,5 ,6 ))||(self.ContainsAny(99,98,5,6)&&!self.ContainsAny(1 ,2,3,4,96)) Your answers cannot include both substantive answers and "not used yet butwill be used in the future"/"not used for anything"/"don't know"/ "choose not to answer". MULTI-SELECT g_project_use 01 Inputs for country development strategy or program design 02 Capacity building of USAID Mission/Operating Unit 03 Capacity building of government in target country 04 Policy development 05 Not used yet but will be used in the future 06 Not used for anything 96 Other specified use 99 Don't know 98 Choose not to answer What were the research results/outputs under these projects used for? Please specify other response. g_project_use.Contains(96) TEXT g_project_use_other E.100 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 21 / 28 Why were the research results/outputs produced under these projects not used for anything? g_project_use.Contains(6) TEXT g_project_no_use When do you expect that research results/outputs produced under these projects will be used? g_project_use.Contains(5) TEXT g_project_use_when What will the research results/outputs under these projects be used for? Please select all that apply. g_project_use.Contains(5) (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98 ) )||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4,9 6)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_exp_use 01 Inputs for country development strategy or program design 02 Capacity building of Mission/Operating unit 03 Capacity building of government in target country 04 Policy development 96 Other (please specify) 99 Don't know 98 Choose not to answer What will the research results/outputs under these projects be used for? Please specify other response. g_project_exp_use.Contains(96) TEXT g_project_use_exp_ot How did you disseminate research results/outputs from these projects? Please select all that apply. $produced (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98,10,11 ))||(self.ContainsAny(99,98,10,11)&&!self .ContainsAny(1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "Not disse minated yet but will be in the future"/"not disseminated"/"don't know"/ "choose not to answer". MULTI-SELECT g_project_dissem 01 Email 02 Social media 03 Podcast 04 Blog 05 Policy/issue brief 06 Virtual results presentation 07 In-person results presentation 08 Dissemination workshop 09 In the middle of dissemination 10 Not disseminated yet but will be in the future 11 Not disseminated 96 Other (please specify) 99 Don't know 98 Choose not to answer How did you disseminate research results/outputs from these projects? Please specify other response. g_project_dissem.Contains(96) TEXT g_project_dissem_ot When do you expect that research results/outputs produced under these projects will be disseminated? g_project_dissem.Contains(10) TEXT g_project_dissem_when E.101 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 22 / 28 G. HESN RESEARCH OUTPUTS AND DISSEMINATION Roster: AUDIENCE generated bymulti-select question g_project_dissem_aud g_audience g_project_dissem_aud.ContainsAny(1,2,3,4,5,6,7,96) Why were the research results/outputs produced under these projects not disseminated? g_project_dissem.Contains(11) TEXT g_project_no_dissem To what audience did you directly disseminate research results/outputs from these projects? Please select all that apply. g_project_dissem.ContainsAny(1,2,3,4,5,6,7,8,9,96) (self.ContainsAny(1,2,3,4,5,6,7,96)&&!self.ContainsAny(9 9,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2 ,3,4,5,6,7,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_dissem_aud 01 USAID Mission(s) 02 External USAID Operating Unit(s) (outside the Global Development Lab) 03 Policymakers in the country where research took place 04 Researchers in the country where research took place 05 Other stakeholder organizations where the research took place 06 Policymakers in other countries 07 Researchers in other countries 96 Other audience 99 Don't know 98 Choose not to answer To what audience did you directly disseminate research results/outputs from these projects? Please specify other audience. g_project_dissem_aud.Contains(96) TEXT g_project_dissem_aud_ot How many times did you speak with the %g_audience% about research results/outputs fromthese projects? self < 500 && self>=0 This number does not seemcorrect, please double check. NUMERIC: INTEGER g_aud_spoke_num Do you know the position of key representatives from the %g_audience% with whom you spoke to about results/outputs fromthese projects? SINGLE-SELECT g_aud_spoke_know 01 Yes 02 No Who were the key representatives (organization and position) from the %g_audience% with whom you spoke about research results/outputs from these projects? For each key person please just list their position and organization. g_aud_spoke_know==1 LIST g_aud_spoke_ppl E.102 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 23 / 28 Please indicate your level of agreement with: the %g_audience% showed strong interest in research results from these projects? SINGLE-SELECT g_aud_interest 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Doesn't know Please indicate your level of agreement with: the %g_audience% indicated that research results/outputs from these projects were useful to their planning or programs? SINGLE-SELECT g_aud_use 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Doesn't know What facilitated the use of research results/outputs produced under these projects? Please select all that apply. $produced (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_facil 01 Timeline for completing the product 02 Daily implementation of activities 03 Support from policymakers due to relevance of topic 04 Geography in-country facilitates dissemination (i.e. easy access to internet, regional distribution point, easy access to urban and rural areas) 05 Local interest/desire for research product 06 Having financial resources to create products 07 Having financial resources to disseminate products 08 Having financial or in-kind incentives for partners 09 Expertise and reputation of HESN Lab 96 Other (please specify) 99 Don't know 98 Choose not to answer What facilitated the use of research results/outputs produced under these projects? Please specify other response. g_project_facil.Contains(96) TEXT g_project_facil_ot Why do you think the items you selected above facilitated the use of results/research outputs fromthese projects? g_project_facil.ContainsAny(1,2,3,4,5,6,7,8,9,96) TEXT g_project_facil_why E.103 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 24 / 28 What inhibited the use of research results/outputs produced under these projects? Please select all that apply. $produced (self.ContainsAny(1,2,3,4,5,6,7,8,9,10,11,96)&&!self.Con tainsAny(99,98 ))||(self.ContainsAny(99,98)&&!self.Conta insAny(1,2,3,4,5,6,7,8,9,10,11,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_inhib 01 Contractual requirements 02 Reporting requirements 03 Delays in completing research products 04 Lack of support from policymakers due to irrelevance of topic 05 Geography in-country inhibits dissemination (i.e. lack of consistent internet, no regional distribution points, difficult access to urban and rural areas) 06 Lack of local interest/desire for research product 07 Lack of financial resources to create products 08 Lack of financial resources to disseminate product 09 Lack of financial or in-kind incentives for partners 10 Limits on ability to publish/copyright issues 11 Political pressure 96 Other (please specify) 99 Don't know 98 Choose not to answer What inhibited the use of research results/outputs produced under the these projects? Please specify other response. g_project_inhib.Contains(96) TEXT g_project_inhib_ot Why do you think the items you selected above inhibited the use of results/research outputs fromthese projects? g_project_inhib.ContainsAny(1,2,3,4,5,6,7,8,9,10,11,96) TEXT g_project_inhib_why Did any additional collaborations result beyond the original scope of these projects? SINGLE-SELECT g_project_additional 01 Yes 00 No 99 Don't know E.104 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 25 / 28 STATIC TEXT Please click on the blue box to proceed to the next section. What are the main objectives of the additional collaborations beyond these projects? Please select all that apply. $additional (self.ContainsAny(1,2,3,4,5,6,7,8,96)&&!self.ContainsAny (99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1 ,2,3,4,5,6,7,8,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_add_obj 01 Inputs for country development strategy or program design 02 Collect data (program monitoring data, demographic, geographic, sectoral, etc.) 03 Impact evaluation of a program 04 Performance evaluation of program 05 Fill gaps in data 06 Capacity building of USAID Mission/Operating Unit 07 Capacity building of partners 08 Policy development 96 Other (please specify) 99 Don't know 98 Choose not to answer What are the main objectives of the additional collaborations beyond these projects? Please specify other response. g_project_add_obj.Contains(96) TEXT g_project_add_obj_ot What are/will be the funding sources for the additional collaborations beyond these projects? Please select all that apply. $additional (self.ContainsAny(1,2,3,96)&&!self.ContainsAny(99,98 ))| |(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_add_fund 01 Global Development Lab 02 USAID Mission 03 External USAID Operating Unit (outside the Global Development Lab) 96 Other (please specify) 99 Don't know 98 Choose not to answer What are/will be the funding sources for the additional collaborations beyond these projects? Please specify other response. g_project_add_fund.Contains(96) TEXT g_project_add_fund_ot In what country/countries will the additional collaborations beyond these projects be implemeneted? Ifmore than one country, please press enter to list each country in a s eparate text box. $additional LIST g_project_add_loc H. RECOMMENDATIONS $familiar E.105 | PERFORMANCE EVALUATION OF HESN USAID.GOV H. RECOMMENDATIONS 26 / 28 STATIC TEXT Please click on the blue box to proceed to the next section. Based on your experience with HESN, what are key areas to help improve partner engagement and the use of research outputs by partners? Please select all that apply. (self.ContainsAny(1,2,3,4,5,96)&&!self.ContainsAny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4, 5,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT h_rec_uptake 01 Greater support from HESN Lab's Agreement Officer's Representative (AOR) 02 Allow more direct contact with partners 03 Make research content less technical 04 More dissemination activities 05 Better alignment with key stakeholder interests 96 Other (please specify) 99 Don't know 98 Choose not to answer Based on your experience with HESN, what are key areas to help improve partner engagement and the use of research outputs by partners? Please specify other response. h_rec_uptake.Contains(96) TEXT h_rec_uptake_other Do you have any other comments or suggestions related to your experience with HESN? SINGLE-SELECT h_has_comment 01 Yes (please specify) 00 No Do you have any other comments or suggestions related to your experience with HESN? Please specify any other comments. h_has_comment==1 TEXT h_comment_specify I. FOLLOW-UP $familiar E.106 | PERFORMANCE EVALUATION OF HESN USAID.GOV I. FOLLOW-UP 27 / 28 In order to learn more about the utility of the HESN mechanism and research outputs, we would like to conduct short in-person or phone interviews with a subsample of online survey respondents. Would you be willing to participate in a short follow-up interview? SINGLE-SELECT i_follow_part 01 Yes 00 No STATIC TEXT $follow Please provide the below information: STATIC TEXT Thank you for your time! Please click on the box below to record the survey start time. DATE: CURRENT TIME i_end_time STATIC TEXT Please click on the blue box to proceed to the next section. First name: $follow TEXT i_first_name Last name: $follow TEXT i_last_name Email: $follow TEXT i_email Phone number (with country code): $follow TEXT i_phone Current organization: $follow TEXT i_organ Current position: $follow TEXT i_position E.107 | PERFORMANCE EVALUATION OF HESN USAID.GOV LEGEND 28 / 28 LEGEND Legend and structure of information in this file This page has been left blank for double-sided copying. Generated by edithgfelix, Apr 06, 2020 12:31 Questionnaire created by edithgfelix, Dec 15, 2019 16:13 Last modified by edithgfelix, Dec 17, 2019 16:58 Shared with: jmeuthalldredge (never edited) Sections: 9, Sub-sections: 7, Questions: 103. Questions with enabling conditions: 48 Questions with validation conditions:20 Rosters: 1 Variables: 0 HESN Innovator MIT IDIN Survey SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION A. INTRODUCTION No sub-sections, No rosters,Questions: 4, Static texts: 2. B. GENERAL INFORMATION No sub-sections, No rosters,Questions: 5, Static texts: 2. C. RESEARCH/WORK EXPERIENCE No sub-sections, No rosters,Questions: 6, Static texts: 2. D. HESN INNOVATION GRANTS No sub-sections, No rosters,Questions: 16, Static texts: 3. E. EXPERIENCE WITH HESN Sub-sections: 4,No rosters,Questions: 15, Static texts: 11. F. HESN USEFULNESS AND SUSTAINABILITY Sub-sections: 3,No rosters,Questions: 17, Static texts: 9. G. HESN RESEARCH OUTPUTS AND DISSEMINATION No sub-sections,Rosters: 1,Questions: 28, Static texts: 2. H. RECOMMENDATIONS No sub-sections, No rosters,Questions: 4, Static texts: 1. I. FOLLOW-UP No sub-sections, No rosters,Questions: 8, Static texts: 3. LEGEND E.109 | PERFORMANCE EVALUATION OF HESN USAID.GOV 1 / 25 E.110 | PERFORMANCE EVALUATION OF HESN USAID.GOV SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION 2 / 25 SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION Basic information Title HESN Innovator MIT IDIN Survey E.111 | PERFORMANCE EVALUATION OF HESN USAID.GOV A. INTRODUCTION 3 / 25 A. INTRODUCTION STATIC TEXT ● This survey gathers information about your perspectives and experience with the Higher Education Solutions Network (HESN)/the Massachusetts Institute of Technology (MIT) International Development Innovation Network (IDIN). ● If you have participated in another USAID Global Development Lab program such as LASER, RTAC, STIP APS, or PEER, please only provide information related to your perspectives and experience with HESN. ● USAID’s Global Development Lab has contracted Mathematica to conduct an independent evaluation of the long-term impacts of HESN. ● USAID is interested in understanding the ultimate impact/utility of HESN projects implemented since 2015 and the types of partnerships thatmay produce policy impact. ● This survey should take approximately 30 minutes to complete. ● Any information you provide that can identify you will be kept strictly confidential. Information provided will be used for research purposes only. Your participation is voluntary and you may choose not to answer any or all questions for any reason. ● You may contact Audrey-Marie Moore, the Mathematica Senior Researcher leading this evaluation at AMoore@mathematica-mpr.com, if you have study questions, concerns or complaints. Thank you for your time and help with this evaluation! Please click on the box below to record the survey start time. DATE: CURRENT TIME a_start_time Do you consent to participate in this survey? SINGLE-SELECT a_consent 01 Yes 00 No Name TEXT prefilled_name SCOPE: IDENTIFYING Email TEXT prefilled_email SCOPE: IDENTIFYING STATIC TEXT Please click on the box to proceed to the next section. B. GENERAL INFORMATION a_consent==1 E.112 | PERFORMANCE EVALUATION OF HESN USAID.GOV B. GENERAL INFORMATION 4 / 25 STATIC TEXT To begin, we would like to ask you some general information about HESN/the Massachusetts Institute of Technology (MIT) International Development Innovation Network (IDIN) STATIC TEXT Please click on the blue box to proceed to the next section. Are you familiar with the the Higher Education Solutions Network (HESN)/the Massachusetts Institute of Technology (MIT) International Development Innovation Network (IDIN)? SINGLE-SELECT b_familiar 01 Yes 00 No How did you first become aware of HESN/MIT IDIN? Please select all that apply. $familiar (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98)) ||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT b_first_aware 01 Other community members 02 A nongovernmental organization 03 A social venture/enterprise 04 A local university 05 MIT International Development Innovation Network (IDIN) member 06 MIT IDIN event 07 MIT IDIN communications 08 MIT IDIN innovation output 09 USAID communications 96 Other (please specify) 99 Don't know 98 Choose not to answer How did you first become aware of HESN/MIT IDIN? Please specify other response. b_first_aware.Contains(96) TEXT b_first_aware_other What role(s) did you play under HESN/MIT IDIN? Please select all that apply. $familiar (self.ContainsAny(1,2,3,4,5,6,96)&&!self.ContainsAny(99, 98)) ||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3 ,4,5,6,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT b_role 01 Innovator 02 Scale-up fellow 03 Microgrant recipient 04 Picogrant recipient 05 Leader of an IDIN innovation center 06 Leader of an IDIN local chapter 96 Other (please specify) 99 Don't know 98 Choose not to answer What role(s) did you play under HESN/MIT IDIN? Please specify other response. b_role.Contains(96) TEXT b_role_other C. RESEARCH/WORK EXPERIENCE $familiar E.113 | PERFORMANCE EVALUATION OF HESN USAID.GOV C. RESEARCH/WORK EXPERIENCE 5 / 25 STATIC TEXT Now, we would like to ask you about your individual research/work sector experience. What are your primary research or work sectors? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) MULTI-SELECT c_primary_sec 01 Agriculture, food distribution, food security 02 Democracy, human rights, and governance 03 Economic growth and trade (including infrastructure) 04 Education 05 Environment and climate change 06 Gender and women’s empowerment 07 Health and nutrition 08 Water and sanitation 09 Crisis and conflict 96 Other (please specify) 99 Don't know 98 Choose not to answer What is your primary research or work sector? Please specify other response. c_primary_sec.Contains(96) TEXT c_primary_sec_other How many years have you worked in international development (poverty alleviation, social innovation, public health, etc.)? self < 80 && self>=0 This number does not seemcorrect, please double check. NUMERIC: INTEGER c_res_dev_yrs Did you collaborate with any of the following entities on a research or innovation project prior to your participation in the HESN? Please select all that apply. (self.ContainsAny(1,2,3,4,5)&&!self.ContainsAny(99,98,6 ))||(self.ContainsAny(99,98,6)&&!self.ContainsAny(1,2,3, 4,5)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer"/"did not collaborate with any of these entiti es". MULTI-SELECT c_prior_collab_ent 01 USAID Operating Unit (ex. Bureau for Global Health, Bureau for Food Security) 02 USAID Mission 03 HESN Global Development Lab university 04 Other international development aid agency 05 Policymakers in developing countries 06 Did not collaborate with any of these entities 99 Don't know 98 Choose not to answer E.114 | PERFORMANCE EVALUATION OF HESN USAID.GOV C. RESEARCH/WORK EXPERIENCE 6 / 25 STATIC TEXT Please click on the blue box to proceed to the next section. What sectors of research/innovation were you involved with during your participation with HESN/MIT IDIN? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT c_dur_sec 01 Agriculture, food distribution, food security 02 Democracy, human rights, and governance 03 Economic growth and trade (including infrastructure) 04 Education 05 Environment and climate change 06 Gender and women’s empowerment 07 Health and nutrition 08 Water and sanitation 09 Crisis and conflict 96 Other (please specify) 99 Don't know 98 Choose not to answer What sectors of research/innovation were you involved with during your participation with HESN/MIT IDIN? Please specify other response. c_dur_sec.Contains(96) TEXT c_dur_sec_other D. HESN INNOVATION GRANTS 7 / 25 D. HESN INNOVATION GRANTS $familiar STATIC TEXT Next, we would like to ask you several questions about your participation in specific HESN/MIT IDIN innovation grants. STATIC TEXT $projects Think about these grants that you considered to best represent HESN's goals. With these grants in mind, please answer Did you apply for an innovation grant under HESN/MIT IDIN? SINGLE-SELECT d_apply_grant 01 Yes 00 No What was the primary reason you chose to apply for an innovation grant under HESN/MIT IDIN? Please select your primary reason. d_apply_grant==1 SINGLE-SELECT d_apply_reason 01 Advancing the use of innovative techniques/products in international development 02 Supporting the work of USAID Operating Units and Missions 03 Gaining research/innovation development experience 96 Other (please specify) 99 Don't know 98 Choose not to answer What was the primary reason you chose to apply for an innovation grant under HESN/MIT IDIN? Please specify other response. d_apply_reason==96 TEXT d_apply_reason_ot How many HESN/MIT IDIN innovation grants have you been involved with since 2015? SINGLE-SELECT d_number_projects 01 1 to 2 grants 02 3 to 5 grants 03 6 to 10 grants 04 More than 10 grants 95 Did not participate in any individual grants 99 Don't know 98 Choose not to answer Of the HESN/MIT IDIN grants that you participated in since 2015, which grants (please list at least ONE grant) best represent the HESN goals of: 1. Accelerating the use of innovative technologies and approaches to address global development challenges 2. Catalyzing an interdisciplinary research environment 3. Supporting evidence-based development decision making Please press enter to list each grant name in a separate text box. $projects LIST d_main_project E.116 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. HESN INNOVATION GRANTS 8 / 25 the questions that follow: What HESN Lab(s) were the grants part of? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,7,8)&&!self.ContainsAny(99 ,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2, 3,4,5,6,7,8)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_lab 01 ResilientAfrica Network (RAN) 02 UC Berkeley – Development Impact Lab (DIL) 03 MIT – International Development Innovation Network (IDIN) 04 MIT – Comprehensive Initiative on Technology Evaluation (CITE) 05 MSU – Global Center for Food Systems Innovation (GCFSI) 06 Texas A&M – Center of Conflict and Development (ConDev) 07 Duke – Social Entrepreneurship Accelerator at Duke (SEAD) 08 William and Mary – AidData 99 Don't know 98 Choose not to answer What specific role(s) did/do you have under these grants? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,96)&&!self.ContainsAny(99, 98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3 ,4,5,6,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_role 01 Innovator 02 Scale-up fellow 03 Microgrant recipient 04 Picogrant recipient 05 Leader of an IDIN innovation center 06 Leader of an IDIN local chapter 96 Other (please specify) 99 Don't know 98 Choose not to answer What specific role(s) did/do you have under these grants? Please specify other response. d_project_role.Contains(96) TEXT d_project_role_other What were/are the funding sources for these grants? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98 ) )||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4,9 6)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_fund 01 HESN (through Global Development Lab)/MIT 02 USAID 03 Private sector 04 Local government 96 Other (please specify) 99 Don't know 98 Choose not to answer What were/are the funding sources for these grants? Please specify other response. d_project_fund.Contains(96) TEXT d_project_fund_other E.117 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. HESN INNOVATION GRANTS 9 / 25 STATIC TEXT Please click on the blue box to proceed to the next section. What sector(s) did/do these grants cover? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_sector 01 Agriculture, food distribution, food security 02 Democracy, human rights, and governance 03 Economic growth and trade (including infrastructure) 04 Education 05 Environment and climate change 06 Gender and women’s empowerment 07 Health and nutrition 08 Water and sanitation 09 Crisis and conflict 96 Other (please specify) 99 Don't know 98 Choose not to answer What sector(s) did/do these grants cover? Please specify other response. d_project_sector.Contains(96) TEXT d_project_sector_other In what country/countries were/are these grants being implemented? Ifmore than one country, please press enter to list each country in a s eparate text box. $projects LIST d_project_loc What were/are the main objectives of these grants? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,96)&&!self.ContainsAny(99, 98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3 ,4,5,6,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_object 01 Prototyping 02 Fabrication 03 Pilot testing 04 Early stage market assessment 05 Dissemination 06 Scale-up 96 Other (please specify) 99 Don't know 98 Choose not to answer What were/are the main objectives of these grants? Please specify other response. d_project_object.Contains(96) TEXT d_project_object_ot During the life of these grants, how much time did you spend supporting the activity, on average? $projects SINGLE-SELECT d_project_time_spent 01 Less than 1/4 of time 02 1/4 to 1/2 of time 03 More than 1/2, but less than 3/4 of time 04 3/4 to full time 99 Don't know 98 Choose not to answer E. EXPERIENCE WITH HESN 10 / 25 E. EXPERIENCE WITH HESN $familiar STATIC TEXT Based on your experience with HESN/MIT IDIN, please indicate how strongly you agree with each of the following statements. STATIC TEXT Please click on each of the sub-section boxes to view and respond to statements. Each sub-section box will turn green once completed. E. EXPERIENCE WITH HESN HESN/MIT IDIN CONTRIBUTED TO: STATIC TEXT HESN/MIT IDIN contributed to: Please indicate how strongly you agree with each of the following statements. Design of innovative technology/techniques related to international development (poverty alleviation, social innovation, public health, etc.) issues. SINGLE-SELECT e_design_eval 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Collecting and analyzing of data related to international development (poverty alleviation, social innovation, public health, etc.) issues. SINGLE-SELECT e_collect_data 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E. EXPERIENCE WITH HESN HESN/MIT IDIN INNOVATION GRANTS EXPANDED OPPORTUNITIES FOR THE FOLLOWING TO PARTICIPATE IN THE DESIGN AND IMPLEMENTATION OF INNOVATIONS: STATIC TEXT HESN/MIT IDIN innovation grants expanded opportunities for the following to participate in the design and implementation of innovations: Please indicate how strongly you agree with each of the following statements. E.118 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. EXPERIENCE WITH HESN 11 / 25 Local community members SINGLE-SELECT e_cmty_members 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Local researchers SINGLE-SELECT e_loc_researchers 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Local students SINGLE-SELECT e_loc_students 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Innovators from other developing countries SINGLE-SELECT e_other_innov 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know U.S. researchers SINGLE-SELECT e_us_researchers 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know U.S. students SINGLE-SELECT e_us_students 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E. EXPERIENCE WITH HESN PARTICIPATING IN HESN/MIT IDIN SUPPORTED GRANTS HAS: STATIC TEXT E.120 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. EXPERIENCE WITH HESN 12 / 25 Participating in HESN/MIT IDIN supported grants has: Please indicate how strongly you agree with each of the following statements. Provided opportunities for people to present innovative tools/technology/techniques to a broader audience. SINGLE-SELECT e_present_innov 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Provided opportunities for innovators like myself to engage with universities outside of the country. SINGLE-SELECT e_engage_outside 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Created new career opportunities for people like myself who focus on using innovative techniques. SINGLE-SELECT e_created_opportunities 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Motivated people to pursue work in international development (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT e_motivated_people 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E. EXPERIENCE WITH HESN HESN/MIT IDIN HAS PROVIDED: STATIC TEXT HESN/MIT IDIN has provided: Please indicate how strongly you agree with each of the following statements. Additional financial resources to support innovation through new donors and partners. SINGLE-SELECT e_financial_resources 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.121 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. EXPERIENCE WITH HESN 13 / 25 Technical resources to support innovation. SINGLE-SELECT e_tech_resources 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Motivation for me to participate in another HESN/MIT IDIN innovation project/activity if given the opportunity. SINGLE-SELECT e_participate_opportunity 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. STATIC TEXT Please click on the blue box to proceed to the next section. E.122 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN USEFULNESS AND SUSTAINABILITY 14 / 25 F. HESN USEFULNESS AND SUSTAINABILITY $familiar STATIC TEXT $familiar Based on your experience with HESN/MIT IDIN, please indicate how strongly you agree with each of the following statements. STATIC TEXT Please click on each of the sub-section boxes to view and respond to statements. Each sub-section box will turn green once completed. F. HESN USEFULNESS AND SUSTAINABILITY THE WORK PERFORMED UNDER THE HESN/MIT IDIN INNOVATION GRANTS: STATIC TEXT The work performed under the HESN/MIT IDIN innovation grants: Please indicate how strongly you agree with each of the following statements. Produced useful research/innovation products for USAID Missions and other stakeholders. SINGLE-SELECT f_useful_research 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Was relevant to development (poverty alleviation, social innovation, public health, etc.) issues in the target country or countries. SINGLE-SELECT f_rel_dev_issues 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Came at a time when the enabling environment was ready for new techniques/tools/technology that help solve international development (poverty alleviation, social innovation, public health, etc.) challenges. SINGLE-SELECT f_timely_environ 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Helped expand my international network. SINGLE-SELECT f_expand_network 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.123 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN USEFULNESS AND SUSTAINABILITY 15 / 25 Helped broaden the use of innovative techniques in my sector. SINGLE-SELECT f_broad_innov 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. F. HESN USEFULNESS AND SUSTAINABILITY MY EXPERIENCE WORKING WITH THE HESN LAB/MIT IDIN: STATIC TEXT My experience working with the HESN Lab/MIT IDIN: Please indicate how strongly you agree with each of the following statements. Has been positive. SINGLE-SELECT f_exp_pos 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has increased my interest in international development (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT f_inc_interest 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has helped me develop additional innovation design skills. SINGLE-SELECT f_dev_skills 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has provided me with opportunities to develop other skills. SINGLE-SELECT f_opp_skill_grow 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.124 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN USEFULNESS AND SUSTAINABILITY 16 / 25 Has provided me with opportunities for career advancement. SINGLE-SELECT f_opp_career 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has led to opportunities for innovation fellowships or follow-on activities. SINGLE-SELECT f_opp_fellow 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has led to opportunities to travel to other countries and present my work. SINGLE-SELECT f_opp_travel 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has provided me with opportunities for dissemination of my work. SINGLE-SELECT f_opp_dissem 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has provided me with opportunities to engage policymakers around my research/innovation findings. SINGLE-SELECT f_opp_policy 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. F. HESN USEFULNESS AND SUSTAINABILITY OPPORTUNITES FOR: STATIC TEXT Opportunities for: Please indicate how strongly you agree with each of the following statements. E.125 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN USEFULNESS AND SUSTAINABILITY 17 / 25 The development and implementation of techniques in international development (poverty alleviation, social innovation, public health, etc.) are plentiful. SINGLE-SELECT f_innov_dev_opp 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know The use of innovative tools/techniques/technologies will grow in the next five years. SINGLE-SELECT f_innov_use_grow 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Continued funding for development of innovative tools/techniques/technologies are sufficient. SINGLE-SELECT f_innov_fund 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. STATIC TEXT Please click on the blue box to proceed to the next section. G. HESN RESEARCH OUTPUTS AND DISSEMINATION 18 / 25 G. HESN RESEARCH OUTPUTS AND DISSEMINATION $familiar && $projects STATIC TEXT Now we would like to ask you a few questions about the dissemination of results/research outputs from HESN/MIT IDIN grants. Once again, think about the HESN/MIT IDIN grants that you considered to best represent HESN's goals. With these grants inmind, please answer the questions that follow: What research/innovation outputs were/will be produced as part of these grants? Please select all that apply. (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98 ) )||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4,9 6)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_research 01 Innovation prototype 02 Pilot testing report 03 Market assessment report 04 Finished innovation 96 Other (please specify) 99 Don't know 98 Choose not to answer What research/innovation outputs were/will be produced as part of these grants? Please specify other response. g_project_research.Contains(96) TEXT g_project_research_other At what innovation stage are these grants currently? MULTI-SELECT g_project_stage 01 Prototyping Please select all that apply. 02 Fabrication 03 Pilot testing 04 Early stage market assessment 05 Dissemination 06 Scale-up 96 Other (please specify) 99 Don't know 98 Choose not to answer At what innovation stage are these grants currently? g_project_stage.Contains(96) TEXT g_project_stage_ot Did these grants already produce SINGLE-SELECT g_project_produced research/innovation outputs for sharing? 01 Yes 00 No 99 Don't know What were the research/innovation outputs under these grants used for? Please select all that apply. $produced (self.ContainsAny(1,2,96)&&!self.ContainsAny(99,98,3,4 ) )||(self.ContainsAny(99,98,3,4)&&!self.ContainsAny(1,2,9 6)) Your answers cannot include both substantive answers and "not used yet butwill be used in the future"/"not used for anything"/"don't know"/ "choose not to answer". E.126 | PERFORMANCE EVALUATION OF HESN MULTI-SELECT g_project_use 01 Innovation design and development 02 Solution to local development problem 03 Not used yet but will be used in the future 04 Not used for anything 96 Other specified use 99 Don't know 98 Choose not to answer E.127 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 19 / 25 What were the research/innovation outputs under these grants used for? Please specify other response. g_project_use.Contains(96) TEXT g_project_use_other Why were the research/innovation outputs produced under these grants not used for anything? g_project_use.Contains(4) TEXT g_project_no_use When do you expect that research/innovation outputs produced under these grants will be used? g_project_use.Contains(3) TEXT g_project_use_when What will the research/innovation outputs under these grants be used for? Please select all that apply. g_project_use.Contains(3) (self.ContainsAny(1,2,96)&&!self.ContainsAny(99,98 ))||( self.ContainsAny(99,98)&&!self.ContainsAny(1,2,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_exp_use 01 Innovation design and development 02 Solution to local development problem 96 Other (please specify) 99 Don't know 98 Choose not to answer What will the research/innovation outputs under these grants be used for? g_project_exp_use.Contains(96) TEXT g_project_exp_use_ot How did you disseminate research/innovation outputs from these grants? Please select all that apply. $produced (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98,10,11 ))||(self.ContainsAny(99,98,10,11)&&!self .ContainsAny(1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "Not disse minated yet but will be in the future"/"not disseminated"/"don't know"/ "choose not to answer". MULTI-SELECT g_project_dissem 01 Email 02 Social media 03 Podcast 04 Blog 05 Policy/issue brief 06 Virtual results presentation 07 In-person results presentation 08 Dissemination workshop 09 In the middle of dissemination 10 Not disseminated yet but will be in the future 11 Not disseminated 96 Other (please specify) 99 Don't know 98 Choose not to answer How did you disseminate research/innovation outputs from these grants? Please specify other response. g_project_dissem.Contains(96) TEXT g_project_dissem_ot Why were the research/innovation outputs produced under these grants not disseminated? g_project_dissem.Contains(11) TEXT g_project_no_dissem E.128 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 20 / 25 G. HESN RESEARCH OUTPUTS AND DISSEMINATION Roster: AUDIENCE generated bymulti-select question g_project_dissem_aud g_audience g_project_dissem_aud.ContainsAny(1,2,3,4,5,6,7,96) When do you expect that research/innovation outputs produced under these grants will be disseminated? g_project_dissem.Contains(10) TEXT g_project_dissem_when To what audience did you directly disseminate research/innovation outputs from these grants? Please select all that apply. g_project_dissem.ContainsAny(1,2,3,4,5,6,7,8,9,96) (self.ContainsAny(1,2,3,4,5,6,7,96)&&!self.ContainsAny(9 9,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2 ,3,4,5,6,7,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_dissem_aud 01 USAID Mission(s) 02 External USAID Operating Unit(s) (outside the Global Development Lab) 03 Policymakers in the country where research/innovation took place 04 Researchers/innovations in the country where research/innovation took place 05 Other stakeholder organizations where the research/innovation took place 06 Policymakers in other countries 07 Researchers/innovators in other countries 96 Other audience 99 Don't know 98 Choose not to answer To what audience did you directly disseminate research/innovation outputs from these grants? Please specify other audience. g_project_dissem_aud.Contains(96) TEXT g_project_dissem_aud_ot How many times did you speak with the %g_audience% about research/innovation outputs from these grants? self < 500 && self>=0 NUMERIC: INTEGER g_aud_spoke_num Do you know the position of key representatives from the %g_audience% with whom you spoke to about research/innovation outputs from these grants? SINGLE-SELECT g_aud_spoke_know 01 Yes 02 No Who were the key representatives (organization and position) from the %g_audience% with whom you spoke about research/innovation outputs from these grants? For each key person please just list their position and organization. g_aud_spoke_know==1 LIST g_aud_spoke_ppl E.129 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 21 / 25 Please indicate your level of agreement with: the %g_audience% showed strong interest in research/innovation outputs from these grants? SINGLE-SELECT g_aud_interest 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Doesn't know Please indicate your level of agreement with: the %g_audience% indicated that research/innovation outputs from these grants were useful to their planning or programs? SINGLE-SELECT g_aud_use 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Doesn't know Did any additional collaborations result beyond the original scope of these grants? SINGLE-SELECT g_project_additional 01 Yes 00 No 99 Don't know What are the main objectives of the additional collaborations beyond these grants? Please select all that apply. $additional (self.ContainsAny(1,2,3,4,5,6,96)&&!self.ContainsAny(99, 98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3 ,4,5,6,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_add_obj 01 Prototyping 02 Fabrication 03 Pilot testing 04 Early stage market assessment 05 Dissemination 06 Scale-up 96 Other (please specify) 99 Don't know 98 Choose not to answer What are the main objectives of the additional collaborations beyond these grants? Please specify other response. g_project_add_obj.Contains(96) TEXT g_project_add_obj_ot What are/will be the funding sources for the additional collaborations beyond these grants? Please select all that apply. $additional (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98 ) )||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4,9 6)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_add_fund 01 HESN (through Global Development Lab)/MIT 02 USAID 03 Private sector 04 Local government 96 Other (please specify) 99 Don't know 98 Choose not to answer What are/will be the funding sources for the additional collaborations beyond these grants? Please specify other response. g_project_add_fund.Contains(96) TEXT g_project_add_fund_ot E.130 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 22 / 25 STATIC TEXT Please click on the blue box to proceed to the next section. In what country/countries will the additional collaborations beyond these grants be implemeneted? Ifmore than one country, please press enter to list each country in a s eparate text box. $additional LIST g_project_add_loc H. RECOMMENDATIONS $familiar E.131 | PERFORMANCE EVALUATION OF HESN USAID.GOV H. RECOMMENDATIONS 23 / 25 STATIC TEXT Please click on the blue box to proceed to the next section. Based on your experience with HESN/MIT DIN, what are key areas to help improve partner engagement and the use of innovation/research outputs by partners? Please select all that apply. (self.ContainsAny(1,2,3,4,96)&&!self.Contains(99 ))||(se lf.Contains(99)&&!self.ContainsAny(1,2,3,4,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT h_rec_uptake 01 Allow more direct contact with partners 02 Make research/innovation content less technical 03 More dissemination activities 04 Better alignment with key stakeholder interests 96 Other (please specify) 99 Don't know 98 Choose not to answer Based on your experience with HESN/MIT DIN, what are key areas to help improve partner engagement and the use of innovation/research outputs by partners? Please specify other response. h_rec_uptake.Contains(96) TEXT h_rec_uptake_other Do you have any other comments or suggestions related to your experience with HESN/MIT IDIN? SINGLE-SELECT h_has_comment 01 Yes (please specify) 00 No Do you have any other comments or suggestions related to your experience with HESN/MIT IDIN? Please specify any other comments. h_has_comment==1 TEXT h_comment_specify I. FOLLOW-UP $familiar E.132 | PERFORMANCE EVALUATION OF HESN USAID.GOV I. FOLLOW-UP 24 / 25 In order to learn more about the utility of the HESN mechanism/MIT IDIN and research/innovation outputs, we would like to conduct short in-person or phone interviews with a subsample of online survey respondents. Would you be willing to participate in a short follow-up interview? SINGLE-SELECT i_follow_part 01 Yes 00 No STATIC TEXT $follow Please provide the below information: STATIC TEXT Thank you for your time! Please click on the box below to record the survey start time. DATE: CURRENT TIME i_end_time STATIC TEXT Please click on the blue box to proceed to the next section. First name: $follow TEXT i_first_name Last name: $follow TEXT i_last_name Email: $follow TEXT i_email Phone number (with country code): $follow TEXT i_phone Current organization: $follow TEXT i_organ Current position: $follow TEXT i_position E.133 | PERFORMANCE EVALUATION OF HESN USAID.GOV LEGEND 25 / 25 LEGEND Legend and structure of information in this file This page has been left blank for double-sided copying. Generated by edithgfelix, Apr 06, 2020 12:31 Questionnaire created by edithgfelix, Dec 15, 2019 17:34 Last modified by edithgfelix, Dec 15, 2019 19:56 Not shared with anyone Sections: 9, Sub-sections: 7, Questions: 103. Questions with enabling conditions: 48 Questions with validation conditions:20 Rosters: 1 Variables: 0 HESN Innovator RAN Survey SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION A. INTRODUCTION No sub-sections, No rosters,Questions: 4, Static texts: 2. B. GENERAL INFORMATION No sub-sections, No rosters,Questions: 5, Static texts: 2. C. RESEARCH/WORK EXPERIENCE No sub-sections, No rosters,Questions: 6, Static texts: 2. D. HESN GRANTS No sub-sections, No rosters,Questions: 17, Static texts: 3. E. EXPERIENCE WITH HESN Sub-sections: 4,No rosters,Questions: 15, Static texts: 11. F. HESN USEFULNESS AND SUSTAINABILITY Sub-sections: 3,No rosters,Questions: 17, Static texts: 9. G. HESN RESEARCH OUTPUTS AND DISSEMINATION No sub-sections,Rosters: 1,Questions: 27, Static texts: 2. H. RECOMMENDATIONS No sub-sections, No rosters,Questions: 4, Static texts: 1. I. FOLLOW-UP No sub-sections, No rosters,Questions: 8, Static texts: 3. APPENDIX A — CATEGORIES LEGEND E.135 | PERFORMANCE EVALUATION OF HESN USAID.GOV 1 / 27 E.136 | PERFORMANCE EVALUATION OF HESN USAID.GOV SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION 2 / 27 SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION Basic information Title HESN Innovator RAN Survey E.137 | PERFORMANCE EVALUATION OF HESN USAID.GOV A. INTRODUCTION 3 / 27 A. INTRODUCTION STATIC TEXT ● This survey gathers information about your perspectives and experience with the Higher Education Solutions Network (HESN)/the ResilientAfrica Network (RAN). ● If you have participated in another USAID Global Development Lab program such as LASER, RTAC, STIP APS, or PEER, please only provide information related to your perspectives and experience with HESN. ● USAID’s Global Development Lab has contracted Mathematica to conduct an independent evaluation of the long-term impacts of HESN. ● USAID is interested in understanding the ultimate impact/utility of HESN projects implemented since 2015 and the types of partnerships thatmay produce policy impact. ● This survey should take approximately 30 minutes to complete. ● Any information you provide that can identify you will be kept strictly confidential. Information provided will be used for research purposes only. Your participation is voluntary and you may choose not to answer any or all questions for any reason. ● You may contact Audrey-Marie Moore, the Mathematica Senior Researcher leading this evaluation at AMoore@mathematica-mpr.com, if you have study questions, concerns or complaints. Thank you for your time and help with this evaluation! Please click on the box below to record the survey start time. DATE: CURRENT TIME a_start_time Do you consent to participate in this survey? SINGLE-SELECT a_consent 01 Yes 00 No Name TEXT prefilled_name SCOPE: IDENTIFYING Email TEXT prefilled_email SCOPE: IDENTIFYING STATIC TEXT Please click on the box to proceed to the next section. B. GENERAL INFORMATION a_consent==1 E.138 | PERFORMANCE EVALUATION OF HESN USAID.GOV B. GENERAL INFORMATION 4 / 27 STATIC TEXT To begin, we would like to ask you some general information about HESN/the ResilientAfrica Network (RAN) STATIC TEXT Please click on the blue box to proceed to the next section. Are you familiar with the the Higher Education Solutions Network (HESN)/the ResilientAfrica Network (RAN)? SINGLE-SELECT b_familiar 01 Yes 00 No How did you first become aware of HESN/RAN? Please select all that apply. $familiar (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98)) ||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT b_first_aware 01 Other community members 02 A nongovernmental organization 03 A social venture/enterprise 04 A local university 05 ResilientAfrica Network (RAN) member 06 RAN event 07 RAN communications 08 RAN innovation output 09 USAID communications 96 Other (please specify) 99 Don't know 98 Choose not to answer How did you first become aware of HESN/RAN? Please specify other response. b_first_aware.Contains(96) TEXT b_first_aware_other What role(s) did you play under HESN/RAN? Please select all that apply. $familiar (self.ContainsAny(1,2,3,96)&&!self.ContainsAny(99,98)) | |(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT b_role 01 RAN grantee 02 Innovator 03 Researcher 96 Other (please specify) 99 Don't know 98 Choose not to answer What role(s) did you play under HESN/RAN? Please specify other response. b_role.Contains(96) TEXT b_role_other C. RESEARCH/WORK EXPERIENCE $familiar E.139 | PERFORMANCE EVALUATION OF HESN USAID.GOV C. RESEARCH/WORK EXPERIENCE 5 / 27 STATIC TEXT Now, we would like to ask you about your individual research/work sector experience. What are your primary research or work sectors? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) MULTI-SELECT c_primary_sec 01 Agriculture, food distribution, food security 02 Democracy, human rights, and governance 03 Economic growth and trade (including infrastructure) 04 Education 05 Environment and climate change 06 Gender and women’s empowerment 07 Health and nutrition 08 Water and sanitation 09 Crisis and conflict 96 Other (please specify) 99 Don't know 98 Choose not to answer What is your primary research or work sector? Please specify other response. c_primary_sec.Contains(96) TEXT c_primary_sec_other How many years have you worked in international development (poverty alleviation, social innovation, public health, etc.)? self < 80 && self>=0 This number does not seemcorrect, please double check. NUMERIC: INTEGER c_res_dev_yrs Did you collaborate with any of the following entities on a research or innovation project prior to your participation in the HESN? Please select all that apply. (self.ContainsAny(1,2,3,4,5)&&!self.ContainsAny(99,98,6 ))||(self.ContainsAny(99,98,6)&&!self.ContainsAny(1,2,3, 4,5)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer"/"did not collaborate with any of these entiti es". MULTI-SELECT c_prior_collab_ent 01 USAID Operating Unit (ex. Bureau for Global Health, Bureau for Food Security) 02 USAID Mission 03 HESN Global Development Lab university 04 Other international development aid agency 05 Policymakers in developing countries 06 Did not collaborate with any of these entities 99 Don't know 98 Choose not to answer E.140 | PERFORMANCE EVALUATION OF HESN USAID.GOV C. RESEARCH/WORK EXPERIENCE 6 / 27 STATIC TEXT Please click on the blue box to proceed to the next section. What sectors of research/innovation were you involved with during your participation with HESN/RAN? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT c_dur_sec 01 Agriculture, food distribution, food security 02 Democracy, human rights, and governance 03 Economic growth and trade (including infrastructure) 04 Education 05 Environment and climate change 06 Gender and women’s empowerment 07 Health and nutrition 08 Water and sanitation 09 Crisis and conflict 96 Other (please specify) 99 Don't know 98 Choose not to answer What sectors of research/innovation were you involved with during your participation with HESN/RAN? Please specify other response. c_dur_sec.Contains(96) TEXT c_dur_sec_other E.141 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. HESN GRANTS 7 / 27 D. HESN GRANTS $familiar STATIC TEXT Next, we would like to ask you several questions about your participation in specific HESN/RAN grants. Did you apply for a grant under HESN/RAN? SINGLE-SELECT d_apply_grant 01 Yes 00 No What type of HESN/RAN grant did you apply for? Please select all that apply. d_apply_grant==1 (self.ContainsAny(1,2,96)&&!self.ContainsAny(99,98 ))||( self.ContainsAny(99,98)&&!self.ContainsAny(1,2,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_apply_grant_type 01 Innovation grant 02 Research grant 96 Other (please specify) 99 Don't know 98 Choose not to answer What was the primary reason you chose to apply for a grant under HESN/RAN? Please select your primary reason. d_apply_grant==1 SINGLE-SELECT d_apply_reason 01 Advancing the use of innovative techniques/products in international development 02 Supporting the work of USAID Operating Units and Missions 03 Gaining research/innovation development experience 96 Other (please specify) 99 Don't know 98 Choose not to answer What was the primary reason you chose to apply for a grant under HESN/RAN? Please specify other response. d_apply_reason==96 TEXT d_apply_reason_ot How many HESN/RAN grants have you been involved with since 2015? SINGLE-SELECT d_number_projects 01 1 to 2 grants 02 3 to 5 grants 03 6 to 10 grants 04 More than 10 grants 95 Did not participate in any individual grants 99 Don't know 98 Choose not to answer E.142 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. HESN GRANTS 8 / 27 STATIC TEXT $projects Think about these grants that you considered to best represent HESN's goals. With these grants in mind, please answer the questions that follow: Of the HESN/RAN grants that you participated in since 2015, which grants (please list at least ONE grant) best represent the HESN goals of: 1. Accelerating the use of innovative technologies and approaches to address global development challenges 2. Catalyzing an interdisciplinary research environment 3. Supporting evidence-based development decision making Please press enter to list each grant name in a separate text box. $projects LIST d_main_project What HESN Lab(s) were these grants part of? MULTI-SELECT d_project_lab 01 ResilientAfrica Network (RAN) 02 UC Berkeley – Development Impact Lab (DIL) 03 MIT – International Development Innovation Network (IDIN) 04 MIT – Comprehensive Initiative on Technology Evaluation (CITE) 05 MSU – Global Center for Food Systems Innovation (GCFSI) 06 Texas A&M – Center of Conflict and Development (ConDev) 07 Duke – Social Entrepreneurship Accelerator at Duke (SEAD) 08 William and Mary – AidData 99 Don't know 98 Choose not to answer Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,7,8)&&!self.ContainsAny(99 ,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2, 3,4,5,6,7,8)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". What specific role(s) did/do you have under these grants? Please select all that apply. $projects (self.ContainsAny(1,2,3,96)&&!self.ContainsAny(99,98 ))| |(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_role 01 RAN grantee 02 Innovator 03 Researcher 96 Other (please specify) 99 Don't know 98 Choose not to answer What specific role(s) did/do you have under these grants? Please specify other response. d_project_role.Contains(96) TEXT d_project_role_other E.143 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. HESN GRANTS 9 / 27 What are the funding sources for these grants? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98 ) )||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4,9 6)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_fund 01 HESN (through Global Development Lab)/RAN 02 USAID 03 Private sector 04 Local government 96 Other (please specify) 99 Don't know 98 Choose not to answer What are the funding sources for these grants? Please specify other response. d_project_fund.Contains(96) TEXT d_project_fund_other What sector(s) did/do these grants cover? MULTI-SELECT d_project_sector Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". 01 Agriculture, food distribution, food security 02 Democracy, human rights, and governance 03 Economic growth and trade (including infrastructure) 04 Education 05 Environment and climate change 06 Gender and women’s empowerment 07 Health and nutrition 08 Water and sanitation 09 Crisis and conflict 96 Other (please specify) 99 Don't know 98 Choose not to answer What sector(s) did/do these grants cover? Please specify other response. d_project_sector.Contains(96) TEXT d_project_sector_other In what country/countries were/are these grants being implemented? Ifmore than one country, please press enter to list each country in a s eparate text box. $projects LIST d_project_loc E.144 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. HESN GRANTS 10 / 27 STATIC TEXT Please click on the blue box to proceed to the next section. What were/are the main objectives of these grants? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,7,8,9,10,11,12,13,14,96)&& !self.ContainsAny(99,98 ))||(self.ContainsAny(99,98)&&!s elf.ContainsAny(1,2,3,4,5,6,7,8,9,10,11,12,13,14,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_object 01 Prototyping 02 Fabrication 03 Pilot testing 04 Early stage market assessment 05 Dissemination 06 Scale-up 07 Inputs for country development strategy or program design 08 Collect data (program monitoring data, demographic, geographic, sectoral, etc.) 09 Impact evaluation of a program 10 Performance evaluation of a program 11 Fill gaps in data 12 Capacity building of USAID Mission/Operating unit 13 Capacity building of partners 14 Policy development 96 Other (please specify) 99 Don't know And 1 other symbols [1] What were/are the main objectives of these grants? Please specify other response. d_project_object.Contains(96) TEXT d_project_object_ot During the life of these grants, how much time did you spend supporting the activity, on average? $projects SINGLE-SELECT d_project_time_spent 01 Less than 1/4 of time 02 1/4 to 1/2 of time 03 More than 1/2, but less than 3/4 of time 04 3/4 to full time 99 Don't know 98 Choose not to answer E.145 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. EXPERIENCE WITH HESN 11 / 27 E. EXPERIENCE WITH HESN $familiar STATIC TEXT Based on your experience with HESN/RAN, please indicate how strongly you agree with each of the following statements. STATIC TEXT Please click on each of the sub-section boxes to view and respond to statements. Each sub-section box will turn green once completed. E. EXPERIENCE WITH HESN HESN/RAN CONTRIBUTED TO: STATIC TEXT HESN/RAN contributed to: Please indicate how strongly you agree with each of the following statements. Design of innovative technology/techniques related to international development (poverty alleviation, social innovation, public health, etc.) issues. SINGLE-SELECT e_design_eval 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Collecting and analyzing of data related to international development (poverty alleviation, social innovation, public health, etc.) issues. SINGLE-SELECT e_collect_data 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E. EXPERIENCE WITH HESN HESN/RAN EXPANDED OPPORTUNITIES FOR THE FOLLOWING TO PARTICIPATE IN THE DESIGN AND IMPLEMENTATION OF INNOVATIONS/RESEARCH: STATIC TEXT HESN/RAN expanded opportunities for the following to participate in the design and implementation of innovations/research: Please indicate how strongly you agree with each of the following statements. STATIC TEE.X1T46 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. EXPERIENCE WITH HESN 12 / 27 Local community members SINGLE-SELECT e_cmty_members 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Local researchers SINGLE-SELECT e_loc_researchers 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Local students SINGLE-SELECT e_loc_students 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Innovators/researchers from other developing countries SINGLE-SELECT e_other_innov 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know U.S. researchers SINGLE-SELECT e_us_researchers 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know U.S. students SINGLE-SELECT e_us_students 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E. EXPERIENCE WITH HESN PARTICIPATING IN HESN/RAN SUPPORTED GRANTS HAS: E.147 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. EXPERIENCE WITH HESN 13 / 27 Participating in HESN/RAN supported grants has: Please indicate how strongly you agree with each of the following statements. Provided opportunities for people to present innovative tools/technology/techniques to a broader audience. SINGLE-SELECT e_present_innov 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Provided opportunities for innovators/researchers like myself to engage with universities outside of the country. SINGLE-SELECT e_engage_outside 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Created new career opportunities for people like myself who focus on using innovative techniques. SINGLE-SELECT e_created_opportunities 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Motivated people to pursue work in international development (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT e_motivated_people 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E. EXPERIENCE WITH HESN HESN/RAN HAS PROVIDED: STATIC TEXT HESN/RAN has provided: Please indicate how strongly you agree with each of the following statements. Additional financial resources to support innovation/research through new donors and partners. SINGLE-SELECT e_financial_resources 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.148 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. EXPERIENCE WITH HESN 14 / 27 Technical resources to support innovation/research. SINGLE-SELECT e_tech_resources 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Motivation for me to participate in another HESN/RAN project/activity if given the opportunity. SINGLE-SELECT e_participate_opportunity 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. STATIC TEXT Please click on the blue box to proceed to the next section. E.149 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN USEFULNESS AND SUSTAINABILITY 15 / 27 F. HESN USEFULNESS AND SUSTAINABILITY $familiar STATIC TEXT $familiar Based on your experience with HESN/RAN, please indicate how strongly you agree with each of the following statements. STATIC TEXT Please click on each of the sub-section boxes to view and respond to statements. Each sub-section box will turn green once completed. F. HESN USEFULNESS AND SUSTAINABILITY THE WORK PERFORMED UNDER THE HESN/RAN GRANTS: STATIC TEXT The work performed under the HESN/RAN grants: Please indicate how strongly you agree with each of the following statements. Produced useful research/innovation products for USAID Missions and other stakeholders. SINGLE-SELECT f_useful_research 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Was relevant to development (poverty alleviation, social innovation, public health, etc.) issues in the target country or countries. SINGLE-SELECT f_rel_dev_issues 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Came at a time when the enabling environment was ready for new techniques/tools/technology that help solve international development (poverty alleviation, social innovation, public health, etc.) challenges. SINGLE-SELECT f_timely_environ 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Helped expand my international network. SINGLE-SELECT f_expand_network 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.150 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN USEFULNESS AND SUSTAINABILITY 16 / 27 Helped broaden the use of innovative techniques in my sector. SINGLE-SELECT f_broad_innov 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. F. HESN USEFULNESS AND SUSTAINABILITY MY EXPERIENCE WORKING WITH THE HESN LAB/RAN: STATIC TEXT My experience working with the HESN Lab/RAN: Please indicate how strongly you agree with each of the following statements. Has been positive. SINGLE-SELECT f_exp_pos 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has increased my interest in international development (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT f_inc_interest 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has helped me develop additional innovation design skills. SINGLE-SELECT f_dev_skills 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has provided me with opportunities to develop other skills. SINGLE-SELECT f_opp_skill_grow 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.151 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN USEFULNESS AND SUSTAINABILITY 17 / 27 Has provided me with opportunities for career advancement. SINGLE-SELECT f_opp_career 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has led to opportunities for innovation fellowships or follow-on activities. SINGLE-SELECT f_opp_fellow 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has led to opportunities to travel to other countries and present my work. SINGLE-SELECT f_opp_travel 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has provided me with opportunities for dissemination of my work. SINGLE-SELECT f_opp_dissem 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has provided me with opportunities to engage policymakers around my research/innovation findings. SINGLE-SELECT f_opp_policy 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. F. HESN USEFULNESS AND SUSTAINABILITY OPPORTUNITES FOR: STATIC TEXT Opportunities for: Please indicate how strongly you agree with each of the following statements. E.152 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN USEFULNESS AND SUSTAINABILITY 18 / 27 The development and implementation of techniques in international development (poverty alleviation, social innovation, public health, etc.) are plentiful. SINGLE-SELECT f_innov_dev_opp 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know The use of innovative tools/techniques/technologies will grow in the next five years. SINGLE-SELECT f_innov_use_grow 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Continued funding for development of innovative tools/techniques/technologies are sufficient. SINGLE-SELECT f_innov_fund 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. STATIC TEXT Please click on the blue box to proceed to the next section. E.153 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 19 / 27 G. HESN RESEARCH OUTPUTS AND DISSEMINATION $familiar && $projects STATIC TEXT Now we would like to ask you a few questions about the dissemination of results/research outputs from HESN/RAN grants. Once again, think about the HESN/RAN grants that you considered to best represent HESN's goals. With these grants in mind, please answer the questions that follow: What research/innovation outputs were/will be produced as part of these grants? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,7,8,9,10,11,96)&&!self.Con tainsAny(99,98 ))||(self.ContainsAny(99,98)&&!self.Conta insAny(1,2,3,4,5,6,7,8,9,10,11,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_research 01 Innovation prototype 02 Pilot testing report 03 Market assessment report 04 Finished innovation 05 Report or publication 06 Policy brief 07 Toolkit 08 Data set/platform/mapping 09 Evaluation 10 Training 11 Event or conference 96 Other (please specify) 99 Don't know 98 Choose not to answer What research/innovation outputs were/will be produced as part of these grants? Please specify other response. g_project_research.Contains(96) TEXT g_project_research_other At what innovation stage are these grants currently? Please select all that apply. MULTI-SELECT g_project_stage 01 Prototyping 02 Fabrication 03 Pilot testing 04 Early stage market assessment 05 Dissemination 06 Scale-up 07 Grant/project was not for the development of an innovation 96 Other (please specify) 99 Don't know 98 Choose not to answer At what innovation stage are these grants currently? g_project_stage.Contains(96) TEXT g_project_stage_ot Did these grants already produce research/innovation outputs for sharing? SINGLE-SELECT g_project_produced 01 Yes 00 No 99 Don't know E.154 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 20 / 27 What were the research/innovation outputs under these grants used for? Please select all that apply. $produced (self.ContainsAny(1,2,4,5,6,7,96)&&!self.ContainsAny(99, 98,8,9 ))||(self.ContainsAny(99,98,8,9)&&!self.ContainsA ny(1,2,4,5,6,7,96)) Your answers cannot include both substantive answers and "not used yet butwill be used in the future"/"not used for anything"/"don't know"/ "choose not to answer". MULTI-SELECT g_project_use 01 Innovation design and development 02 Solution to local development problem 04 Inputs for country development strategy or program design 05 Capacity building of USAID Mission/Operating Unit 06 Capacity building of government in target country 07 Policy development 08 Not used yet but will be used in the future 09 Not used for anything 96 Other specified use 99 Don't know 98 Choose not to answer What were the research/innovation outputs under these grants used for? Please specify other response. g_project_use.Contains(96) TEXT g_project_use_other Why were the research/innovation outputs produced under these grants not used for anything? g_project_use.Contains(9) TEXT g_project_no_use When do you expect that research/innovation outputs produced under these grants will be used? g_project_use.Contains(8) TEXT g_project_use_when What will the research/innovation outputs under these grants be used for? Please select all that apply. g_project_use.Contains(8) (self.ContainsAny(1,2,3,4,5,6,7,96)&&!self.ContainsAny(9 9,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2 ,3,4,5,6,7,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_exp_use 01 Innovation design and development 02 Solution to local development problem 04 Inputs for country development strategy or program design 05 Capacity building of USAID Mission/Operating Unit 06 Capacity building of government in target country 07 Policy development 96 Other specified use 99 Don't know 98 Choose not to answer What will the research/innovation outputs under these grants be used for? g_project_exp_use.Contains(96) TEXT g_project_exp_use_ot G. HESN RESEARCH OUTPUTS AND DISSEMINATION 21 / 27 How did you disseminate research/innovation outputs from these grants? Please select all that apply. $produced (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98,10,11 ))||(self.ContainsAny(99,98,10,11)&&!self .ContainsAny(1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "Not disse minated yet but will be in the future"/"not disseminated"/"don't know"/ "choose not to answer". MULTI-SELECT g_project_dissem 01 Email 02 Social media 03 Podcast 04 Blog 05 Policy/issue brief 06 Virtual results presentation 07 In-person results presentation 08 Dissemination workshop 09 In the middle of dissemination 10 Not disseminated yet but will be in the future 11 Not disseminated 96 Other (please specify) 99 Don't know 98 Choose not to answer How did you disseminate research/innovation outputs from these grants? Please specify other response. g_project_dissem.Contains(96) TEXT g_project_dissem_ot Why were the research/innovation outputs produced under these grants not disseminated? g_project_dissem.Contains(11) TEXT g_project_no_dissem When do you expect that research/innovation outputs produced under these grants will be disseminated? g_project_dissem.Contains(10) TEXT g_project_dissem_when To what audience did you directly disseminate research/innovation outputs from these grants? Please select all that apply. g_project_dissem.ContainsAny(1,2,3,4,5,6,7,8,9,96) (self.ContainsAny(1,2,3,4,5,6,7,96)&&!self.ContainsAny(9 9,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2 ,3,4,5,6,7,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_dissem_aud 01 USAID Mission(s) 02 External USAID Operating Unit(s) (outside the Global Development Lab) 03 Policymakers in the country where research/innovation took place 04 Researchers/innovations in the country where research/innovation took place 05 Other stakeholder organizations where the research/innovation took place 06 Policymakers in other countries 07 Researchers/innovators in other countries 96 Other audience 99 Don't know 98 Choose not to answer To what audience did you directly disseminate research/innovation outputs from these grants? Please specify other audience. g_project_dissem_aud.Contains(96) TEXT g_project_dissem_aud_ot E.156 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 22 / 27 G. HESN RESEARCH OUTPUTS AND DISSEMINATION Roster: AUDIENCE generated bymulti-select question g_project_dissem_aud g_audience g_project_dissem_aud.ContainsAny(1,2,3,4,5,6,7,96) How many times did you speak with the %g_audience% about research/innovation outputs from these grants? NUMERIC: INTEGER g_aud_spoke_num Do you know the position of key representatives from the %g_audience% with whom you spoke to about research/innovation outputs from these grants? SINGLE-SELECT g_aud_spoke_know 01 Yes 02 No Who were the key representatives (organization and position) from the %g_audience% with whom you spoke about research/innovation outputs from these grants? For each key person please just list their position and organization. g_aud_spoke_know==1 LIST g_aud_spoke_ppl Please indicate your level of agreement with: the %g_audience% showed strong interest in research/innovation outputs from these grants? SINGLE-SELECT g_aud_interest 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Doesn't know Please indicate your level of agreement with: the %g_audience% indicated that research/innovation outputs from these grants were useful to their planning or programs? SINGLE-SELECT g_aud_use 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Doesn't know Did any additional collaborations result beyond the original scope of these grants? SINGLE-SELECT g_project_additional 01 Yes 00 No 99 Don't know E.157 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 23 / 27 STATIC TEXT Please click on the blue box to proceed to the next section. What are the main objectives of the additional collaborations beyond these grants? Please select all that apply. $additional (self.ContainsAny(1,2,3,4,5,6,7,8,9,10,11,12,13,14,96)&& !self.ContainsAny(99,98 ))||(self.ContainsAny(99,98)&&!s elf.ContainsAny(1,2,3,4,5,6,7,8,9,10,11,12,13,14,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_add_obj 01 Prototyping 02 Fabrication 03 Pilot testing 04 Early stage market assessment 05 Dissemination 06 Scale-up 07 Inputs for country development strategy or program design 08 Collect data (program monitoring data, demographic, geographic, sectoral, etc.) 09 Impact evaluation of a program 10 Performance evaluation of a program 11 Fill gaps in data 12 Capacity building of USAID Mission/Operating unit 13 Capacity building of partners 14 Policy development 96 Other (please specify) 99 Don't know And 1 other symbols [2] What are the main objectives of the additional collaborations beyond these grants? Please specify other response. g_project_add_obj.Contains(96) TEXT g_project_add_obj_ot What are/will be the funding sources for the additional collaborations beyond these grants? Please select all that apply. $additional (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98 ) )||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4,9 6)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_add_fund 01 HESN (through Global Development Lab)/RAN 02 USAID 03 Private sector 04 Local government 96 Other (please specify) 99 Don't know 98 Choose not to answer In what country/countries will the additional collaborations beyond these grants be implemeneted? Ifmore than one country, please press enter to list each country in a s eparate text box. $additional LIST g_project_add_loc H. RECOMMENDATIONS $familiar E.158 | PERFORMANCE EVALUATION OF HESN USAID.GOV H. RECOMMENDATIONS 24 / 27 STATIC TEXT Please click on the blue box to proceed to the next section. Based on your experience with HESN/RAN what are key areas to help improve partner engagement and the use of innovation/research outputs by partners? Please select all that apply. (self.ContainsAny(1,2,3,4,96)&&!self.Contains(99 ))||(se lf.Contains(99)&&!self.ContainsAny(1,2,3,4,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT h_rec_uptake 01 Allow more direct contact with partners 02 Make research/innovation content less technical 03 More dissemination activities 04 Better alignment with key stakeholder interests 96 Other (please specify) 99 Don't know 98 Choose not to answer Based on your experience with HESN/RAN, what are key areas to help improve partner engagement and the use of innovation/research outputs by partners? Please specify other response. h_rec_uptake.Contains(96) TEXT h_rec_uptake_other Do you have any other comments or suggestions related to your experience with HESN/RAN? SINGLE-SELECT h_has_comment 01 Yes (please specify) 00 No Do you have any other comments or suggestions related to your experience with HESN/RAN? Please specify any other comments. h_has_comment==1 TEXT h_comment_specify I. FOLLOW-UP $familiar E.159 | PERFORMANCE EVALUATION OF HESN USAID.GOV I. FOLLOW-UP 25 / 27 In order to learn more about the utility of the HESN mechanism/RAN and research/innovation outputs, we would like to conduct short in-person or phone interviews with a subsample of online survey respondents. Would you be willing to participate in a short follow-up interview? SINGLE-SELECT i_follow_part 01 Yes 00 No STATIC TEXT $follow Please provide the below information: STATIC TEXT Thank you for your time! Please click on the box below to record the survey start time. DATE: CURRENT TIME i_end_time STATIC TEXT Please click on the blue box to proceed to the next section. First name: $follow TEXT i_first_name Last name: $follow TEXT i_last_name Email: $follow TEXT i_email Phone number (with country code): $follow TEXT i_phone Current organization: $follow TEXT i_organ Current position: $follow TEXT i_position APPENDIX A — CATEGORIES E.160 | PERFORMANCE EVALUATION OF HESN USAID.GOV APPENDIX A — CATEGORIES 26 / 27 [1] d_project_object: Whatwere/are themain objectives of these grants? Categories: 1:Prototyping, 2:Fabrication, 3:Pilot testing, 4:Early stage market assessment, 5:Dissemination, 6:Scale-up, 7:Inputs for country development strategy or programdesign, 8:Collect data (programmonitoring data, demographic, geographic, sectoral, etc.), 9:Impact evaluati on of a program, 10:Performance evaluation of a program, 11:Fill gaps in data, 12:Capacity building of USAIDMission/Operating unit, 13:Cap acity building of partners, 14:Policy development, 96:Other (please specify), 99:Don't know, 98:Choose not to answer [2] g_project_add_obj:What are themain objectives of the additional collaborations beyond these grants? Categories: 1:Prototyping, 2:Fabrication, 3:Pilot testing, 4:Early stage market assessment, 5:Dissemination, 6:Scale-up, 7:Inputs for country development strategy or programdesign, 8:Collect data (programmonitoring data, demographic, geographic, sectoral, etc.), 9:Impact evaluati on of a program, 10:Performance evaluation of a program, 11:Fill gaps in data, 12:Capacity building of USAIDMission/Operating unit, 13:Cap acity building of partners, 14:Policy development, 96:Other (please specify), 99:Don't know, 98:Choose not to answer LEGEND E.161 | PERFORMANCE EVALUATION OF HESN USAID.GOV LEGEND 27 / 27 Legend and structure of information in this file This page has been left blank for double-sided copying. Generated by edithgfelix, Apr 06, 2020 12:30 Questionnaire created by edithgfelix, Dec 17, 2019 19:07 Last modified by edithgfelix, Dec 18, 2019 12:01 Not shared with anyone Sections: 9, Sub-sections: 7, Questions: 103. Questions with enabling conditions: 48 Questions with validation conditions:20 Rosters: 1 Variables: 0 HESN Innovator DIL Survey SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION A. INTRODUCTION No sub-sections, No rosters,Questions: 4, Static texts: 2. B. GENERAL INFORMATION No sub-sections, No rosters,Questions: 5, Static texts: 2. C. RESEARCH/WORK EXPERIENCE No sub-sections, No rosters,Questions: 6, Static texts: 2. D. HESN GRANTS/RESEARCH PROJECTS No sub-sections, No rosters,Questions: 17, Static texts: 3. E. EXPERIENCE WITH HESN Sub-sections: 4,No rosters,Questions: 15, Static texts: 11. F. HESN USEFULNESS AND SUSTAINABILITY Sub-sections: 3,No rosters,Questions: 17, Static texts: 9. G. HESN RESEARCH OUTPUTS AND DISSEMINATION No sub-sections,Rosters: 1,Questions: 27, Static texts: 2. H. RECOMMENDATIONS No sub-sections, No rosters,Questions: 4, Static texts: 1. I. FOLLOW-UP No sub-sections, No rosters,Questions: 8, Static texts: 3. APPENDIX A — CATEGORIES LEGEND E.163 | PERFORMANCE EVALUATION OF HESN USAID.GOV 1 / 27 E.164 | PERFORMANCE EVALUATION OF HESN USAID.GOV SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION 2 / 27 SURVEY IDENTIFICATION INFORMATION QUESTIONNAIRE DESCRIPTION Basic information Title HESN Innovator DIL Survey E.165 | PERFORMANCE EVALUATION OF HESN USAID.GOV A. INTRODUCTION 3 / 27 A. INTRODUCTION STATIC TEXT ● This survey gathers information about your perspectives and experience with the Higher Education Solutions Network (HESN)/the University of California Berkeley's Development Impact Lab (DIL). ● If you have participated in another USAID Global Development Lab program such as LASER, RTAC, STIP APS, or PEER, please only provide information related to your perspectives and experience with HESN. ● USAID’s Global Development Lab has contracted Mathematica to conduct an independent evaluation of the long-term impacts of HESN. ● USAID is interested in understanding the ultimate impact/utility of HESN projects implemented since 2015 and the types of partnerships thatmay produce policy impact. ● This survey should take approximately 30 minutes to complete. ● Any information you provide that can identify you will be kept strictly confidential. Information provided will be used for research purposes only. Your participation is voluntary and you may choose not to answer any or all questions for any reason. ● You may contact Audrey-Marie Moore, the Mathematica Senior Researcher leading this evaluation at AMoore@mathematica-mpr.com, if you have study questions, concerns or complaints. Thank you for your time and help with this evaluation! Please click on the box below to record the survey start time. DATE: CURRENT TIME a_start_time Do you consent to participate in this survey? SINGLE-SELECT a_consent 01 Yes 00 No Name TEXT prefilled_name SCOPE: IDENTIFYING Email TEXT prefilled_email SCOPE: IDENTIFYING STATIC TEXT Please click on the box to proceed to the next section. B. GENERAL INFORMATION E a_consent==1 E.166 | PERFORMANCE EVALUATION OF HESN USAID.GOV B. GENERAL INFORMATION 4 / 27 STATIC TEXT To begin, we would like to ask you some general information about HESN/the University of California Berkeley's Development Impact Lab (DIL). STATIC TEXT Please click on the blue box to proceed to the next section. Are you familiar with the Higher Education Solutions Network (HESN)/the University of California Berkeley's Development Impact Lab (DIL)? SINGLE-SELECT b_familiar 01 Yes 00 No How did you first become aware of HESN/DIL? Please select all that apply. $familiar (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98)) ||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT b_first_aware 01 Other community members 02 A nongovernmental organization 03 A social venture/enterprise 04 A local university 05 DIL member 06 DIL event 07 DIL communications 08 DIL research/innovation output 09 USAID communications 96 Other (please specify) 99 Don't know 98 Choose not to answer How did you first become aware of HESN/DIL? Please specify other response. b_first_aware.Contains(96) TEXT b_first_aware_other What role(s) did you play under HESN/DIL? Please select all that apply. $familiar (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98)) ||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4,96 )) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT b_role 01 DIL Innovate grantee 02 DIL Explore grantee 03 Innovator 04 Researcher 96 Other (please specify) 99 Don't know 98 Choose not to answer What role(s) did you play under HESN/DIL? Please specify other response. b_role.Contains(96) TEXT b_role_other C. RESEARCH/WORK EXPERIENCE $familiar E.167 | PERFORMANCE EVALUATION OF HESN USAID.GOV C. RESEARCH/WORK EXPERIENCE 5 / 27 STATIC TEXT Now, we would like to ask you about your individual research/work sector experience. What are your primary research or work sectors? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) MULTI-SELECT c_primary_sec 01 Agriculture, food distribution, food security 02 Democracy, human rights, and governance 03 Economic growth and trade (including infrastructure) 04 Education 05 Environment and climate change 06 Gender and women’s empowerment 07 Health and nutrition 08 Water and sanitation 09 Crisis and conflict 96 Other (please specify) 99 Don't know 98 Choose not to answer What is your primary research or work sector? Please specify other response. c_primary_sec.Contains(96) TEXT c_primary_sec_other How many years have you worked in international development (poverty alleviation, social innovation, public health, etc.)? self < 80 && self>=0 This number does not seemcorrect, please double check. NUMERIC: INTEGER c_res_dev_yrs Did you collaborate with any of the following entities on a research or innovation project prior to your participation in the HESN/DIL? Please select all that apply. (self.ContainsAny(1,2,3,4,5)&&!self.ContainsAny(99,98,6 ))||(self.ContainsAny(99,98,6)&&!self.ContainsAny(1,2,3, 4,5)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer"/"did not collaborate with any of these entiti es". MULTI-SELECT c_prior_collab_ent 01 USAID Operating Unit (ex. Bureau for Global Health, Bureau for Food Security) 02 USAID Mission 03 HESN Global Development Lab university 04 Other international development aid agency 05 Policymakers in developing countries 06 Did not collaborate with any of these entities 99 Don't know 98 Choose not to answer E.168 | PERFORMANCE EVALUATION OF HESN USAID.GOV C. RESEARCH/WORK EXPERIENCE 6 / 27 STATIC TEXT Please click on the blue box to proceed to the next section. What sectors of research/innovation were you involved with during your participation with HESN/DIL? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT c_dur_sec 01 Agriculture, food distribution, food security 02 Democracy, human rights, and governance 03 Economic growth and trade (including infrastructure) 04 Education 05 Environment and climate change 06 Gender and women’s empowerment 07 Health and nutrition 08 Water and sanitation 09 Crisis and conflict 96 Other (please specify) 99 Don't know 98 Choose not to answer What sectors of research/innovation were you involved with during your participation with HESN/DIL? Please specify other response. c_dur_sec.Contains(96) TEXT c_dur_sec_other E.169 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. HESN GRANTS/RESEARCH PROJECTS 7 / 27 D. HESN GRANTS/RESEARCH PROJECTS $familiar STATIC TEXT Next, we would like to ask you several questions about your participation in specific HESN/DIL grants/research projects, which may include specific activities such as developing an innovation, conducting a study/evaluation, undertaking a country-level mapping/geo-referencing activity, etc. Did you apply for a grant under HESN/DIL? SINGLE-SELECT d_apply_grant 01 Yes 00 No What type of HESN/DIL grant did you apply for? Please select all that apply. d_apply_grant==1 (self.ContainsAny(1,2,96)&&!self.ContainsAny(99,98 ))||( self.ContainsAny(99,98)&&!self.ContainsAny(1,2,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_apply_grant_type 01 DIL Innovate grant 02 DIL Explore grant 96 Other (please specify) 99 Don't know 98 Choose not to answer What was the primary reason you chose to apply for a grant under HESN/DIL? Please select your primary reason. d_apply_grant==1 SINGLE-SELECT d_apply_reason 01 Advancing the use of innovative techniques/products in international development 02 Supporting the work of USAID Operating Units and Missions 03 Gaining research/innovation development experience 96 Other (please specify) 99 Don't know 98 Choose not to answer What was the primary reason you chose to apply for a grant under HESN/DIL? Please specify other response. d_apply_reason==96 TEXT d_apply_reason_ot How many HESN/DIL grants/research projects have you been involved with since 2015? SINGLE-SELECT d_number_projects 01 1 to 2 grants/research projects/research projects 02 3 to 5 grants/research projects/research projects 03 6 to 10 grants/research projects/research projects 04 More than 10 grants/research projects 95 Did not participate in any individual grants/research projects 99 Don't know 98 Choose not to answer E.170 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. HESN GRANTS/RESEARCH PROJECTS 8 / 27 STATIC TEXT $projects Think about these grants/research projects that you considered to best represent HESN's goals. With these grants/research projects/research projects in mind, please answer the questions that follow: Of the HESN/DIL grants/research projects that you participated in since 2015, which grants/research projects (please list at least ONE) best represent the HESN goals of: 1. Accelerating the use of innovative technologies and approaches to address global development challenges 2. Catalyzing an interdisciplinary research environment 3. Supporting evidence-based development decision making Please press enter to list each grant name in a separate text box. $projects LIST d_main_project What HESN Lab(s) were these grants/research projects part of? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,7,8)&&!self.ContainsAny(99 ,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2, 3,4,5,6,7,8)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_lab 01 ResilientAfrica Network (RAN) 02 UC Berkeley – Development Impact Lab (DIL) 03 MIT – International Development Innovation Network (IDIN) 04 MIT – Comprehensive Initiative on Technology Evaluation (CITE) 05 MSU – Global Center for Food Systems Innovation (GCFSI) 06 Texas A&M – Center of Conflict and Development (ConDev) 07 Duke – Social Entrepreneurship Accelerator at Duke (SEAD) 08 William and Mary – AidData 99 Don't know 98 Choose not to answer What specific role(s) did/do you have under these grants/research projects? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98 ) )||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4,9 6)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_role 01 DIL Innovate grantee 02 DIL Explore grantee 03 Innovator 04 Researcher 96 Other (please specify) 99 Don't know 98 Choose not to answer What specific role(s) did/do you have under these grants/research projects? Please specify other response. d_project_role.Contains(96) TEXT d_project_role_other E.171 | PERFORMANCE EVALUATION OF HESN USAID.GOV D. HESN GRANTS/RESEARCH PROJECTS 9 / 27 What are the funding sources for these grants/research projects? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98 ) )||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4,9 6)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_fund 01 HESN (through Global Development Lab)/DIL 02 USAID 03 Private sector 04 Local government 96 Other (please specify) 99 Don't know 98 Choose not to answer What are the funding sources for these grants/research projects? Please specify other response. d_project_fund.Contains(96) TEXT d_project_fund_other What sector(s) did/do these grants/research projects cover? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny (1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_sector 01 Agriculture, food distribution, food security 02 Democracy, human rights, and governance 03 Economic growth and trade (including infrastructure) 04 Education 05 Environment and climate change 06 Gender and women’s empowerment 07 Health and nutrition 08 Water and sanitation 09 Crisis and conflict 96 Other (please specify) 99 Don't know 98 Choose not to answer What sector(s) did/do these grants/research projects cover? Please specify other response. d_project_sector.Contains(96) TEXT d_project_sector_other In what country/countries were/are these grants/research projects being implemented? Ifmore than one country, please press enter to list each country in a s eparate text box. $projects LIST d_project_loc D. HESN GRANTS/RESEARCH PROJECTS 10 / 27 STATIC TEXT Please click on the blue box to proceed to the next section. E.172 | PERFORMANCE EVALUATION OF HESN USAID.GOV What were/are the main objectives of these grants/research projects? Please select all that apply. $projects (self.ContainsAny(1,2,3,4,5,6,7,8,9,10,11,12,13,14,96)&& !self.ContainsAny(99,98 ))||(self.ContainsAny(99,98)&&!s elf.ContainsAny(1,2,3,4,5,6,7,8,9,10,11,12,13,14,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT d_project_object 01 Prototyping 02 Fabrication 03 Pilot testing 04 Early stage market assessment 05 Dissemination 06 Scale-up 07 Inputs for country development strategy or program design 08 Collect data (program monitoring data, demographic, geographic, sectoral, etc.) 09 Impact evaluation of a program 10 Performance evaluation of a program 11 Fill gaps in data 12 Capacity building of USAID Mission/Operating unit 13 Capacity building of partners 14 Policy development 96 Other (please specify) 99 Don't know And 1 other symbols [1] What were/are the main objectives of these grants/research projects? Please specify other response. d_project_object.Contains(96) TEXT d_project_object_ot During the life of these grants/research projects, how much time did you spend supporting the activity, on average? $projects SINGLE-SELECT d_project_time_spent 01 Less than 1/4 of time 02 1/4 to 1/2 of time 03 More than 1/2, but less than 3/4 of time 04 3/4 to full time 99 Don't know 98 Choose not to answer E. EXPERIENCE WITH HESN 11 / 27 E. EXPERIENCE WITH HESN $familiar STATIC TEXT Based on your experience with HESN/DIL, please indicate how strongly you agree with each of the following statements. STATIC TEXT Please click on each of the sub-section boxes to view and respond to statements. Each sub-section box will turn green once completed. E. EXPERIENCE WITH HESN HESN/DIL CONTRIBUTED TO: STATIC TEXT HESN/DIL contributed to: Please indicate how strongly you agree with each of the following statements. Design of innovative technology/techniques related to international development (poverty alleviation, social innovation, public health, etc.) issues. SINGLE-SELECT e_design_eval 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Collecting and analyzing of data related to international development (poverty alleviation, social innovation, public health, etc.) issues. SINGLE-SELECT e_collect_data 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E. EXPERIENCE WITH HESN HESN/DIL EXPANDED OPPORTUNITIES FOR THE FOLLOWING TO PARTICIPATE IN THE DESIGN AND IMPLEMENTATION OF INNOVATIONS/RESEARCH: STATIC TEXT HESN/DIL expanded opportunities for the following to participate in the design and implementation of innovations/research: Please indicate how strongly you agree with each of the following statements. E.173 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. EXPERIENCE WITH HESN 12 / 27 Local community members SINGLE-SELECT e_cmty_members 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Local researchers SINGLE-SELECT e_loc_researchers 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Local students SINGLE-SELECT e_loc_students 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Innovators/researchers from other developing countries SINGLE-SELECT e_other_innov 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know U.S. researchers SINGLE-SELECT e_us_researchers 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know U.S. students SINGLE-SELECT e_us_students 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E. EXPERIENCE WITH HESN PARTICIPATING IN HESN/DIL SUPPORTED GRANTS/RESEARCH PROJECTS HAS: STATIC TEXT E.175 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. EXPERIENCE WITH HESN 13 / 27 Participating in HESN/DIL supported grants/research projects has: Please indicate how strongly you agree with each of the following statements. Provided opportunities for people to present innovative tools/technology/techniques to a broader audience. SINGLE-SELECT e_present_innov 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Provided opportunities for innovators/researchers like myself to engage with universities outside of the country. SINGLE-SELECT e_engage_outside 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Created new career opportunities for people like myself who focus on using innovative techniques. SINGLE-SELECT e_created_opportunities 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Motivated people to pursue work in international development (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT e_motivated_people 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. E. EXPERIENCE WITH HESN HESN/DIL HAS PROVIDED: STATIC TEXT HESN/DIL has provided: Please indicate how strongly you agree with each of the following statements. Additional financial resources to support innovation/research through new donors and partners. SINGLE-SELECT e_financial_resources 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.176 | PERFORMANCE EVALUATION OF HESN USAID.GOV E. EXPERIENCE WITH HESN 14 / 27 Technical resources to support innovation/research. SINGLE-SELECT e_tech_resources 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Motivation for me to participate in another HESN/DIL project/activity if given the opportunity. SINGLE-SELECT e_participate_opportunity 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. STATIC TEXT Please click on the blue box to proceed to the next section. E.177 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN USEFULNESS AND SUSTAINABILITY 15 / 27 F. HESN USEFULNESS AND SUSTAINABILITY $familiar STATIC TEXT $familiar Based on your experience with HESN/DIL, please indicate how strongly you agree with each of the following statements. STATIC TEXT Please click on each of the sub-section boxes to view and respond to statements. Each sub-section box will turn green once completed. F. HESN USEFULNESS AND SUSTAINABILITY THE WORK PERFORMED UNDER THE HESN/DIL GRANTS/RESEARCH PROJECTS: STATIC TEXT The work performed under the HESN/DIL grants/research projects: Please indicate how strongly you agree with each of the following statements. Produced useful research/innovation products for USAID Missions and other stakeholders. SINGLE-SELECT f_useful_research 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Was relevant to development (poverty alleviation, social innovation, public health, etc.) issues in the target country or countries. SINGLE-SELECT f_rel_dev_issues 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Came at a time when the enabling environment was ready for new techniques/tools/technology that help solve international development (poverty alleviation, social innovation, public health, etc.) challenges. SINGLE-SELECT f_timely_environ 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Helped expand my international network. SINGLE-SELECT f_expand_network 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.178 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN USEFULNESS AND SUSTAINABILITY 16 / 27 Helped broaden the use of innovative techniques in my sector. SINGLE-SELECT f_broad_innov 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. F. HESN USEFULNESS AND SUSTAINABILITY MY EXPERIENCE WORKING WITH THE HESN LAB/DIL: STATIC TEXT My experience working with the HESN Lab/DIL: Please indicate how strongly you agree with each of the following statements. Has been positive. SINGLE-SELECT f_exp_pos 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has increased my interest in international development (poverty alleviation, social innovation, public health, etc.). SINGLE-SELECT f_inc_interest 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has helped me develop additional innovation design skills. SINGLE-SELECT f_dev_skills 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has provided me with opportunities to develop other skills. SINGLE-SELECT f_opp_skill_grow 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know E.179 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN USEFULNESS AND SUSTAINABILITY 17 / 27 Has provided me with opportunities for career advancement. SINGLE-SELECT f_opp_career 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has led to opportunities for innovation fellowships or follow-on activities. SINGLE-SELECT f_opp_fellow 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has led to opportunities to travel to other countries and present my work. SINGLE-SELECT f_opp_travel 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has provided me with opportunities for dissemination of my work. SINGLE-SELECT f_opp_dissem 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Has provided me with opportunities to engage policymakers around my research/innovation findings. SINGLE-SELECT f_opp_policy 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. F. HESN USEFULNESS AND SUSTAINABILITY OPPORTUNITES FOR: STATIC TEXT Opportunities for: Please indicate how strongly you agree with each of the following statements. E.180 | PERFORMANCE EVALUATION OF HESN USAID.GOV F. HESN USEFULNESS AND SUSTAINABILITY 18 / 27 The development and implementation of techniques in international development (poverty alleviation, social innovation, public health, etc.) are plentiful. SINGLE-SELECT f_innov_dev_opp 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know The use of innovative tools/techniques/technologies will grow in the next five years. SINGLE-SELECT f_innov_use_grow 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know Continued funding for development of innovative tools/techniques/technologies are sufficient. SINGLE-SELECT f_innov_fund 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Do not know STATIC TEXT Please click on the box below to continue. STATIC TEXT Please click on the blue box to proceed to the next section. E.181 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 19 / 27 G. HESN RESEARCH OUTPUTS AND DISSEMINATION $familiar && $projects STATIC TEXT Now we would like to ask you a few questions about the dissemination of results/research outputs from HESN/DIL grants/research projects. Once again, think about the HESN/DIL grants/research projects that you considered to best represent HESN's goals. With these grants/research projects in mind, please answer the questions that follow: What research/innovation outputs were/will be produced as part of these grants/research projects? Please select all that apply. (self.ContainsAny(1,2,3,4,5,6,7,8,9,10,11,96)&&!self.Con tainsAny(99,98 ))||(self.ContainsAny(99,98)&&!self.Conta insAny(1,2,3,4,5,6,7,8,9,10,11,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_research 01 Innovation prototype 02 Pilot testing report 03 Market assessment report 04 Finished innovation 05 Report or publication 06 Policy brief 07 Toolkit 08 Data set/platform/mapping 09 Evaluation 10 Training 11 Event or conference 96 Other (please specify) 99 Don't know 98 Choose not to answer What research/innovation outputs were/will be produced as part of these grants/research projects? Please specify other response. g_project_research.Contains(96) TEXT g_project_research_other At what innovation stage are these grants/research projects currently? Please select all that apply. MULTI-SELECT g_project_stage 01 Prototyping 02 Fabrication 03 Pilot testing 04 Early stage market assessment 05 Dissemination 06 Scale-up 07 Grant/project was not for the development of an innovation 96 Other (please specify) 99 Don't know 98 Choose not to answer At what innovation stage are these grants/research projects currently? g_project_stage.Contains(96) TEXT g_project_stage_ot Did these grants/research projects already produce research/innovation outputs for sharing? SINGLE-SELECT g_project_produced 01 Yes 00 No 99 Don't know E.182 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 20 / 27 What were the research/innovation outputs under these grants/research projects used for? Please select all that apply. $produced (self.ContainsAny(1,2,4,5,6,7,96)&&!self.ContainsAny(99, 98,8,9 ))||(self.ContainsAny(99,98,8,9)&&!self.ContainsA ny(1,2,4,5,6,7,96)) Your answers cannot include both substantive answers and "not used yet butwill be used in the future"/"not used for anything"/"don't know"/ "choose not to answer". MULTI-SELECT g_project_use 01 Innovation design and development 02 Solution to local development problem 04 Inputs for country development strategy or program design 05 Capacity building of USAID Mission/Operating Unit 06 Capacity building of government in target country 07 Policy development 08 Not used yet but will be used in the future 09 Not used for anything 96 Other specified use 99 Don't know 98 Choose not to answer What were the research/innovation outputs under these grants/research projects used for? Please specify other response. g_project_use.Contains(96) TEXT g_project_use_other Why were the research/innovation outputs produced under these grants/research projects not used for anything? g_project_use.Contains(9) TEXT g_project_no_use When do you expect that research/innovation outputs produced under these grants/research projects will be used? g_project_use.Contains(8) TEXT g_project_use_when What will the research/innovation outputs under these grants/research projects be used for? Please select all that apply. g_project_use.Contains(8) (self.ContainsAny(1,2,3,4,5,6,7,96)&&!self.ContainsAny(9 9,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2 ,3,4,5,6,7,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_exp_use 01 Innovation design and development 02 Solution to local development problem 04 Inputs for country development strategy or program design 05 Capacity building of USAID Mission/Operating Unit 06 Capacity building of government in target country 07 Policy development 96 Other specified use 99 Don't know 98 Choose not to answer What will the research/innovation outputs under these grants/research projects be used for? g_project_exp_use.Contains(96) TEXT g_project_exp_use_ot G. HESN RESEARCH OUTPUTS AND DISSEMINATION 21 / 27 How did you disseminate research/innovation outputs from these grants/research projects? Please select all that apply. $produced (self.ContainsAny(1,2,3,4,5,6,7,8,9,96)&&!self.ContainsA ny(99,98,10,11 ))||(self.ContainsAny(99,98,10,11)&&!self .ContainsAny(1,2,3,4,5,6,7,8,9,96)) Your answers cannot include both substantive answers and "Not disse minated yet but will be in the future"/"not disseminated"/"don't know"/ "choose not to answer". MULTI-SELECT g_project_dissem 01 Email 02 Social media 03 Podcast 04 Blog 05 Policy/issue brief 06 Virtual results presentation 07 In-person results presentation 08 Dissemination workshop 09 In the middle of dissemination 10 Not disseminated yet but will be in the future 11 Not disseminated 96 Other (please specify) 99 Don't know 98 Choose not to answer How did you disseminate research/innovation outputs from these grants/research projects? Please specify other response. g_project_dissem.Contains(96) TEXT g_project_dissem_ot Why were the research/innovation outputs produced under these grants/research projects not disseminated? g_project_dissem.Contains(11) TEXT g_project_no_dissem When do you expect that research/innovation outputs produced under these grants/research projects will be disseminated? g_project_dissem.Contains(10) TEXT g_project_dissem_when To what audience did you directly disseminate research/innovation outputs from these grants/research projects? Please select all that apply. g_project_dissem.ContainsAny(1,2,3,4,5,6,7,8,9,96) (self.ContainsAny(1,2,3,4,5,6,7,96)&&!self.ContainsAny(9 9,98 ))||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2 ,3,4,5,6,7,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_dissem_aud 01 USAID Mission(s) 02 External USAID Operating Unit(s) (outside the Global Development Lab) 03 Policymakers in the country where research/innovation took place 04 Researchers/innovations in the country where research/innovation took place 05 Other stakeholder organizations where the research/innovation took place 06 Policymakers in other countries 07 Researchers/innovators in other countries 96 Other audience 99 Don't know 98 Choose not to answer To what audience did you directly disseminate research/innovation outputs from these grants/research projects? Please specify other audience. g_project_dissem_aud.Contains(96) TEXT g_project_dissem_aud_ot E.184 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 22 / 27 G. HESN RESEARCH OUTPUTS AND DISSEMINATION Roster: AUDIENCE generated bymulti-select question g_project_dissem_aud g_audience g_project_dissem_aud.ContainsAny(1,2,3,4,5,6,7,96) How many times did you speak with the %g_audience% about research/innovation outputs from these grants/research projects? NUMERIC: INTEGER g_aud_spoke_num Do you know the position of key representatives from the %g_audience% with whom you spoke to about research/innovation outputs from these grants/research projects? SINGLE-SELECT g_aud_spoke_know 01 Yes 02 No Who were the key representatives (organization and position) from the %g_audience% with whom you spoke about research/innovation outputs from these grants/research projects? For each key person please just list their position and organization. g_aud_spoke_know==1 LIST g_aud_spoke_ppl Please indicate your level of agreement with: the %g_audience% showed strong interest in research/innovation outputs from these grants/research projects? SINGLE-SELECT g_aud_interest 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Doesn't know Please indicate your level of agreement with: the %g_audience% indicated that research/innovation outputs from these grants/research projects were useful to their planning or programs? SINGLE-SELECT g_aud_use 01 Strongly agree 02 Agree 03 Disagree 04 Strongly disagree 95 Does not apply 99 Doesn't know Did any additional collaborations result beyond the original scope of these grants/research projects? SINGLE-SELECT g_project_additional 01 Yes 00 No 99 Don't know E.185 | PERFORMANCE EVALUATION OF HESN USAID.GOV G. HESN RESEARCH OUTPUTS AND DISSEMINATION 23 / 27 STATIC TEXT Please click on the blue box to proceed to the next section. What are the main objectives of the additional collaborations beyond these grants/research projects? Please select all that apply. $additional (self.ContainsAny(1,2,3,4,5,6,7,8,9,10,11,12,13,14,96)&& !self.ContainsAny(99,98 ))||(self.ContainsAny(99,98)&&!s elf.ContainsAny(1,2,3,4,5,6,7,8,9,10,11,12,13,14,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_add_obj 01 Prototyping 02 Fabrication 03 Pilot testing 04 Early stage market assessment 05 Dissemination 06 Scale-up 07 Inputs for country development strategy or program design 08 Collect data (program monitoring data, demographic, geographic, sectoral, etc.) 09 Impact evaluation of a program 10 Performance evaluation of a program 11 Fill gaps in data 12 Capacity building of USAID Mission/Operating unit 13 Capacity building of partners 14 Policy development 96 Other (please specify) 99 Don't know And 1 other symbols [2] What are the main objectives of the additional collaborations beyond these grants/research projects? Please specify other response. g_project_add_obj.Contains(96) TEXT g_project_add_obj_ot What are/will be the funding sources for the additional collaborations beyond these grants/research projects? Please select all that apply. $additional (self.ContainsAny(1,2,3,4,96)&&!self.ContainsAny(99,98 ) )||(self.ContainsAny(99,98)&&!self.ContainsAny(1,2,3,4,9 6)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT g_project_add_fund 01 HESN (through Global Development Lab)/DIL 02 USAID 03 Private sector 04 Local government 96 Other (please specify) 99 Don't know 98 Choose not to answer In what country/countries will the additional collaborations beyond these grants/research projects be implemeneted? Ifmore than one country, please press enter to list each country in a s eparate text box. $additional LIST g_project_add_loc H. RECOMMENDATIONS $familiar E.186 | PERFORMANCE EVALUATION OF HESN USAID.GOV H. RECOMMENDATIONS 24 / 27 STATIC TEXT Please click on the blue box to proceed to the next section. Based on your experience with HESN/DIL what are key areas to help improve partner engagement and the use of innovation/research outputs by partners? Please select all that apply. (self.ContainsAny(1,2,3,4,96)&&!self.Contains(99 ))||(se lf.Contains(99)&&!self.ContainsAny(1,2,3,4,96)) Your answers cannot include both substantive answers and "don't kno w"/"choose not to answer". MULTI-SELECT h_rec_uptake 01 Allow more direct contact with partners 02 Make research/innovation content less technical 03 More dissemination activities 04 Better alignment with key stakeholder interests 96 Other (please specify) 99 Don't know 98 Choose not to answer Based on your experience with HESN/DIL, what are key areas to help improve partner engagement and the use of innovation/research outputs by partners? Please specify other response. h_rec_uptake.Contains(96) TEXT h_rec_uptake_other Do you have any other comments or suggestions related to your experience with HESN/DIL? SINGLE-SELECT h_has_comment 01 Yes (please specify) 00 No Do you have any other comments or suggestions related to your experience with HESN/DIL? Please specify any other comments. h_has_comment==1 TEXT h_comment_specify I. FOLLOW-UP $familiar E.187 | PERFORMANCE EVALUATION OF HESN USAID.GOV I. FOLLOW-UP 25 / 27 In order to learn more about the utility of the HESN mechanism/DIL and research/innovation outputs, we would like to conduct short in- person or phone interviews with a subsample of online survey respondents. Would you be willing to participate in a short follow-up interview? SINGLE-SELECT i_follow_part 01 Yes 00 No STATIC TEXT $follow Please provide the below information: STATIC TEXT Thank you for your time! Please click on the box below to record the survey start time. DATE: CURRENT TIME i_end_time STATIC TEXT Please click on the blue box to proceed to the next section. First name: $follow TEXT i_first_name Last name: $follow TEXT i_last_name Email: $follow TEXT i_email Phone number (with country code): $follow TEXT i_phone Current organization: $follow TEXT i_organ Current position: $follow TEXT i_position APPENDIX A — CATEGORIES E.188 | PERFORMANCE EVALUATION OF HESN USAID.GOV APPENDIX A — CATEGORIES 26 / 27 [1] d_project_object:What were/are the main objectives of these grants/research projects? Categories: 1:Prototyping, 2:Fabrication, 3:Pilot testing, 4:Early stage market assessment, 5:Dissemination, 6:Scale-up, 7:Inputs for country development strategy or programdesign, 8:Collect data (programmonitoring data, demographic, geographic, sectoral, etc.), 9:Impact evaluati on of a program, 10:Performance evaluation of a program, 11:Fill gaps in data, 12:Capacity building of USAIDMission/Operating unit, 13:Cap acity building of partners, 14:Policy development, 96:Other (please specify), 99:Don't know, 98:Choose not to answer [2] g_project_add_obj:What are themain objectives of the additional collaborations beyond these grants/research projects? Categories: 1:Prototyping, 2:Fabrication, 3:Pilot testing, 4:Early stage market assessment, 5:Dissemination, 6:Scale-up, 7:Inputs for country development strategy or programdesign, 8:Collect data (programmonitoring data, demographic, geographic, sectoral, etc.), 9:Impact evaluati on of a program, 10:Performance evaluation of a program, 11:Fill gaps in data, 12:Capacity building of USAIDMission/Operating unit, 13:Cap acity building of partners, 14:Policy development, 96:Other (please specify), 99:Don't know, 98:Choose not to answer LEGEND E.189 | PERFORMANCE EVALUATION OF HESN USAID.GOV LEGEND 27 / 27 Legend and structure of information in this file This page has been left blank for double-sided copying. USAID MISSIONS AND OPERATING UNITS USAID-funded HESN Key Informant Interview Protocol March 4, 2020 Audrey Moore, Edith Felix, and Josh Meuth Alldredge Submitted to: U.S. Global Development Lab Ronald Reagan Building and International Trade Center 1300 Pennsylvania Ave NW Washington, DC 20004 Attn: Karen Fowle M&E Specialist Contract number: AID-OAA-A-16-00025 Submitted by: Mathematica International Division 1100 First Street, NE, 12th Floor Washington, DC 20002-4221 Telephone: (202) 484-9220 Facsimile: (202) 863-1763 E.191 | PERFORMANCE EVALUATION OF HESN USAID.GOV INTRODUCTION & CONSENT Hello! My name is __________ and I work with Mathematica. We are collecting information about the USAID Higher Education Solutions Network (HESN) program. HESN is a cooperative agreement between USAID and eight university-based development labs (e.g. MIT – International Development Innovation Network (IDIN), MIT – Comprehensive Initiative on Technology Evaluation (CITE), and AidData). The eight labs use HESN funding from USAID Missions/Operating Units to incubate, test, and accelerate solutions to development challenges. I would like to talk to you about your experience working on projects associated with HESN. The interview should take about 60-90 minutes. Our interview will focus on the use and utility of the HESN program and outputs/results from HESN-funded activities, aspects that facilitated or hindered implementation and use of research results, the lessons learned, and the types of changes that could improve similar work in the future. Any information provided by you will be kept strictly confidential by the people conducting this study. The information you provide will be used for the purposes of this evaluation only. The final report that summarizes this research may contain quotes from this interview, but we will not attribute them to you. Your participation is voluntary, and you may choose not to answer any of the questions we will ask during the interview. In other words, you have the choice not to participate and there will be no consequences for declining to participate or asking to end the interview early. If you have questions, concerns or complaints about the study or your rights as a participant you can contact Danice Guzman from Notre Dame at dbrown16@nd.edu or Karen Fowle from USAID at kfowle@usaid.gov. If you have any questions, feel free to ask at any time. Do you agree to participate in this interview? Finally, we would like to record our discussion so that I can focus on the discussion while speaking with you and refer to your exact comments using the recording later. The recordings will be kept in a secure, locked file and will only be heard by the researchers involved in this study. Before we start, I would like to remind you that you need to turn your cellphone off while we are talking, as that could interfere with the recording. Do we have your consent to record this interview? E.192 | PERFORMANCE EVALUATION OF HESN USAID.GOV INTRODUCTION Thank you again for participating in this interview. Let’s start by talking about the HESN program and your involvement with HESN-supported activities. 1. How long have you been with [ENTER ORGANIZATION NAME] and what is your position? 2. Tell me about your relationship to HESN, specifically [ENTER NAME OF RELEVANT HESN LAB]. 3. What was [USAID MISSION or OU] role [or its interactions] with the HESN program and LAB NAME? Use and utility of HESN Now we would like to talk about how your organization became involved with HESN/LAB NAME and the perceived benefits and challenges associated with HESN/LAB NAME activities. 4. What were USAID’s initial goals in working with the HESN program/LAB NAME? a. PROBE: What activity(ies) did you complete (or support) under HESN/LAB NAME? b. PROBE: To what extent did the HESN/LAB NAME activities support your Mission/Unit’s objectives/goals? Why or why not? c. PROBE: What has been the greatest benefit of working with HESN/LAB NAME? d. PROBE: What has been the greatest challenge of working with HESN/LAB NAME? e. PROBE: To what extent did the work under HESN/LAB NAME help create new (or strengthen existing) partnerships to improve research, innovation, and knowledge in international development? How? f. PROBE: To what extent do you think HESN-funded activities improved the use of research/innovation results and outputs across participating organizations? Please provide a specific example of how HESN-funded outputs/results were used by your Mission/OU? g. To what extent did things such as contractual or reporting requirements, funding sources, setting up partnerships or the timing of the buy-ins facilitate (or hinder) implementation of HESN-funded activities and the use of their results/outputs? 1. What about support/buy-in from other key stakeholders? h. To what extent did you see differences in the utility/or uptake of outputs/results based on whether the activity was a core (funded through the Global Development Lab) or buy-in (funded through a USAID Mission or other Operating Unit) activity? 1. What differences did you notice? 2. If there were differences, why do you think the differences existed? (please ask for an example). E.193 | PERFORMANCE EVALUATION OF HESN USAID.GOV 5. What do you see as the most important result (or achievement) of your organization’s work with HESN/LAB NAME? a. PROBE: To what extent do you think HESN/LAB NAME-supported activities can contribute to policy changes? Why or why not? Do you have any examples of this from your work with HESN/LAB NAME? b. PROBE: What do you see as the main benefits of HESN/LAB NAME to your partners? The development community? Policy and other decision-makers? Challenges? 6. How have you applied learnings from HESN outputs/results to your programming, or decision making (if at all)? a. PROBE: To what extent were the results of the HESN work relevant to your Mission/Operating Unit’s overall objectives? Planning processes? b. PROBE: To what extent was the timing of the HESN activity(ies) important to your Mission/Operating Unit’s objectives? To policy or planning processes at USAID? Why or why not? c. PROBE: To what extent have other partners used the results/outputs of the HESN￾supported activity(ies) to inform planning processes? Related work? Why or why not? Engagement in international development We understand that one of the goals of HESN was to facilitate increased engagement of universities, researchers, and other stakeholders in international development, in poverty alleviation, in positive social change. 7. To what extent has HESN contributed to changes at HEIs or in HEI networks that increase their engagement in international development, in poverty alleviation, in positive social change. Conclusion We are almost finished with the interview and I appreciate the time you have given me today. I have just a couple more questions to wrap up our discussion. 8. If you were to go back and have an opportunity to “redesign” HESN, what (if any) changes would you make? Why? 9. If you were to summarize your lessons or “take-aways” from having participated in HESN activities, what two lessons could you share with us? 10. Thinking ahead to the future, what recommendations might you offer the donor community, for example USAID or similar, when designing these types of research mechanisms? 11. Do you have anything else you would like to share? E.194 | PERFORMANCE EVALUATION OF HESN USAID.GOV Those are all the questions that I have for you today. Do you have any questions or concerns you would like to raise? THANK YOU SO MUCH FOR YOUR TIME! E.195 | PERFORMANCE EVALUATION OF HESN USAID.GOV This page has been left blank for double-sided copying. Anchor HESN LABS USAID-funded HESN Key Informant Interview Protocol March, 4, 2020 Audrey Moore, Edith Felix, and Josh Meuth Alldredge Submitted to: U.S. Global Development Lab Ronald Reagan Building and International Trade Center 1300 Pennsylvania Ave NW Washington, DC 20004 Attn: Karen Fowle M&E Specialist Contract number: AID-OAA-A-16-00025 Submitted by: Mathematica International Division 1100 First Street, NE, 12th Floor Washington, DC 20002-4221 Telephone: (202) 484-9220 Facsimile: (202) 863-1763 E.197 | PERFORMANCE EVALUATION OF HESN USAID.GOV INTRODUCTION & CONSENT Hello! My name is __________ and I work with Mathematica. We are collecting information about the USAID Higher Education Solutions Network (HESN) program. HESN is a cooperative agreement between USAID and eight university-based development labs (e.g. MIT – International Development Innovation Network (IDIN), MIT – Comprehensive Initiative on Technology Evaluation (CITE), and AidData). The eight labs use HESN funding from USAID Missions/Operating Units to incubate, test, and accelerate solutions to development challenges. I would like to talk to you about your experience working on projects associated with HESN. The interview should take about 60-90 minutes. Our interview will focus on the use and utility of the HESN program and outputs/results from HESN-funded activities, aspects that facilitated or hindered implementation and use of research results, the lessons learned, and the types of changes that could improve similar work in the future. Any information provided by you will be kept strictly confidential by the people conducting this study. The information you provide will be used for the purposes of this evaluation only. The final report that summarizes this research may contain quotes from this interview, but we will not attribute them to you. Your participation is voluntary, and you may choose not to answer any of the questions we will ask during the interview. In other words, you have the choice not to participate and there will be no consequences for declining to participate or asking to end the interview early. If you have questions, concerns or complaints about the study or your rights as a participant you can contact Danice Guzman from Notre Dame at dbrown16@nd.edu or Karen Fowle from USAID at kfowle@usaid.gov. If you have any questions, feel free to ask at any time. Do you agree to participate in this interview? Finally, we would like to record our discussion so that I can focus on the discussion while speaking with you and refer to your exact comments using the recording later. The recordings will be kept in a secure, locked file and will only be heard by the researchers involved in this study. Before we start, I would like to remind you that you need to turn your cellphone off while we are talking, as that could interfere with the recording. Do we have your consent to record this interview? E.198 | PERFORMANCE EVALUATION OF HESN USAID.GOV INTRODUCTION Thank you again for participating in this interview. Let’s start by talking about the HESN program and your involvement with HESN-supported activities. 1. How long have you been with [ENTER ORGANIZATION NAME] and what is your position? 2. Tell me about your engagement with HESN. a. What was your organization’s role [or its interactions] with HESN b. How did you become a part of HESN? c. What activities were completed under HESN? Use and utility of HESN Now we would like to talk about how your organization became involved with HESN and the perceived benefits and challenges associated with HESN activities. 3. What were your initial goals in working with the HESN program? a. PROBE: What activity(ies) did your organization complete (or support) under HESN? b. PROBE: To what extent did the HESN activities support your organizations objectives/goals? Why or why not? c. PROBE: What has been the greatest benefit of working with HESN? d. PROBE: What has been the greatest challenge of working with HESN? e. PROBE: To what extent did the work under HESN help create new (or strengthen existing) partnerships to improve research, innovation, and knowledge in international development? How? f. PROBE: To what extent do you think HESN-funded activities improved the use of research/innovation results and outputs across participating organizations? Please provide a specific example of how HESN-funded outputs/results were used by your organization? 4. To what extent did things such as contractual or reporting requirements, funding sources, setting up partnerships or the timing of the buy-ins facilitate (or hinder) implementation of HESN-funded activities and the use of their results/outputs? a. What about support from [Donor, if respondent is Lab, or HEI, if respondent is USAID]? b. What about support/buy-in from other key stakeholders? 5. To what extent did you see differences in the utility/or uptake of outputs/results based on whether the activity was a core (funded through the Global Development Lab) or buy-in (funded through a USAID Mission or other Operating Unit) activity? a. What differences did you notice? b. If there were differences, why do you think the differences existed? (please ask for an example). E.199 | PERFORMANCE EVALUATION OF HESN USAID.GOV 6. What do you see as the most important result (or achievement) of your organization’s work with HESN? a. PROBE: To what extent do you think HESN/LAB NAME-supported activities can contribute to policy changes? Why or why not? Do you have any examples of this from your work with HESN/LAB NAME? b. PROBE: What do you see as the main benefits of HESN/LAB NAME to your partners? The development community? Policy and other decision-makers? Challenges? 7. To what extent have other partners used the results/outputs of the HESN-supported activity(ies) to inform planning processes? Related work? Why or why not? 8. When you think of the overall HESN activity(ies) and the network of partners that supported the work, what do you think is the greatest strength of the partnerships? a. What aspects of the HESN activity(ies) facilitated (or hindered) the creation of strong partnerships? Why? 9. How useful do you think the HESN-funded outputs/results were to USAID Operating units, Missions, policymakers, and or other key stakeholders? Why? 10. When you think of the overall HESN-funded activity(ies) and the network of partners that supported the work, what do you think is the greatest strength of the partnerships? a. How has(have) the HESN-funded activity(ies) raised awareness on how your sector or research area contributes to development? Poverty alleviation? b. What aspects of the HESN-funded activity(ies) facilitated (or hindered) the creation of strong partnerships? Why? Engagement in international development We understand that one of the goals of HESN was to facilitate increased engagement of universities, researchers, and other stakeholders in international development, in poverty alleviation, in positive social change. 11. To what extent has HESN contributed to changes at HEIs or in HEI networks that increase their engagement in international development, in poverty alleviation, in positive social change. a. PROBE: New or expanded development related classes? b. PROBE: New or expanded internship or fellowship opportunities for students? c. PROBE: New or expanded degree programs? d. PROBE: Increased capacity of faculty/students to conduct research in international settings? e. PROBE: Development professionals proficient in data management and use because of HESN Lab-funded technical assistance? f. PROBE: Increased participation in hubs, summits, conferences? E.200 | PERFORMANCE EVALUATION OF HESN USAID.GOV g. PROBE: How HESN Labs use research in other work? h. PROBE: New opportunities for consulting in international development? Conclusion We are almost finished with the interview and I appreciate the time you have given me today. I have just a couple more questions to wrap up our discussion. 12. Now that HESN is ending, to what extent are you seeking internal or external funding to continue these research activities? a. PROBE: From whom? b. PROBE: For which activities [including research, teaching, innovations]? 13. If you were to go back and have an opportunity to “redesign” HESN, what (if any) changes would you make? Why? 14. If you were to summarize your lessons or “take-aways” from having participated in HESN activities, what two lessons could you share with us? 15. Thinking ahead to the future, what recommendations might you offer the donor community, for example USAID or similar, when designing these types of research mechanisms? 16. Do you have anything else you would like to share? Those are all the questions that I have for you today. Do you have any questions or concerns you would like to raise? THANK YOU SO MUCH FOR YOUR TIME! E.201 | PERFORMANCE EVALUATION OF HESN USAID.GOV This page has been left blank for double-sided copying. Anchor HESN PARTNERS (I.E. HEI, NGO) USAID-funded HESN Key Informant Interview Protocol March, 4, 2020 Audrey Moore, Edith Felix, and Josh Meuth Alldredge Submitted to: U.S. Global Development Lab Ronald Reagan Building and International Trade Center 1300 Pennsylvania Ave NW Washington, DC 20004 Attn: Karen Fowle M&E Specialist Contract number: AID-OAA-A-16-00025 Submitted by: Mathematica International Division 1100 First Street, NE, 12th Floor Washington, DC 20002-4221 Telephone: (202) 484-9220 Facsimile: (202) 863-1763 E.203 | PERFORMANCE EVALUATION OF HESN USAID.GOV INTRODUCTION & CONSENT Hello! My name is __________ and I work with Mathematica. We are collecting information about the USAID Higher Education Solutions Network (HESN) program. HESN is a cooperative agreement between USAID and eight university-based development labs (e.g. MIT – International Development Innovation Network (IDIN), MIT – Comprehensive Initiative on Technology Evaluation (CITE), and AidData). The eight labs use HESN funding from USAID Missions/Operating Units to incubate, test, and accelerate solutions to development challenges. I would like to talk to you about your experience working on projects associated with HESN. The interview should take about 60-90 minutes. Our interview will focus on the use and utility of the HESN program and outputs/results from HESN-funded activities, aspects that facilitated or hindered implementation and use of research results, the lessons learned, and the types of changes that could improve similar work in the future. Any information provided by you will be kept strictly confidential by the people conducting this study. The information you provide will be used for the purposes of this evaluation only. The final report that summarizes this research may contain quotes from this interview, but we will not attribute them to you. Your participation is voluntary, and you may choose not to answer any of the questions we will ask during the interview. In other words, you have the choice not to participate and there will be no consequences for declining to participate or asking to end the interview early. If you have questions, concerns or complaints about the study or your rights as a participant you can contact Danice Guzman from Notre Dame at dbrown16@nd.edu or Karen Fowle from USAID at kfowle@usaid.gov. If you have any questions, feel free to ask at any time. Do you agree to participate in this interview? Finally, we would like to record our discussion so that I can focus on the discussion while speaking with you and refer to your exact comments using the recording later. The recordings will be kept in a secure, locked file and will only be heard by the researchers involved in this study. Before we start, I would like to remind you that you need to turn your cellphone off while we are talking, as that could interfere with the recording. Do we have your consent to record this interview? E.204 | PERFORMANCE EVALUATION OF HESN USAID.GOV INTRODUCTION Thank you again for participating in this interview. Let’s start by talking about the HESN program and your involvement with HESN-supported activities. 1. How long have you been with [ENTER ORGANIZATION NAME] and what is your position? 2. Tell me about your relationship to HESN, specifically [ENTER NAME OF RELEVANT HESN LAB]. 3. What was your organization’s role [or its interactions] with the HESN program and [LAB NAME] (if you know of any)? Use and utility of HESN Now we would like to talk about how your organization became involved with [HESN/LAB NAME] and the perceived benefits and challenges associated with [HESN/LAB NAME] activities. 4. What were your organizations initial goals in working with [HESN /LAB NAME]? a. PROBE: What activity(ies) did your organization complete (or support) under [HESN/LAB NAME]? b. PROBE: To what extent did the [HESN/LAB NAME] activities support your organizations objectives/goals? Why or why not? c. PROBE: What has been the greatest benefit of working with [HESN/LAB NAME]? d. PROBE: What has been the greatest challenge of working with [HESN/LAB NAME]? e. PROBE: To what extent did the work under [HESN/LAB NAME] help create new (or strengthen existing) partnerships to improve research, innovation, and knowledge in international development? How? f. PROBE: To what extent do you think HESN-funded activities improved the use of research/innovation results and outputs across participating organizations? Please provide a specific example of how HESN-funded outputs/results were used by your organization? 5. What do you see as the most important result (or achievement) of your organization’s work with [HESN/LAB NAME]? a. PROBE: To what extent do you think [HESN/LAB NAME]-supported activities can contribute to policy changes? Why or why not? Do you have any examples of this from your work with [HESN/LAB NAME]? b. PROBE: What do you see as the main benefits of [HESN/LAB NAME] to your partners? The development community? Policy and other decision-makers? Challenges? 6. How useful do you think the HESN-funded outputs/results were to USAID Operating units, Missions, policymakers, and or other key stakeholders? Why? E.205 | PERFORMANCE EVALUATION OF HESN USAID.GOV 7. When you think of the overall HESN-funded activity(ies) and the network of partners that supported the work, what do you think is the greatest strength of the partnerships? a. How has(have) the HESN-funded activity(ies) raised awareness on how your sector or research area contributes to development? Poverty alleviation? b. What aspects of the HESN-funded activity(ies) facilitated (or hindered) the creation of strong partnerships? Why? Engagement in international development We understand that one of the goals of HESN was to facilitate increased engagement of universities, researchers, and other stakeholders in international development, in poverty alleviation, in positive social change. 8. To what extent has HESN contributed to changes at HEIs or in HEI networks that increase their engagement in international development, in poverty alleviation, in positive social change. a. PROBE: New or expanded development related classes? b. PROBE: New or expanded internship or fellowship opportunities for students? c. PROBE: New or expanded degree programs? d. PROBE: Increased capacity of faculty/students to conduct research in international settings? e. PROBE: Development professionals proficient in data management and use because of HESN Lab-funded technical assistance? f. PROBE: Increased participation in hubs, summits, conferences? g. PROBE: How HESN Labs use research in other work? h. PROBE: New opportunities for consulting in international development? Conclusion We are almost finished with the interview and I appreciate the time you have given me today. I have just a couple more questions to wrap up our discussion. 9. If you were to go back and have an opportunity to “redesign” HESN, what (if any) changes would you make? Why? 10. If you were to summarize your lessons or “take-aways” from having participated in HESN activities, what two lessons could you share with us? 11. Thinking ahead to the future, what recommendations might you offer the donor community, for example USAID or similar, when designing these types of research mechanisms? 12. Do you have anything else you would like to share? Those are all the questions that I have for you today. Do you have any questions or concerns you would like to raise? E.206 | PERFORMANCE EVALUATION OF HESN USAID.GOV THANK YOU SO MUCH FOR YOUR TIME! E.207 | PERFORMANCE EVALUATION OF HESN USAID.GOV This page has been left blank for double-sided copying. RESEARCHERS AND STUDENTS USAID-funded HESN Key Informant Interview Protocol March 4, 2020 Audrey Moore, Edith Felix, and Josh Meuth Alldredge Submitted to: U.S. Global Development Lab Ronald Reagan Building and International Trade Center 1300 Pennsylvania Ave NW Washington, DC 20004 Attn: Karen Fowle M&E Specialist Contract number: AID-OAA-A-16-00025 Submitted by: Mathematica International Division 1100 First Street, NE, 12th Floor Washington, DC 20002-4221 Telephone: (202) 484-9220 Facsimile: (202) 863-1763 E.209 | PERFORMANCE EVALUATION OF HESN USAID.GOV INTRODUCTION & CONSENT Hello! My name is __________ and I work with Mathematica. We are collecting information about the USAID Higher Education Solutions Network (HESN) program. HESN is a cooperative agreement between USAID and eight university-based development labs (e.g. MIT – International Development Innovation Network (IDIN), MIT – Comprehensive Initiative on Technology Evaluation (CITE), and AidData). The eight labs use HESN funding from USAID Missions/Operating Units to incubate, test, and accelerate solutions to development challenges I would like to talk to you about your experience working on projects associated with HESN. The interview should take about 60-90 minutes. Our interview will focus on the use and utility of the HESN program and outputs/results from HESN-funded activities, aspects that facilitated or hindered implementation and use of outputs/results, the lessons learned, and the types of changes that could improve similar work in the future. Any information provided by you will be kept strictly confidential by the people conducting this study. The information you provide will be used for the purposes of this evaluation only. The final report that summarizes this research may contain quotes from this interview, but we will not attribute them to you. Your participation is voluntary, and you may choose not to answer any of the questions we will ask during the interview. In other words, you have the choice not to participate or to end your participation at any point and there will be no consequences for declining to participate. If you have questions, concerns or complaints about the study or your rights as a participant you can contact Danice Guzman from Notre Dame at dbrown16@nd.edu or Karen Fowle from USAID at kfowle@usaid.gov. If you have any questions, feel free to ask at any time. Do you agree to participate in this interview? Finally, we would like to record our discussion so that I can focus on the discussion while speaking with you and refer to your exact comments using the recording later. The recordings will be kept in a secure, locked file and will only be heard by the researchers involved in this study. Before we start, I would like to remind you that you need to turn your cellphone off while we are talking, as that could interfere with the recording. Do we have your consent to record this interview? E.210 | PERFORMANCE EVALUATION OF HESN USAID.GOV INTRODUCTION Thank you again for participating in this interview. Let’s start by talking about HESN and your involvement with LAB NAME. 1. Tell me how you learned about the HESN-supported LAB NAME [FACILIATOR: verify that interviewee knows what HESN is relative to the specific HESN Lab]? 2. What was your role [or interactions] with LAB NAME? 3. How long did you work [or have you been working] on LAB NAME-related activities? Use and utility of HESN Now we would like to talk about the benefits and challenges associated with the activity(ies) that you engaged in as it relates to LAB NAME. 4. What were the main goals of the LAB NAME-related activity(ies) that you were engaged with through your work? a. PROBE: Tell me about the activity(ies). b. PROBE: To what extent were the outputs/results from these activity(ies) used by others? How? c. PROBE: To what extent do you think that activities such as those funded under LAB NAME contribute to the following areas? [FOR EACH PROBE, PLEASE ASK WHY OR WHY NOT]? 1. How research activities or programs are designed or implemented)? 2. How government agencies engage in decision-making or policy development in your sector? 3. How the results of research contribute to community development? community networks? social cohesion? 4. How research results contribute to poverty alleviation? social change? d. What has been the greatest benefit to working with LAB NAME? Greatest challenge? e. PROBE: Have you worked on any other research projects other than the HESN-funded activity? f. If yes, what other research activity(ies) have you worked on? g. With whom/organization? h. How did your other experience(s) compare with your work under the HESN-funded activity? 5. How was the LAB NAME work that you completed relevant to the needs of your department? University? Development community? E.211 | PERFORMANCE EVALUATION OF HESN USAID.GOV a. PROBE: To what extent was the research focus important to the department? University? Community? b. PROBE: To what extent did the research address development issues in your region or country? c. PROBE: To what extent did the research activity come at a time that supported your professional needs? Specific needs of the University? d. PROBE: What did you learn from the work completed through LAB NAME? How did you apply the learning to your personal goals? e. PROBE: If the work you completed was not relevant, why not? 6. When you think about the LAB NAME activity(ies) that you worked on, what do you think was the most important contribution that came out of the work? a. PROBE: How have the resulting products contributed to your work or career? Department/university? International development field? Local communities/beneficiaries? b. PROBE: How has(have) the LAB NAME activity(ies) raised awareness on how your sector or research area contributes to development? Poverty alleviation? c. PROBE: What do you see as the strengths of having a research program such as HESN? d. PROBE: What are the weaknesses or challenges of having a research/innovation and cooperation program such as HESN? Engagement in international development We understand that one of the goals of HESN was to facilitate increased engagement of universities, researchers, innovators, and other stakeholders in international development. 7. To what extent has HESN-funded work carried out by LAB NAME contributed to changes at your institution (or your institution’s networks)? Tell me about those changes. [FACILITATOR: REQUEST EXAMPLES FOR A-K BELOW] a. PROBE: New or expanded development related classes? b. PROBE: New or expanded degree programs? c. PROBE: Innovation-related programs (e.g. accelerators, incubators, maker spaces, tech transfer programs)? d. PROBE: New or expanded internship or fellowship opportunities for students? e. PROBE: New partnerships/expanded networks? f. PROBE: Increased capacity of faculty/students to conduct research in international settings? g. PROBE: Development professionals proficient in data management and use because of HESN Lab-funded technical assistance? h. PROBE: Increased participation in hubs, summits, conferences? i. PROBE: New opportunities for consulting in international development? j. PROBE: How universities manages grants? E.212 | PERFORMANCE EVALUATION OF HESN USAID.GOV Conclusion We are almost finished with the interview and I appreciate the time you have given me today. I have just a couple more questions to wrap up our discussion. 8. To what extent did/are you seeking additional funding to continue the LAB NAME-related activity(ies)? a. PROBE: From whom? b. PROBE: For which components/activities? 9. If you were to summarize your lessons or “take aways” from having participated in HESN￾funded LAB NAME activities, what two lessons could you share with us? 10. Thinking ahead to the future, what recommendations might you offer the donor community when designing programs to support research/innovation and collaboration? 11. Do you have anything else you would like to share? Those are all the questions that I have for you today. Do you have any questions or concerns you would like to raise? THANK YOU SO MUCH FOR YOUR TIME! E.213 | PERFORMANCE EVALUATION OF HESN USAID.GOV This page has been left blank for double-sided copying. INNOVATORS USAID-funded HESN Key Informant Interview Protocol March 4, 2020 Audrey Moore, Edith Felix, and Josh Meuth Alldredge Submitted to: U.S. Global Development Lab Ronald Reagan Building and International Trade Center 1300 Pennsylvania Ave NW Washington, DC 20004 Attn: Karen Fowle M&E Specialist Contract number: AID-OAA-A-16-00025 Submitted by: Mathematica International Division 1100 First Street, NE, 12th Floor Washington, DC 20002-4221 Telephone: (202) 484-9220 Facsimile: (202) 863-1763 E.215 | PERFORMANCE EVALUATION OF HESN USAID.GOV INTRODUCTION & CONSENT Hello! My name is __________ and I work with Mathematica. We are collecting information about the USAID Higher Education Solutions Network (HESN) program. HESN is a cooperative agreement between USAID and eight university-based development labs (e.g. MIT – International Development Innovation Network (IDIN), MIT – Comprehensive Initiative on Technology Evaluation (CITE), and AidData). The eight labs use HESN funding from USAID Missions/Operating Units to incubate, test, and accelerate solutions to development challenges. I would like to talk to you about your experience working on activities associated with HESN. The interview should take about 60-90 minutes. Our interview will focus on the use and utility of the HESN program and outputs/results from HESN-funded activities, aspects that facilitated or hindered implementation and use of outputs/ results, the lessons learned, and the types of changes that could improve similar work in the future. Any information provided by you will be kept strictly confidential by the people conducting this study. The information you provide will be used for the purposes of this evaluation only. The final report that summarizes this research may contain quotes from this interview, but we will not attribute them to you. Your participation is voluntary, and you may choose not to answer any of the questions we will ask during the interview. In other words, you have the choice not to participate or to end your participation at any point and there will be no consequences for declining to participate. If you have questions, concerns or complaints about the study or your rights as a participant you can contact Danice Guzman from Notre Dame at dbrown16@nd.edu or Karen Fowle from USAID at kfowle@usaid.gov. If you have any questions, feel free to ask at any time. Do you agree to participate in this interview? Finally, we would like to record our discussion so that I can focus on the discussion while speaking with you and refer to your exact comments using the recording later. The recordings will be kept in a secure, locked file and will only be heard by the researchers involved in this study. Before we start, I would like to remind you that you need to turn your cellphone off while we are talking, as that could interfere with the recording. Do we have your consent to record this interview? E.216 | PERFORMANCE EVALUATION OF HESN USAID.GOV INTRODUCTION Thank you again for participating in this interview. Let’s start by talking about HESN and your involvement with LAB NAME. 1. Tell me how you learned about the HESN-support lab: LAB NAME [FACILIATOR: verify that interviewee knows what HESN is relative to the specific HESN Lab]? 2. What was your role [or interactions] with LAB NAME? 3. How long did you work [or have you been working] on LAB NAME-related activities? Use and utility of HESN Now we would like to talk about the benefits and challenges associated with the activity(ies) that you engaged in as it relates to LAB NAME. 4. What were the main goals of the LAB NAME-related activity(ies) that you were engaged with through your work? a. PROBE: Tell me about the activity(ies). b. PROBE: To what extent were the outputs/results from your activity(ies) or innovation used by others? How? c. PROBE: To what extent do you think that activities such as those funded under LAB NAME can contribute to the following? [PLEASE PROBE WHY OR WHY NOT]?] 1. How programs or innovations are designed or implemented? 2. How governments engage around decision-making or policy-development related to [TOPIC OF INNOVATION] 3. How the innovation contributes to community development? community networks? social cohesion? 4. Other areas such as poverty alleviation, social change? d. PROBE: What has been the greatest benefit to working with LAB NAME? Greatest challenge? e. PROBE: Have you had an opportunity to design and develop other innovative projects or activities? f. If yes, who (or what organization) did you work with on the innovation? g. How does your experience with this other organization(s) compare to the HESN innovation experience? 5. How was the LAB NAME work that you completed relevant to the needs of your sector of work? Development community? a. To what extent did the funding for the innovation come at a time that allowed you to make a contribution to your community? Sector? International development? Other? E.217 | PERFORMANCE EVALUATION OF HESN USAID.GOV b. How was the innovation relevant to your interests? c. PROBE: What did you learn from the work completed with support from LAB NAME? How did you apply the learning to your personal goals? d. PROBE: If the work you completed was not relevant, why not? 6. When you think about the LAB NAME activity(ies) that you worked on, what do you think was the most important contribution that came out of the work? a. PROBE: How have the resulting products contributed to your work or career? International development field? Local communities/beneficiaries? b. PROBE: How has(have) the LAB NAME activity(ies) raised awareness on how your sector of work contributes to development? Poverty alleviation? Engagement in international development We understand that one of the goals of HESN was to facilitate increased engagement of innovators, researchers, universities, and other stakeholders in international development. 7. To what extent has HESN-funded work carried out by LAB NAME contributed to changes at your organization or community? Tell me about those changes. [FACILITATOR: REQUEST EXAMPLES FOR A-I BELOW] a. PROBE: Increased participation in accelerators, incubators, maker spaces, technology transfer programs, summits, conferences? b. PROBE: New partnerships/expanded networks? c. PROBE: New opportunities for consulting in innovation design and development? IF INNOVATOR IS FROM AN HEI: a. PROBE: New or expanded development related classes? b. PROBE: New or expanded degree programs?PROBE: New or expanded internship or fellowship opportunities for students? c. PROBE: Increased capacity of faculty/students to design and development innovations? d. PROBE: Development professionals proficient in innovation design and development and use because of LAB NAME-funded technical assistance? Conclusion We are almost finished with the interview and I appreciate the time you have given me today. I have just a couple more questions to wrap up our discussion. 8. To what extent did/are you seeking additional funding to continue the LAB NAME-related activity(ies)? a. PROBE: From whom? b. PROBE: For which components/activities? E.218 | PERFORMANCE EVALUATION OF HESN USAID.GOV 9. If you were to summarize your lessons or “take aways” from having participated in HESN￾funded LAB NAME activities, what two lessons could you share with us? 10. Thinking ahead to the future, what recommendations might you offer the donor community when designing programs to support innovation and collaboration? 11. Do you have anything else you would like to share? Those are all the questions that I have for you today. Do you have any questions or concerns you would like to raise? THANK YOU SO MUCH FOR YOUR TIME! E.219 | PERFORMANCE EVALUATION OF HESN USAID.GOV This page has been left blank for double-sided copying. USAID-funded HESN Key Informant Interview Protocol POLICYMAKERS Audrey Moore, Edith Felix, and Josh Meuth Alldredge E.221 | PERFORMANCE EVALUATION OF HESN USAID.GOV Introduction & consent Hello! My name is __________ and I work with Mathematica. We are collecting information about the USAID Higher Education Solutions Network (HESN) program. HESN is a cooperative agreement between USAID and eight university-based development labs (e.g. AidData, the Resilient Action Network (RAN), MIT – International Development Innovation Network (IDIN), and MIT – Comprehensive Initiative on Technology Evaluation (CITE)). The eight labs use HESN funding from USAID Missions/Operating Units to incubate, test, and accelerate solutions to development challenges. I would like to talk to you about your experience working on projects associated with HESN. The interview should take about 60 minutes. Our interview will focus on the use and utility of the HESN program and outputs/results from HESN-funded activities, aspects that facilitated or hindered implementation and use of research results, the lessons learned, and the types of changes that could improve similar work in the future. Any information provided by you will be kept strictly confidential by the people conducting this study. The information you provide will be used for the purposes of this evaluation only. The final report that summarizes this research may contain quotes from this interview, but we will not attribute them to you. Your participation is voluntary, and you may choose not to answer any of the questions we will ask during the interview. In other words, you have the choice not to participate and there will be no consequences for declining to participate or asking to end the interview early. If you have questions, concerns or complaints about the study or your rights as a participant you can contact Danice Guzman from Notre Dame at dbrown16@nd.edu or Karen Fowle from USAID at kfowle@usaid.gov. If you have any questions, feel free to ask at any time. Do you agree to participate in this interview? Finally, we would like to record our discussion so that I can focus on the discussion while speaking with you and refer to your exact comments using the recording later. The recordings will be kept in a secure, locked file and will only be heard by the researchers involved in this study. Before we start, I would like to remind you that you need to turn your cellphone off while we are talking, as that could interfere with the recording. Do we have your consent to record this interview? E.222 | PERFORMANCE EVALUATION OF HESN USAID.GOV Introduction Thank you again for participating in this interview. Let’s start by talking about the HESN program and your involvement with [LAB NAME], which was supported with HESN funding. 1. How long have you been with [GOVERNMENT MINISTRY NAME] and what is your position? 2. Tell me about your relationship to [LAB NAME]. 3. What was your ministry’s role, or its interactions, with the [LAB NAME] (if you know of any)? Use and utility of HESN Now we would like to talk about your ministry’s goals with [PROJECT NAME/LAB NAME] and the perceived benefits and challenges associated with [PROJECT NAME/LAB NAME] activities. 4. What were your ministry’s initial goals in working with [HESN /LAB NAME]? a. PROBE: What activity(ies) did your ministry complete (or support) under [PROJECT NAME/LAB NAME]? Interviewer: be prepared to remind the interviewee of what the project did. b. PROBE: To what extent did the [PROJECT NAME/LAB NAME] activities support your ministry’s objectives/goals? Why or why not? a. PROBE: What has been the greatest benefit of working with [LAB NAME]? b. PROBE: What has been the greatest challenge of working with [LAB NAME]? c. PROBE: To what extent did the work under [PROJECT NAME] help create new (or strengthen existing) partnerships to improve research, innovation, and knowledge in international development? For example, a new working relationship with [LAB NAME] or universities? d. PROBE: Did the work under [PROJECT NAME/LAB NAME] increase the capacity of your ministry or of local partners to use data or research in decisionmaking? e. PROBE: To what extent do you think [LAB NAME] activities improved the use of research/innovation results and outputs across organizations in [COUNTRY]? Please provide a specific example of how [LAB NAME] outputs/results were used by your ministry or other organizations. 5. How was the work that you completed with [LAB NAME] relevant to the needs of your sector of work? Development community? E.223 | PERFORMANCE EVALUATION OF HESN USAID.GOV 6. Have you worked on any other research projects other than the LAB NAME activity with [LAB NAME]? a. PROBE: If yes, what other research activity(ies) have you worked on? b. PROBE: With whom/ which organization? c. PROBE: How did your other experience(s) compare with your work under the LAB NAME activity? 7. What do you see as the most important result (or achievement) of your ministry’s work with [PROJECT NAME/LAB NAME]? f. PROBE: To what extent do you think [PROJECT NAME/LAB NAME]-supported activities can contribute to policy changes? Why or why not? Do you have any examples of this from your work with [PROJECT NAME/LAB NAME]? g. PROBE: What do you see as the main benefits of [PROJECT NAME/LAB NAME] to your partners? The development community? Policy and other decision-makers? Challenges? 8. When you think of the overall LAB NAME activity(ies) and the network of partners that supported the work, what do you think is the greatest strength of the partnerships? a. PROBE: How has(have) the LAB NAME activity(ies) raised awareness on how your sector of work contributes to development? Poverty alleviation? b. PROBE: What aspects of the LAB NAME activity(ies) facilitated (or hindered) the creation of strong partnerships? Why? Conclusion We are almost finished with the interview and I appreciate the time you have given me today. I have just a couple more questions to wrap up our discussion. 9. If you were to summarize your lessons or “take-aways” from having participated in HESN￾supported activities, what two lessons could you share with us? 10. Thinking ahead to the future, what recommendations might you offer the donor community, for example USAID or similar, when designing programs like HESN that focus on supporting solutions to development challenges? 11. Do you have anything else you would like to share? Those are all the questions that I have for you today. Do you have any questions or concerns you would like to raise? THANK YOU SO MUCH FOR YOUR TIME! E.224 | PERFORMANCE EVALUATION OF HESN USAID.GOV ANNEX F EVALUATION TEAM This page has been left blank for double-sided copying. F.3 | PERFORMANCE EVALUATION OF HESN USAID.GOV ANNEX F. EVALUATION TEAM Audrey Moore, Ph.D. (University of Minnesota) is a senior researcher in Mathematica’s international division and served as the project director for the HESN evaluation. She is an educational economist with over 20 years of experience working in research, evaluation, educational policy analysis, and analysis of educational development projects focusing on children and youth. Dr. Moore has worked in over 12 countries across Latin America, Africa, the Middle East, and the U.S. and has extensive experience conducting cost-benefit and cost￾effectiveness analysis of interventions; evaluating workforce development projects for youth; measuring early grade reading in primary school; and evaluating health and education interventions. Edith Felix, M.P.I.A. (University of California, San Diego) is a research analyst in Mathematica's international division and served as a technical staff member on the HESN evaluation. She has a decade of experience working in the design, implementation, and analysis of rigorous evaluations of programs covering a variety of sectors including education, community-driven development, agriculture, and water and sanitation. She has worked across Latin America, Southeast Asia, and Africa. Ms. Felix has extensive experience in survey design; quantitative and qualitative data collection; and analysis of survey and administrative data. Josh Meuth Alldredge, M.G.P.S. (University of Texas at Austin) is a research analyst in Mathematica’s international division and served as a technical staff member on the HESN evaluation. He specializes in synthesizing evidence for meta-evaluations, collecting and conducting analysis on qualitative data, and visualizing data for policymaker audiences. Mr. Meuth Alldredge brings 8 years of experience in international research and evaluation, and his principal area of work is Latin America and the Caribbean, where he supports evaluations of education programs and labor policies. F.4 | PERFORMANCE EVALUATION OF HESN This page has been left blank for double-sided copying. USAID.GOV F.5 | PERFORMANCE EVALUATION OF HESN USAID.GOV Audrey-Marie Schuh Moore Senior Researcher Experience Mathematica Project Director, the U.S. Agency for International Development (USAID) YouthPower2 Learning and Evaluation (YP2LE) (2019-). Lead management of YP2LE activities for Mathematica, including the primary liaison to the prime contractor, Making Cents International. Serve as a task lead for the positive youth development research agenda activities. Provide technical oversight for the design, implementation, and analysis of youth studies and assessments, including impact evaluations and rapid assessments. Project Director, USAID, Higher Education Network Solutions (HESN) Performance Evaluation (2019-). Lead management of the USAID Higher Education Solutions Network final performance evaluation including work plan development, data collection, analysis, and writing reports. Conduct political economy analysis of HESN and how grants contributed to innovative technological solutions in low and middle income countries. Provide overall leadership for the evaluation team, including meeting with the client and key stakeholders as required. Senior Researcher, Millennium Challenge Corporation, Cabo Verde WASH Evaluation (2017-). Support the project director to develop the evaluation design, data collection instruments, support data collection, and analysis. Provide training to data collectors in Cabo Verde and support teams as required. Lead the qualitative study. Project Director, Millennium Challenge Corporation Cabo Verde Land Management Information Systems Evaluation (2016-). Lead management of the Cabo Verde evaluation including work plan development, data collection, analysis, and writing reports. Provide overall leadership for the evaluation team, including meeting with the client and key stakeholders as required. Project Director, Millennium Challenge Corporation Guatemala Threshold Education (2016-). Lead management of the Guatemala Threshold Education Program evaluations including an RCT evaluation of examining the impacts of teacher professional development, pedagogical support, learning circles, and parent networks on student learning in middle school education. Lead development of the evaluation design, work plan development, oversee data collection, analysis, and writing reports. Provide leadership the Technical Vocational Education System performance evaluation team, including developing protocols, observing instruction, and conducting interviews with instructors, businesses and students to understand the contributions of the program to student transition to employment. Lead political economy analysis of institutional strengthening evaluation to understand the facilitators and barriers to improved teacher recruitment systems and financing of middle and secondary education. Education 2003 Ph.D., Educational Policy and Administration, University of Minnesota 1998 M.A., Public Affairs, Humphrey Institute of Public Affairs, University of Minnesota 1991 B.A., International Relations, University of Minnesota Positions 2015- Mathematica 2009- Adjunct Faculty, Georgetown University 2011-2015 FHI 360 2012-2015 Director, Research and Evaluation 2011-2012 Director, Education Quality Improvement Program, and Director, Monitoring, Research, and Evaluation 2003-2010 Deputy Director, Education Quality Improvement Program, AED 1996-2003 University of Minnesota 2002-2003 Coordinator, Continuing Professional Studies 2002-2003 Teaching Specialist 1998-2002 Administrative Fellow 1996-1997 Research Assistant 1998-2002 Research Consultant, Independent Contractor F.6 | PERFORMANCE EVALUATION OF HESN USAID.GOV Senior Researcher, Millennium Challenge Corporation, Morocco Secondary Education Activity (2016-). Serve as the Teacher Professional Development Specialist on the project. Provide leadership to the design and implementation of the performance evaluation, develop and implement socio-emotional assessments, classroom observations, and student learning assessments. Support the design of relevant qualitative and quantitative instruments related to measuring teacher outcomes. Support impact evaluation team. Project Director, Department of Labor/ILAB (2019-2020). Led management of the Interim Evaluation of the Northern Triangle Labor Market Information systems evaluation including work plan development, data collection, analysis, and writing reports. Provided overall leadership for the evaluation team, including meeting with the client and key stakeholders as required. Project Director, USAID, YouthPower (2018-2020). Served as the primary contact for the prime. Provided technical guidance and expertise on youth programming. Oversaw data analysis and writing for youth assessments conducted by the YouthPower team in Liberia and the Kyrgyz Republic. Co-led a task order review that examined the common elements of USAID youth programming and the barriers and facilitators to implementation. Project Director, Millennium Challenge Corporation Urban Water Supply and Sanitation and Hygiene Evaluation (2018-2019). Led management of the Mozambique WASH performance evaluation including development of the evaluation design, work plan development, overseeing data collection, analysis, and writing reports. Senior Researcher, Millennium Challenge Corporation, Georgia II General Education Project (2018). Trained project staff on the Stallings Classroom Observation Instrument so they could train enumerators in Georgia and collect classroom level data related to teacher time on task for the evaluation. Georgetown University; Washington, DC As Adjunct Faculty (2009-) in the Georgetown Foreign Policy School, co-teach a course on Practical Evaluation. Course includes an introduction to the basic concepts of evaluation, evaluation design, methods, analysis, and implementation. FHI 360; Washington, DC As Director, Research and Evaluation (2012-2015), led the monitoring and evaluation technical assistance team in the Global Learning Group and served as principal investigator on key impact evaluations. Provided technical assistance and training to field offices in design and implementation of data collection protocols, monitoring and evaluation, economic analysis, education finance, and strategic planning. Trained data collectors in the use of classroom observation protocols, survey design, interview and focus group techniques, and data collection surveys. Provided leadership in technical research areas including school effectiveness, opportunity to learn/early grade reading assessments, evaluation complementary education, cost-benefit and cost-effectiveness analysis, econometrics, and secondary education. Participated in the following projects: Home Office (HO) Monitoring and Evaluation (M&E) Technical Adviser, Haiti An n aprann li ak ekri (2015). Provided home office technical support for this five-year, reading program in Haiti funded by USAID. Served as the HO technical support person and helped design a quasi-experimental baseline study that includes measuring time on task, health indicators, the literacy environment, and reading levels in primary schools across Haiti. Wrote M&E plan, designed data collection instruments, and supported M&E start-up with the local M&E subcontractor. Home Office Monitoring and Evaluation Technical Adviser, Programa de Desarollo Educativo de Guinea Ecuatorial Equatorial Guinea (2014-2015). Provided home office technical support for this education program. Served as the HO technical support person and helped design a quasi-experimental baseline study that included measuring time on task and reading levels in primary schools across Equatorial Guinea. Worked with the FHI 360 biostatistics department to define a weighted sample, trained data collectors on data collection instruments, helped design the data collection instruments, and supported the F.7 | PERFORMANCE EVALUATION OF HESN USAID.GOV data analysis and writing process. The program, funded by Hess Oil company, seeks to improve students’ learning in the early grades and work readiness in secondary education. Coprincipal Investigator, UNICEF Kenya Impact Evaluation of Educational Interventions in the Arid and Semi-Arid Lands (ASAL) Region (2013-2015). Oversaw various evaluation components for this two-year non-experimental design study for the Education Office at UNICEF Kenya. Managed the evaluation process, managed client and subcontractor relations, assessed the feasibility of access to potential sites to participate based on security issues in the region, helped develop baseline and instruments, redesigned endline instruments and logistics for the endline data collection, supported impact analyses and cost benefit analysis, and participated in the process study. This evaluation measures the impact of four sets of interventions—water, sanitation, and hygiene; solar power; child-friendly schools; and communities for development—on enrollment and retention of students in the ASAL region. It was initially as a quasi￾experimental design with assignment to six clusters including a control group, but due to significant contamination of the sample, the study was changed to an observational study. Coprincipal Investigator, International Initiative for Impact Evaluation Impact Evaluation: Assessing the Impact of School Autonomy: Lessons from the Closure of the Community Managed Schools Program (EDUCO) Program (2013-2015). Led technical activities for this two-year quasi￾experimental design study. Worked closely with FHI 360 biostatistics group to design the mixed-methods, hierarchical growth model study; designed data collection instruments and participated in pilot-testing the instruments in El Salvador; trained the local subcontractor and data collectors on the instruments; and worked with team to conduct impact analysis (quantitative and qualitative), including a cost-effectiveness analysis of the policy change. Conducted three workshops for the Ministry of Education on designing and managing evaluations. This evaluation examines the impact of a school autonomy policy change that took place in 2010, effectively ending the EDUCO program in El Salvador. Findings show that although school￾based management supports reduced teacher absenteeism and indirectly supported learning, centralized control provides teachers with more stable employment and is more cost-effective. Research Analyst, Time-on-Task Data Analysis for the World Bank (2013). Provided support for analyzing data used for a large time-on-task study across Latin America. Completed data analysis and writing on reports that had already been started in four countries. Translated reports into Spanish and Portuguese. Completed an analysis and report of interviews with school supervisors, which complemented the classroom observation data. Trained more than 1,200 data collectors across Latin America to collect data for this study. Director, Education Quality Improvement Program and Director, Monitoring, Research, and Evaluation (2011-2012), led all program activities including management, administrative oversight, and technical leadership. Continued the following project started at AED: Coprincipal Investigator, Opportunity to Learn (2008-2012). Co-led this four-year study across Guatemala, Honduras, Nepal, Ethiopia, and Mozambique, as part of the Education Quality Improvement Program 2 (EQUIP2) program. Served as lead investigator for the Guatemala, Honduras, and Mozambique studies, designed instruments for data collection, conducted proportional sampling plan for the studies, trained interns to collect data in the field, analyzed the data, and wrote reports. Coauthored the book that consolidates data from across the case studies. The study measured teacher and student time on task and correlated the results to early-grade reading outcomes. AED; Washington, DC Deputy Director, Education Quality Improvement Program (2003-2010). Helped direct this $9.5 million program. Oversaw all project activities related to research and evaluation design, and implementation activities. Led a series of research studies on improving school effectiveness in Ethiopia, Guatemala, Honduras, Mozambique, and Nepal. Trained more than 1,200 data collectors in classroom observation, assessment, and focus group techniques in Brazil, Colombia, El Salvador, Honduras, Jamaica, Mexico, and Peru. Conducted a series of evaluation workshops to build the capacity of the Ministry of Education in El F.8 | PERFORMANCE EVALUATION OF HESN USAID.GOV Salvador. Served as the HO project director and M&E technical lead for projects in Jordan, Ghana, Guatemala, and the Republic of Georgia. Participated in the following projects: Research Consultant, The World Bank (2009-2010). Conducted a five-day workshop for educators from the Latin America and the Caribbean region on how to use and implement the Stallings Observation Instrument in classrooms. Conducted a five-day workshop for 25 educators and Secretariat of Education staff in Brazil. Coprincipal Investigator, The Cost-Effectiveness of Complementary Education (2004-2008). Co-led this four-year study that examined the cost and cost-effectiveness of complementary programs across 10 countries, as part of the EQUIP2 program. Designed instruments for data collection, analyzed secondary data, analyzed the data, and wrote reports. Coauthored the book that consolidates data from across the case studies. Home Office Project Director/Monitoring and Evaluation Technical Lead, Jordan Education Reform for the Knowledge Economy (ERfKE) (2004-2008). Served as the primary counterpart for the in￾country project director and assisted in all aspects of project management and implementation. Served as the main M&E technical lead, developed the performance monitoring plan, reported to USAID, designed three formative evaluations, and supported data analysis of learning data collected for the national school assessments. The USAID-funded Jordan ERfKE program sought to expand access to early childhood education, enhance the curriculum for the new Management and Information Stream (MIS) in grades 11 and 12 to better prepare youth for the workforce, and strengthen the Ministry of Education’s efforts to advance education in the South of Jordan through targeting resources to enhance connectivity. The formative evaluations used mixed methods to measure the impact of the new curriculum on MIS student knowledge compared to the teacher-based delivery method, assess changes in the professional development skills of teachers, and support the project to make changes to annual workplans based on findings from the evaluations. A third qualitative evaluation measured changes in the education system to determine movement toward sustainability. Home Office Project Director/Monitoring and Evaluation Technical Lead, Ghana Basic Education Comprehensive Assessment System (2003-2007). Served as the primary counterpart for the in-country project director and assisted in all aspects of project management and implementation. Served as the main M&E technical lead, developed the performance monitoring plan, reported to USAID, and supported data analysis of learning data collected for the national school assessments. Supported the data analysis and writing of a student motivation study. The purpose of the project was to work with the Ministry of Education to design and implement a national and school-level assessment system, including the development of item banks and multiple tests. Monitoring and Evaluation Technical Lead, Georgia Education Sector Reform Project (2004-2006). Served as the main M&E technical lead, developed the performance monitoring plan, reported to USAID, and the development of outcome measures to understand project contributions to the reform of the educational resource centers and their role in leading education reform in the post-Rose Revolution period. Contributed to the development of accreditation tools to assess general, vocational and higher education institutions, and delivered capacity-building workshops to Georgian educators, accreditation specialists, and Ministry personnel on accreditation tools, administrative financing, and assessments. University of Minnesota; Minneapolis, MN As Coordinator, Continuing Professional Studies (2002-2003) in the College of Education and Human Development, worked with College deans, department chairs, faculty, and internal University and external constituents to develop and market continuing professional education programs for educators and related professionals throughout the state, nation, and targeted international locations. Conducted market analysis, program and budget development and management, and marketing and evaluation for the college’s Summer Session. Served as a member of the Dean’s Administrative Council. Oversaw the development, implementation, and assessment of professional development programs and workshops held through the Urban Principal Leadership Program. F.9 | PERFORMANCE EVALUATION OF HESN USAID.GOV As Teaching Specialist (2002-2003) in the College of Education and Human Development, taught an undergraduate class as part of the College’s Leadership Minor for undergraduate students. As Administrative Fellow (1998-2002) in the Office of Continuing Professional Education, assisted in planning professional development for Minnesota administrators. Designed, implemented, and analyzed needs assessments and surveys for the director of Continuing Professional Development. Participated in the following projects: Research Collaborator, University of Minnesota (2001). Evaluated UNICEF Program evaluations written in Portuguese and Spanish. Provided direct feedback about the extent to which UNICEF country programs were meeting UNICEF educational objectives. As Research Assistant (1997) for the State and Local Policy Program in the Hubert Humphrey Institute of Public Affairs, conducted 25 focus groups and more than 30 interviews with educators in Minnesota. Transcribed, analyzed, and assisted in writing final report. As Research Assistant (1996-1997) for the University of Minnesota Extension, Gaylord, used research and writing skills to design surveys, conducted a pre- and post-assessment of the students in the program, and assisted in writing grant proposals and intermediate program evaluations for program sponsors. Independent Contractor; Minneapolis, MN As Independent Contractor (1998-2002), participated in the following projects: Research Consultant, Academy for Educational Development (AED) (2002). Conducted quantitative data analysis for the Uganda Data Analysis project and provided a written report to AED on Uganda educational data. Research Consultant, Miske Witt and Associates (2001). Evaluated educational documents related to educational programs in Ghana, Uganda, and Kenya. Research Consultant, Miske Witt and Associates (2000). Conducted quantitative, economic data analysis for the World Bank. Assisted in writing the final evaluation report on the Female Stipend program (FSSAP) for the World Bank The data analysis was used to evaluate FSSAP in Bangladesh. The final report included interpretation of quantitative findings, recommendations for the program, and recommendations for improving data collection for future evaluations. Research Consultant, UNICEF Oman (1999). Assisted the Omani Ministry of Education in the development of a five-year education plan and framework. Conducted a workshop on planning and development of educational plans. Researcher, Economics and Sociology Department, University of Sao Paulo (1999). Designed and implemented independent research and evaluation of an alternative education program for adolescents in rural areas. Research focused on determining opportunity costs of participation in the alternative education program and included site visits of five of nine locations, including visits to family farms, and interviews with facilitators, parents, students, and coordination team members. Researcher, Secretariat of Education, Lavras (1998). Designed and conducted a research project to determine the feasibility of developing school, business, and community partnerships within the public education system of Lavras in Minas Gerais Brazil. Conducted market analysis through interviews with educators and businesses. Assisted in developing a pilot project between a middle school and a business in Lavras. Papers and Publications Moore, Audrey-marie Schuh, Amber Gove, and Karen Tietjen. “Great Expectations: A Framework for Assessing and Understanding Key Factors Affecting Student Learning of Foundational Reading Skills.” New Directions for Child and Adolescent Development, vol. 155, March 2017, pp. 13-30. F.10 | PERFORMANCE EVALUATION OF HESN USAID.GOV Moore, Audrey-marie Schuh, Joseph DeStefano, and Elizabeth Adelman. “Opportunity to Learn: A High Impact Strategy for Improving Educational Outcomes in Developing Countries.” Education Quality Improvement Program 2. Washington, DC: FHI 360, 2012. Moore, Audrey-marie Schuh, Annie Smiley, Joseph DeStefano, and Elizabeth Adelman. “The Right to Quality Education: How Use of Time and the Language of Instruction Impact the Rights of Students.” World Studies of Education Journal, vol. 13, no. 2, 2012, pp. 67-86. Moore, Audrey-marie Schuh, Joseph DeStefano, and Elizabeth Adelman. “Time Misspent, Opportunities Lost: Use of Time in School and Learning.” In Policy Debates in Comparative, International and Development Education, edited by John N. Hawkins, and W. James Jacob. New York: Palgrave MacMillan, 2011. DeStefano, Joseph, and Audrey-marie Schuh Moore. “The Roles of Non-State Providers in Ten Complementary Education Programmes.” Development in Practice, vol. 20, nos. 4-5, 2010, pp. 511-526. Moore, Audrey-marie Schuh, Joseph DeStefano, and John Gillies. “Creating an Opportunity to Learn Through Complementary Models of Education.” Journal of Education for International Development, vol. 3, 2009. Moore, Audrey-marie Schuh, Joseph DeStefano, Arushi Terway, and David Balwanz. “Secondary Education: Action to Fill the Teacher Gap.” Education Quality Improvement Program. Washington, DC: Academy for Educational Development, 2008. Moore, Audrey-marie Schuh, Joseph DeStefano, Arushi Terway, and David Balwanz. “Expanding Secondary Education for Sub-Saharan Africa: Where Are the Teachers?” Education Quality Improvement Program 2. Washington, DC: Academy for Educational Development, 2008. DeStefano, Joseph, Audrey-marie Schuh Moore, David Balwanz, and Ash Hartwell. “Meeting EFA: Reaching the Underserved Through Complementary Models of Effective Schooling—Compendium Book.” Education Quality Improvement Program 2. Washington, DC: Academy for Educational Development, 2007. Moore, Audrey-marie Schuh. “Meeting EFA: Guatemala Education Quality Improvement Program 2 Education Quality Improvement Program 2—PRONADE.” Education Quality Improvement Program 2. Washington, DC: Academy for Educational Development, 2007. DeStefano, Joseph, Ash Hartwell, Audrey-marie Schuh Moore, and Jane Benbow. “A Cross-National Cost￾Benefit Analysis of Complementary (Out-of-School) Programs.” Journal of International Cooperation in Education, vol. 9, no. 1, 2006, pp. 71-88. DeStefano, Joseph, Audrey-marie Schuh Moore, David Balwanz, and Ash Hartwell. “Meeting EFA: Reaching the Underserved Through Complementary Models of Effective Schooling.” Education Quality Improvement Program 2. Washington, DC: Academy for Educational Development, 2006. DeStefano Joseph, Ash Hartwell, Audrey-marie Schuh Moore and Jane Benbow. “Meeting EFA: Cost￾Effectiveness of Complementary Approaches.” Education Quality Improvement Program 2. Washington, DC: Academy for Educational Development, 2005. Moore, Audrey-marie Schuh. “Meeting EFA: Honduras—Educatodos.” Education Quality Improvement Program 2. Washington, DC: Academy for Educational Development, 2005. Chabott, Collette, and Audrey-marie Schuh Moore. “Meeting EFA: Bangladesh—BRAC.” Washington, DC: Academy for Educational Development, 2005. Moore, Audrey-marie Schuh. “Book Review of Trabalho docente: formação e identidades.” Education Review, June 2002 (online journal). Moore, Audrey-marie Schuh, and David W. Chapman. “The Delivery of Development Assistance in a Loosely Coupled System: A Case Study.” International Journal of Educational Development, 2000. Memberships and Professional Activities 2010- Reviewer Comparative Education Review and International Journal for Educational Development 2003- Member, American Evaluation Association F.11 | PERFORMANCE EVALUATION OF HESN USAID.GOV 1998- Member, Comparative International Education Society 2014-2015 Cochair, Monitoring and Evaluation Working Group, Basic Education Coalition 2000-2003 Member, Association for Research on Nonprofit Organizations and Voluntary Action Languages Portuguese (native), Spanish (fluent), French (basic proficiency) This page has been left blank for double-sided copying. F.13 | PERFORMANCE EVALUATION OF HESN USAID.GOV Edith Felix Research Analyst Selected Experience Mathematica Research Analyst, Advancing Local Dairy Development in Nigeria (ALDDN) Program Evaluation (2020–). Design and conduct an impact and process evaluation of the ALDDN program funded by the Bill & Melinda Gates Foundation. The program aims to address key constraints limiting dairy development in Nigeria by improving dairy farmers’ access to fodder, water, and animal health services. ALDDN also aims to improve nutritional outcomes and women’s empowerment. The program also involves advocacy for a favorable policy environment that encourages local sourcing of milk. Research Analyst, An Outcome Evaluation of Generation Programs in Five Countries (2020–). Design and conduct an outcome evaluation of Generation programs in Brazil, India, Kenya, the United Kingdom, and the United States. Generation seeks to improve employment outcomes for disconnected youth through a seven-component methodology that includes intensive workforce training boot camps tailored to employers' needs, as well as post-training job placement support. Research Analyst, Expanding the Reach of Impact Evaluation (ERIE): Higher Education Solutions Network (HESN) Evaluation (2019–). Support mixed-methods ex post process evaluation of HESN, a cooperative agreement among USAID and seven top universities designed to channel the ingenuity of university students, researchers, and faculty toward global development through a set of core funds and buy-in activities from USAID missions and bureaus. The evaluation includes a document review, an online survey, key informant interviews with relevant project stakeholders, and qualitative fieldwork in three case study countries: Colombia, Ghana, and Uganda. Research Analyst, Cabo Verde II Compact Water, Sanitation, and Hygiene (WASH) Project Evaluation (2016–). Assist with evaluation activities, including evaluation design, data collection, and data analysis. This is a rigorous, mixed-methods performance evaluation of a $39.7 million WASH project funded by MCC and aimed at increasing access, use, and quality of WASH services in Cabo Verde. Project Manager, Cabo Verde II Compact WASH Project Evaluation (2016–). Assist with internal project management activities, monitoring spending, and preparing monthly client progress reports. Research Analyst, Rapid Assessment of USAID YouthPower Activities (2019–2020). Supported mixed￾methods rapid assessment of activities conducted under the USAID YouthPower project, including education activities implemented in Brazil, El Salvador, Honduras, Kenya, Mozambique, Nicaragua, and Tanzania. The assessment included a document review, an online survey, and key informant interviews with relevant project stakeholders. Research Analyst, Rapid Feedback Monitoring, Evaluation, Research, and Learning (RF MERL): Integrated Digital Adherence Technology Initiative (IDAT) India Pilot (2019–2020). Support Education 2011 Master in Pacific International Affairs, School of International Relations and Pacific Studies, University of California, San Diego 2009 B.A., Economics and Chinese, University of Notre Dame Positions 2015- Mathematica 2013-2015 Project Coordinator, Innovations for Poverty Action 2012-2013 Freelance Research Analyst, Euromonitor International 2010-2012 Innovations for Poverty Action 2011-2012 Project Associate 2010Project Associate Intern 2009 Consular Section Intern, U.S. Embassy Beijing 2008-2009 Assistant, University of Notre Dame Center for Social Concerns F.14 | PERFORMANCE EVALUATION OF HESN USAID.GOV quantitative analysis component of a USAID-funded descriptive study intended to contribute to a broader understanding of technology to engage tuberculosis patients in India. Research Analyst, Latin America and the Caribbean Reads: Education and Security White Paper (2017–2020). This comprehensive literature review, funded by USAID, aims to synthesize the global evidence demonstrating the relationship between education programs with violence prevention and crime reduction, including through correlated factors such as skills development (cognitive and noncognitive) and other risk and protective factors and behaviors. Education programs included in the study range from early childhood development, primary, secondary, and postsecondary school, and workforce development training to education support services, decentralization, and institutional strengthening. Research Analyst, Latin America and the Caribbean Reads: Workforce Development (WFD) Learning Agenda (2015–2020). Assist with the design, analysis, and reporting for a study on WFD programming in Central America. This study, funded by the U.S. Agency for International Development (USAID,) aims to synthesize and report on USAID investments in WFD programs in Central America and reports annually about six active WFD projects in the region, including their work to assess the local labor market, build the capacity of local organizations to conduct WFD training, conduct policy change or institutional reform, roll out training activities, and help beneficiaries complete training and obtain employment. The reports will provide information for USAID reporting to Congress regarding key program achievements of the Central America Strategy. In addition, the community of practice formed through this project aims to facilitate communication and the exchange of learnings among the existing WFD projects, as well as identify best practices to inform design and implementation of future WFD programs. Certified Reviewer, What Works Clearinghouse (2015–2019). Review, assess, and rate the rigor of research evaluating the effectiveness of various educational interventions across different topic areas. Reviewed studies that lead to the development of practice guides, which summarize for educators the types of interventions that have proven effectiveness. This project is sponsored by the U.S. Department of Education, Institute of Education Sciences and provides education practitioners and policymakers with timely assessments of the quality of research evidence on educational interventions. Research Analyst, Guatemala Threshold Education Project (GTEP) Evaluation (2016–2017). Assisted with evaluation design activities. The GTEP was funded by the Millennium Challenge Corporation (MCC) and included various interventions to strengthen lower secondary education. The GTEP was evaluated through a mixed-method evaluation design, including a randomized controlled trial, a performance evaluation, and an implementation study. Research Analyst, Latin America and the Caribbean Reads: Cost analysis of Leer Juntos Aprender Juntos (2015–2017). Worked with the lead researcher on the cost analysis of the Leer Juntos, Aprender Juntos (Read Together, Learn Together) early-grade reading intervention in Peru and Guatemala. Research Analyst, Latin America and the Caribbean Reads: Secondary analysis of end-of-grade test score data from Honduras (2015–2017). Helped conduct the secondary data analysis of end-of-grade test score data from Honduras. The goal of the analysis was to generate policy-relevant information about the assessment activities undertaken in Honduras in recent years. Research Analyst, Impact Evaluation Services in Tanzania (2015–2016). Performed quality checks on raw data submitted by the local data collection firm. Worked with the lead researcher on the analysis and reporting for a small pilot study within the larger study investigating the implementation and benefits of an effort to provide solar photovoltaic systems to households and small businesses. The larger project, funded by MCC, is aimed at improving living standards and reducing poverty in Tanzania by improving energy￾related infrastructure. Innovations for Poverty Action (IPA); Manila, Philippines As Project Coordinator (2013-2015), managed the implementation in 198 municipalities of a randomized control trail evaluation of Kapitbisig Laban Sa Kahirapan-Comprehensive and Integrated Delivery of Social Services (Kalahi-CIDSS), a community-driven development project implemented by the Philippine F.15 | PERFORMANCE EVALUATION OF HESN USAID.GOV government and funded by MCC. Monitored implementation of Kalahi-CIDSS to ensure compliance with the evaluation design. Prepared for and oversaw longitudinal data collection activities including refining instruments and managing the data collection firm hired by IPA and data cleaning process. Analyzed, wrote, and presented impact evaluation findings with the IPA research team. Coordinated with and kept the IPA research team and stakeholders informed about project-related activities. Supervised and managed IPA project staff including field staff and a research analyst. Prepared and tracked project budgets and invoices. Participated in the following projects: Project Coordinator, Community-Driven Development Project: Kapitbisig Laban Sa Kahirapan￾Comprehensive and Integrated Delivery of Social Services Project, Baseline Study (2013). Analyzed baseline survey data on 5,940 households and 80 villages with Stata. Co-wrote the impact evaluation baseline report. Prepared final baseline survey data sets and analysis files for public use. Presented baseline findings to government stakeholders. Interim Study (2013-2015): Designed new household and village survey modules and carried out pilot fieldwork to refine village behavioral assessment instruments. Oversaw data collection firm’s implementation of 2,400 household surveys, 80 village surveys, and behavioral assessments. Conducted impact analysis with Stata using baseline and interim survey data. Co-wrote the Impact Evaluation of the Kalahi-CIDSS: Interim Report. Prepared final interim survey data sets and analysis files for public use. Euromonitor International; Mexico City, Mexico As Freelance Research Analyst (2012-2013), implemented market research methodology in the analysis of key consumer industries in Mexico. Developed research plans and strategies in collaboration with research analysts in the United States and Mexico. Conducted secondary and primary trade interviews with relevant industry players in Mexico. Used Excel to calculate market sizes and shares. Produced comprehensive written and statistical market analyses including final industry reports for clients. Innovations for Poverty Action; Mexico City, Mexico As Project Associate (2011-2012), managed three research projects for an IPA Mexico principal investigator. Developed presentations in Spanish for Mexican government officials. Produced progress and financial reports for project funders. Assisted in the negotiation of the launch of public policy evaluation projects with government agencies and banks. Wrote and edited grants to acquire funding for randomized control trial projects in Mexico. Cleaned and analyzed large panel data sets in Stata. Provided research assistance for the preparation of economic research working papers. Presented information on program evaluation methods during IPA outreach workshops. Participated in the following projects: Project Associate, Mexico Bank Research Project: Information Disclosure, Advice, and Social Comparisons (2011-2012). Evaluated the impact of an information disclosure randomized experiment on credit card holder behavior. Used Stata to analyze a 20-month panel data set of 180,000 credit card holders. Collaborated with investigators in writing and editing a working paper. Project Associate, Agriculture Technology Adoption Project: Technological Guarantee Program (2011-2012). Collaborated with the Mexican government’s Trust Funds for Rural Development in the design and implementation of an agriculture technology pilot program for Mexican farmers. Prepared nine grant budgets and proposals, including a big 3ie award-winning grant to acquire funding for implementing a rigorous evaluation of the program. Codeveloped, implemented, and analyzed the results of a baseline agriculture survey. As Project Associate Intern, Microcredit for Women in Mexico (2010), monitored a randomized control trial evaluation of microcredit for women in 250 communities. Managed the client household verification process carried out in the field by bank coordinators. Designed PowerPoint presentation, handouts, and interactive skits to train bank managers, administrators, coordinators, and loan officers. Executed daylong training on proper project implementation for all staff members in three bank branches. Participated in negotiations of the structure of loan officer incentives system with project partners. Organized and updated Excel database of microfinance client data. F.16 | PERFORMANCE EVALUATION OF HESN USAID.GOV U.S. Embassy Beijing; Beijing, China As Consular Section Intern (2009), produced weekly summary reports of important technology information for visa officers. Communicated with public through the U.S. Embassy Beijing Visa Blog. Identified visa applicant refusal trends using Excel and produced a final summary report. University of Notre Dame Center for Social Concerns; Notre Dame, IN As Assistant (2008-2009) to the assistant director of the Center for Social Concerns, organized logistics for 250-student Appalachia seminar. Compiled and catalogued literature on Appalachia region. Researched service-learning opportunities offered by top 25 service-learning institutions in the United States and created a detailed Excel spreadsheet of the data. Selected Papers and Publications Bagby, Emilie, Edith Felix, Patricia Costa, Beryl Seiler, and Nancy Murray. “Latin American and the Caribbean (LAC) Reading Evaluation Contract: Third Annual Central America Workforce Development Report.” Submitted to the U.S. Agency for International Development. Washington, DC: Mathematica, July 2020. Null, Clair, Edith Felix, Audrey Moore, Abbie Turiansky, Jane Fortson, Evan Fantozzi, Dara Bernstein, Emma Pottinger, Matthew Ribar, and Loay Hidmi. “The Water, Sanitation, and Hygiene (WASH) Project in Cabo Verde: Interim Evaluation Report.” Report submitted to the Millennium Challenge Corporation. Washington, DC: Mathematica, March 2020. Moore, Audrey-Marie, Edith Felix, and Josh Meuth Alldredge. “The Higher Education Solutions Network (*HESN) Evaluation Design Report.” Submitted to the U.S. Agency for International Development. Washington, DC: Mathematica, October 2019. Bagby, Emilie, Edith Felix, Patricia Costa, Nancy Murray, and Beryl Seiler. “Latin American and the Caribbean (LAC) Reading Evaluation Contract: Second Annual Central America Workforce Development Report.” Submitted to the U.S. Agency for International Development. Washington, DC: Mathematica, August 2019. Bagby, Emilie, Edith Felix, Patricia Costa, and Nancy Murray. “Latin American and the Caribbean (LAC) Reading Evaluation Contract: First Annual Central America Workforce Development Report.” Submitted to the U.S. Agency for International Development. Washington, DC: Mathematica Policy Research, April 2018. Felix, Edith, Sarah Liuzzi, Audrey Moore, Seth Morgan, Catalina Torrente, and Anthony DeWees. “Guatemala Threshold Program: Evaluation Design Report.” Final evaluation design report submitted to the Millennium Challenge Corporation. Washington, DC: Mathematica Policy Research, February 2018. Null, Clair, Audrey-Marie Moore, Edith Felix, and Chantal Toledo. “The Water, Sanitation, and Hygiene (WASH) Project in Cabo Verde: Evaluation Design Report.” Report submitted to the Millennium Challenge Corporation. Princeton, NJ: Mathematica Policy Research, May 2017. Vohra, Divya, Edith Felix, Duncan Chaplin, and Arif Mamun. “The Kigoma Solar Activity in Tanzania: Evaluation Findings.” Issue Brief. Washington, DC: Mathematica Policy Research, March 2017. Beatty, Amanda, Ariel BenYishay, Simona Demel, Edith Felix, Elisabeth King, Aniceto Orbeta, and Menno Pradhan. “Impact Evaluation of the Kalahi-CIDSS: Interim Report.” Manila, Philippines: Innovations for Poverty Action, 2015. Beatty, Amanda., Ariel BenYishay, Edith Felix, Elisabeth King, Allan Lalisan, Aniceto Orbeta, Menno Pradhan, and Sukhmani Sethi. “Impact Evaluation of the Kalahi-CIDSS: Baseline Report.” Manila, Philippines: Innovations for Poverty Action, 2014. F.17 | PERFORMANCE EVALUATION OF HESN USAID.GOV Selected Presentations Felix, Edith and Jansen Mayor “Community-Driven Development Projects: Assessing Quality of Participation in Decision-Making.” Presented at the Making Impact Evaluation Matter Conference, Manila, Philippines, 2014. Felix, Edith. “Gender Perspectives: Kalahi-CIDSS Impact Evaluation.” Presented at the Millennium Challenge Corporation, Millennium Challenge Account-Philippines, and Department of Social Welfare and Development Compact National M&E Conference, Cebu, Philippines, 2014. Continuing Education 2007 Language and Cultural Studies Program, Council on International Educational Exchange Study Center, East China Normal University Honors and Awards 2009 Award for Excellence in Chinese Studies, University of Notre Dame Languages Native fluency in Spanish, Intermediate Mandarin This page has been left blank for double-sided copying. F.19 | PERFORMANCE EVALUATION OF HESN USAID.GOV Josh Meuth Alldredge Research Analyst Experience Mathematica Research Analyst, U.S. Department of Labor (DOL) Bureau of International Labor Affairs (ILAB) Synthesis Review (2019-). Conduct thorough review of evaluations of DOL ILAB-funded projects to improve labor law enforcement in low- and middle-income countries. This work includes systematically applying a review rubric to ILAB studies to extract key project (and project assessment) characteristics, developing a system to aggregate and analyze outcomes reported in evaluations, and producing a report and visualizations to convey trends in ILAB project outcomes. Research Analyst, Higher Education Solutions Network (HESN) (2019-). Develop evaluation design and surveys, and conduct data collection, analysis, and reporting to assess the performance of the U.S. Agency for International Development (USAID) HESN granting mechanism over the last seven years. This project will determine how the HESN mechanism has influenced the capacities of international development research labs and how it has affected the relationships between higher education institutions, which house the labs, and USAID missions in need of research and evaluation support around the world. Research Analyst and Project Manager, Labor Market Supply and Demand in the Northern Triangle: Leveraging Data to Build an Efficient Labor Market (2019-). Evaluate the success of efforts by ILAB in DOL to improve labor market systems in El Salvador, Guatemala, and Honduras. Conducted a desk review, developed a data collection protocol, conducted qualitative data collection in Central America, analyzed qualitative data, and reported to stakeholders. Research Analyst, Latin America and the Caribbean Reads Evaluations (2018-). Conduct systematic research and writing for a white paper that synthesizes global evidence on the relationship between a variety of education interventions and security-related outcomes among youth, including violence prevention, crime reduction, and migration mitigation. Education 2018 M.G.P.S., International Development, Lyndon B. Johnson School of Public Affairs, University of Texas at Austin 2011 B.A., Politics, Whitman College Positions 2018- Mathematica 2017-2018 University of Texas at Austin 2017-2018 Graduate Research Fellow 2017-2018 Pro Bono Consultant, LBJ School of Public Affairs 2017 Corruption Coder, Government Department 2017 Data Analyst, Ubongo Learning 2014-2016 Launch High School 2014-2015 Board President 2013-2016 Board Member 2013-2016 Outreach Manager, Community Partnership for Child Development 2012-2013 Princeton in Latin America Fellow and Director of External Relations, Unidad Académica Campesina 2011-2012 Ethnographic Researcher and Analyst, Thaines & Bodah Center for Education & Development, Programa de Desenvolvimento Sustentável da Nova Esperança Research Analyst, YouthPower Cross-Sectoral Youth Assessments (CSYAs) in Liberia and the Kyrgyz Republic (2018-2019). Conducted qualitative analysis and reporting using transcripts from focus group and key informant interviews of youth, employers, business leaders, human rights advocates, health practitioners, and government policymakers. Reports from the CSYAs help USAID program and mission staff improve approaches to education, health, entrepreneurship, political engagement, employment, and migration. F.20 | PERFORMANCE EVALUATION OF HESN USAID.GOV Research Analyst, Projects with Human Dignity Foundation (2018-2019). Supported writing, mapping, and visuals development for project briefs and reports on women’s empowerment projects in India and campaigns to raise awareness about (and end the practice of) female genital mutilation in Europe. University of Texas at Austin; Austin, TX As Graduate Research Fellow (2017-2018) for the Data4Peace in Innovations for Peace and Development, led a team of undergraduate and graduate students researching the risks in the implementation of the 2016 Colombian Peace Agreement. Located and prepared subnational data on municipal fragility, illicit drug production, armed group recruitment, violence, poverty, family socialization, and other indicators. Performed analysis using Stata to understand recruitment and recidivism. Coauthored and designed briefs and reports for Colombian officials, academics, and civil society organizations to use in peace accord implementation. Disseminated new analysis through blogs, working papers, and project reports. Co-organized the 2017 Data4Peace Hackathon Conference in Bogotá, supported by USAID. Designed graphics and report formats for the conference. As Pro Bono Consultant (2017-2018) on the Beyond the Border policy research project, coordinated through the Lyndon B. Johnson School of Public Affairs, researched and wrote policy memos on the background of Central American migration, the mandates of the Federal Police, and migrants’ interactions with organized crime and law enforcement. Conducted key informant interviews with migration experts in Spanish and English, and performed subsequent qualitative analysis. Conducted informal interviews with Central American migrants in Spanish. Developed monitoring and evaluation tools to assess risks experienced by migrants, including a database of crimes against migrants using reports from hundreds of Mexican news outlets. Mapped criminal activity against migrants using geolocated data points and ArcGIS. Co-edited final client reports, co-designed presentation slides, and designed fact sheets. Elected by consultancy team to present research findings and policy recommendations in Spanish to the Federal Police commissioner in Mexico City. As Corruption Coder (2017) in the Government Department, examined reports from audits of municipal governments produced by the Honduran Tribunal Superior de Cuentas. Coded the type and gravity of possibly corrupt acts based on a typology of infractions. Ubongo Learning; Dar es Salaam, Tanzania As Data Analyst (2017), used Excel and Stata to analyze national survey data on parent behaviors, finding that exposure to Ubongo content was strongly correlated with reductions in child abuse and improvements in supportive parenting over time. Analyzed data on USAID-funded pilot of rural screening clubs, monitored fidelity of program implementation, and offered recommendations for scaling up Ubongo distribution to last￾mile villages and refugee camps. Worked with allied scholars to support monitoring, evaluation, and learning (MEL) plans and build research designs for randomized control trials and a longitudinal study. Co-wrote grant applications and reports for bilateral donors and foundations. Community Partnership for Child Development (CPCD); Colorado Springs, CO As Outreach Manager (2013-2016), developed and implemented communications strategies to increase client recruitment and donor awareness. Ensured CPCD communications were in compliance with federal performance standards and policies. Designed and held focus group discussions in Spanish to explore barriers to Head Start enrollment among Latino families. Evaluated and updated communications strategies regularly to ensure relevance and impact. Organized large events and built partnerships and referral systems with allied agencies. Designed graphics and produced targeted language for print advertising and press releases. Managed teams of outreach staff and volunteers. Unidad Académica Campesina; Carmen Pampa, Bolivia As Princeton in Latin America Fellow and Director of External Relations (2012-2013), cultivated relationships with donors, collaborators, and other institutions. Collected video for fundraising content. Coordinated events, professional visits, guest lectures, and exchange programs with universities abroad. F.21 | PERFORMANCE EVALUATION OF HESN USAID.GOV Developed project proposals for USAID’s American Schools and Hospitals Abroad program, crafted budgets, and secured grant funding. Co-founded and coordinated political awareness and discussion program for students. Thaines & Bodah Center for Education & Development, Programa de Desenvolvimento Sustentável da Nova Esperança; Brazil As Ethnographic Researcher and Analyst (2011-2012), performed ethnographic research through semi￾structured interviews, participant observation, and informal documentation in Nova Esperança to assess the challenges and assets of the community. Analyzed qualitative data with tools from development and linguistics literature. Published findings and recommendations. Produced short reports. Secured grant funding for a community-designed agriculture education program. Papers and Publications Denly, Mike, Daniela Hernández Salazar, Josh Meuth Alldredge, Alejandra Tello, and Eduardo Velazquez. “Authoritarian Reversals and Dissent: Evidence from Venezuela.” Graduate seminar paper. Austin, Texas: University of Texas at Austin, May 2018. Meuth Alldredge, Josh. “Improving Education Outcomes in LMIC Contexts: A Synthesis of Parent Behavior Change Program Evaluations.” Graduate seminar paper. Austin, Texas: University of Texas at Austin, May 2018. Tello, Alejandra, and Josh Meuth Alldredge. “Choosing Children: Why Low-Income Families Distribute Education Resources in Unequal Ways.” Graduate seminar paper. Austin, Texas: University of Texas at Austin, May 2018. Meuth Alldredge, Josh, Rafael Ch, Angelika Rettberg, María Paula Rojas, and Michael Weintraub. “Understanding New Patterns of Violence in Colombia.” Political Violence at a Glance Blog, June 2017. Meuth Alldredge, Josh. “Education Access: Planning for Opportunity in Mexican Social Interest Housing.” Graduate seminar paper. Austin, Texas: University of Texas at Austin, May 2017. Meuth Alldredge, Josh, Eliane Thaines Bodah, Brian Bodah, Alcindo Neckel, and Emanuelle Goellner. “Challenges and Perspectives of Language Education Technology in Brazil: From Confronting Native Language Loss to Implementing EFL Classes.” In Handbook of Research on Foreign Language Education in the Digital Age. Hershey, Pennsylvania: IGI Global, July 2016. Meuth Alldredge, Josh. “New Hopes in Nova Esperança: An Analysis of Social and Environmental Problems in a Northern Amazonian Indigenous Community.” In Conversas Entre Educadores: Novos Diálogos. Passo Fundo, Brazil: THAINES & BODAH Center for Education and Development, May 2012. Honors and Awards 2016-2018 Recipient, James M. and Claudia U. Richter Graduate Fellowship, Lyndon B Johnson School of Public Affairs 2014 Graduate, Leadership Now Program, Leadership Pikes Peak, Colorado Springs, Colorado 2012-2013 Recipient, Princeton in Latin America Fellowship, Bolivia Information Technology Stata, R, ArcGIS, Adobe InDesign, Adobe Photoshop, Microsoft Suite, Tableau Languages Spanish (professional fluency), Portuguese (intermediate), Swahili (beginner) This page has been left blank for double-sided copying. ANNEX G LASER BRIEF This page has been left blank for double-sided copying. Policy Brief | March 2021 EVALUATING LONG-TERM ASSISTANCE AND SERVICES FOR RESEARCH (LASER): Learning from Performance and Systems Approaches This policy brief outlines possible evaluation designs that would allow USAID to learn from—and improve—the performance of its ongoing Long-term Assistance and Services for Research (LASER) program. The brief draws insights from recent evaluations of similar USAID programs, including Higher Education Solutions Network 1.0 (HESN) and Partnerships for Enhanced Engagement in Research (PEER). Cooperative agreement overview The USAID Innovation, Technology, and Research (ITR) Hub launched LASER in 2018 to improve development outcomes through research engagement. The five-year, $70 million cooperative agreement includes: ● $20 million in core funding from ITR to researchers to explore development questions that span regions and sectors, the answers to which can benefit the broader development community; and ● Up to $50 million from USAID Missions, Bureaus, and Independent Offices (MBIOs) to partner with universities through buy-in agreements. The buy-ins focus on identifying evidence gaps, conducting research, and deploying evidence-based solutions specific to MBIO needs. Purdue University leads the LASER Partners for University-Led Solutions Engine (PULSE) consortium. The consortia are composed of dozens of universities across 24 low- and middle-income countries (LMICs) and eight universities in the US. This network boasts more than 2,300 researchers and practitioners across 56 countries. The LASER award for PULSE runs through 2023. The LASER Results Framework In devising solutions to complex development challenges, practitioners (including USAID MBIOs) require strong evidence and technical expertise. Encapsulating LASER's theory of change, USAID states that "closer collaboration between academic researchers, development practitioners, policymakers, and donors results in new research that is readily translated into useful policies, products, and practices as evidence-based solutions to development challenges." Exhibit 1. LASER Results Framework Note: IR refers to intermediate result and HEI refers to higher education institution. 2 Evaluating progress, improving performance USAID seeks to assess and improve LASER's implementation and results by commissioning a program performance evaluation. The timing, research questions, data sources, and analytical methodology for this evaluation are not yet decided. The following subsections discuss the relative advantages and disadvantages of different evaluation options. Evaluation timing options USAID is interested in understanding the LASER model's validity, its implementation experience, and the specific strategies it used to pursue, design, and implement research. Experience evaluating complex funding programs such as HESN 1.0 and PEER suggests that end-of-project evaluations provide USAID with the most comprehensive view of a program's design validity and results. These final evaluations also provide essential information that stakeholders, including MBIOs, can use to plan future funding structures. The disadvantage of final evaluations is that findings are not available along the way to improve ongoing project performance and adjust the course if needed to meet outcomes. If USAID is most interested in gathering formative data related to LASER's unique implementation strategies (e.g., Research for Development [R4D]), a midterm evaluation is more appropriate. A midterm performance evaluation would allow USAID and LASER PULSE to use findings to adapt and improve implementation processes during the final years of programming. Based on results, LASER staff can adjust their workplan, change their approaches to engaging partners and MBIOs, and work with USAID to adjust any institutional barriers as needed. Tables 1–4, at the end of this brief, highlight the four midterm and final evaluation options that provide findings related to LASER's design, strategies, interim and final results. Suggested research questions Based on LASER's results framework and USAID discussions, we proposed the following research questions to guide the performance evaluation design selection. Midterm evaluation questions: 1. How well has the blend of core and buy-in activities met the needs of USAID Missions, Bureaus, and Independent Offices (MBIOs) to date? 2. How resilient was the LASER program to unforeseen circumstances, such as the COVID￾19 pandemic? How did LASER’s design support or limit its resiliency? 3. How well has LASER achieved its planned outputs in the first three years of the program? To what extent has LASER made progress toward its intermediate results (IRs)? 4. What incentives and relationships among LASER stakeholders facilitated progress toward meeting the goals (IRs)? How? 5. What incentives and relationships among LASER stakeholders hindered the program’s progress toward its goals (IRs)? How? 6. How could relationships and incentives among LASER stakeholders be improved to facilitate progress toward program IRs? 7. How do midterm results from LASER in those areas compare to mid-project results from similar USAID funding systems such as PEER, Research Technical Assistance Center (RTAC), Higher Education for Leadership, Education, and Exchange (HELIX), and Health Research Program (HARPNET)? 8. What changes can USAID and LASER PULSE make to the LASER strategies, funding structure, or implementation process in its final two years to ensure the program meets its original IRs? End-of-project evaluation questions: 9. How have LASER's Comprehensive Success Factors Analysis (CSFA), Research for Development (R4D) convenings, and Embedded Research Translation (ERT) strategies affected MBIOs’ satisfaction and uptake of research results and academic and news media uptake of research results? (This is also a midterm evaluation question.) 3 10. How has LASER affected the engagement and growth in international development research among individual researchers? Local research institutions? Purdue University? Where has LASER made the biggest contributions? 11. How has LASER contributed to institutional strengthening of local research institutions? Researchers? USAID? 12. To what extent have other programs adopted strategies similar to CSFA and ERT as a result of LASER’s dissemination activities? 13. What was the cost-per-output under LASER, and did the project’s efficiency meet expectations? 14. How resilient was the LASER program to unforeseen circumstances, such as the COVID￾19 pandemic? How did LASER adapt to these challenging circumstances? (This is also a midterm evaluation question.) 15. How does LASER's value-for-money compare to similar cooperative agreements in terms of research and policy impacts? 16. What other factors—and questions—should USAID explore to gain insights into the best way to engage universities and MBIOs to further the international development research agenda? 17. Which aspects of USAID models to fund research best contribute to the use of research for policy and decision making in LMICs? To research uptake and use for community development? 18. Comparing university-based models such as LASER, HESN 1.0, and PEER to other implementer-based models such as RTAC, what advantages do university-based implementers provide in helping USAID meet its research goals? What are the disadvantages or weaknesses in these models? (This is also a midterm evaluation question.) 19. MBIOs’ use of research-focused cooperative agreements (HESN 1.0, LASER, PEER) varies widely. What drives the demand and use of these agreements in the research space? 20. What type of funding design or model is best suited to helping USAID reach its objectives of research uptake, evidence-based decision making, and contributions to international development? Tables 1-4 present these research questions with associated evaluation designs. Research questions may be added or removed to help USAID achieve a clear understanding of LASER's progress and challenges. Evaluation data sources Each research question and associated evaluation design requires collecting data across a variety of sources. The potential data sources for the proposed designs include: Stakeholders ● PULSE Leadership ● USAID ITR leadership and AORs ● PULSE Consortium partner staff ● USAID MBIO staff ● Researchers from the PULSE network across US and LMIC universities ● Policymakers and policy experts in LMICs where LASER has been active Document Review ● Publications produced using PULSE funding ● LASER PULSE program documentation, including indicator targets and results ● Past evaluations of similar funding programs (e.g., HESN 1.0, PEER, other non-USAID programs) ● Program documentation from ongoing, similar funding programs, such as RTAC Proposed evaluation designs LASER involves a broad set of activities, stakeholders, and beneficiaries, so no one evaluation methodology fully captures the cooperative agreement's insights. However, by collecting data on LASER, examining outcomes from similar funding models, and deploying various analytical approaches, evaluators can generate insights that respond to the research questions proposed in this brief. This section presents four evaluation options with distinct components and advantages. Options 1 and 2 are midterm 4 evaluations, whereas options 3 and 4 are final evaluations. Midterm Evaluation Option 1: Assessing LASER implementation and system dynamics. This mixed-methods performance evaluation includes two components: (1) a qualitative implementation study and (2) a rapid political economy analysis (PEA). This design answers questions related to LASER implementation and dynamics among stakeholders that can affect the program's outcomes, including its incentives and relationships (see Table 1). This evaluation focuses on providing USAID with a snapshot of current implementer performance and formative feedback to improve activities over the project's remaining life. The qualitative implementation study employs one round of key informant interviews (KIIs) and document review to understand outputs and outcomes to date. The team draws on rapid-cycle fidelity of implementation criteria (e.g., acceptability, adherence, cost) to understand LASER's progress toward its goals and objectives. The criteria guide the development of KII and document review protocols. The evaluation team maps performance based on each criterion and uses the results to help the LASER team adjust workplan activities for future years. The implementation study allows USAID to understand successful project elements, the level of uptake of research activities, support for the activities, and the effects of changes made to the program workplan. The main steps to the implementation study include: Step 1. Review the theory of change and identify the goals and objectives. Step 2. Conduct meetings with implementers to deepen understanding of the interventions. Step 3. Develop KII protocols using the fidelity of implementation criteria. Step 4. Conduct document review and finalize instrumentation, including pilot-testing. Step 5. Conduct interviews. Step 6. Code and analyze KII and administrative data. Step 7. Interpret findings. The second component of this midterm evaluation option is a rapid political economy analysis. PEA provides an inside view of the incentives and relationships at play in the LASER program. Political economy is embedded in the KII protocols through a series of questions related to the organization of power, decision making, and economic resources—areas that are the foundation of systems change. This approach looks beyond the visible or surface-level manifestations of problems to investigate "root systems" in politics, history, social organization, and formal and informal institutions that facilitate or hinder changes to systems. The analysis seeks to understand which groups those problems affect, and how various actors across the social system are incentivized to maintain (or ignore) the status quo. The PEA uses interviews and stakeholder power analysis to understand the enabling and hindering factors to uptake and use over time. It maps power dynamics among different stakeholders to help donors and implementers navigate the implementation and sustainability process. We recommend using the World Bank problem￾driven approach, or the Department for International Development (now the Foreign, Commonwealth & Development Office) Drivers of Change approach, to understand the political economy of "research uptake and use" or the success of R4D, ERT, and CSFA because these areas are the most tightly coupled to LASER's outcomes. Annex X provides more detail on PEA, but the main steps include: Step. 1. Identify the problem the evaluation will focus on trying to understand. This is often a problem where technical analysis and engagement have failed to provide operational traction. Step 2. Develop KII and document review protocols to collect data from key stakeholders and documents. The protocols should cover three dimensions: (1) relevant structural factors that 5 influence stakeholder positions; (2) existing institutions, including institutional dysfunctions that channel behavior and ongoing institutional change; and (3) stakeholder interests and constellations. To reduce costs and increase efficiency, the PEA questions can be included as a module within the implementation study. Step 3. Analyze data and identify recommendations for moving forward with operational engagement to overcome the structural and institutional challenges. The goal of PEA is to create a new roadmap for change that helps programs reach their outcomes. Data collection methods: Document review, KIIs, administrative data from the monitoring evaluation and learning (MEL) plan review, cost data. Data analysis: Coding and analysis of KIIs; mapping of results of the fidelity of implementation criterion; mapping of stakeholder power analysis; cost analysis. Timeline for completing evaluation: 4–6 months Cost: Lower to medium cost option Benefits of this design: This qualitative implementation study paired with a rapid PEA is best for providing formative guidance to the implementer. It assesses their performance to date and helps them understand institutional barriers that prevent LASER from reaching its outcomes. It provides recommendations for course corrections based on performance to date. Midterm Evaluation Option 2: Assessing early results of LASER. This midterm performance evaluation design uses an implementation study to understand how LASER's unique approach to engaging partners through R4D, ERT, and CSFA contributes to reaching its targeted outcomes. The evaluation draws on one round of interviews with LASER PULSE and its partners, researchers, USAID key staff, and other stakeholders directly involved in LASER's implementation. The implementation study employs document reviews and key informant interviews to map current activities against the original workplan, accounting for necessary changes. It also identifies facilitators and barriers to implementation, and how those facilitators and barriers may affect progress toward desired outcomes. For example, the implementation study can examine how PULSE’s partner engagement strategies broaden the diversity of researchers they work with, or how exactly ERT protocols improve the immediate usability of research. The evaluation team will also review administrative data from the MEL plan to determine planned versus actual targets. The main steps to the implementation study include: Step 1. Review the theory of change and identify the goals and objectives. Step 2. Conduct meetings with implementers to deepen understanding of the interventions. Step 3. Develop an online survey and KII protocols to gather perceptions and data around LASER implementation strategies. Step 4. Conduct document review and finalize instrumentation, including pilot testing. Step 5. Conduct online survey and analyze results. Step 6. Refine and focus interview protocols based on survey findings. Step 7. Conduct interviews. Step 8. Code and analyze KII and administrative data. Step 9. Interpret findings. Data collection methods: Online survey, KIIs, document review, and administrative MEL data. Data analysis: Descriptive quantitative analysis, qualitative coding and thematic analysis. Timeline for completing the evaluation: 4–5 months. Cost: Lower cost option Benefits of this design: The implementation study provides formative guidance for project improvements that can be implemented over the next two to three years. The results focus on the 6 unique strategies LASER is implementing and how those strategies can be improved over time to reach the program's outcomes. It is a low-cost, rapid option that provides targeted data. Final Performance Evaluation Option 3: Assessing mature results and cost-efficiency of LASER. This final mixed-methods performance evaluation includes (1) an implementation study and (2) a cost analysis. This design assesses whether LASER met its theory of change outcomes, examines the lessons learned from implementation, and assesses the value-added of the different strategies and funding mechanisms LASER used. This design option could build on the midterm implementation study if USAID chooses to implement a midterm and final evaluation or draw on one round of data collection at the end of the program. The qualitative implementation study will use document review, KIIs, focus groups, and a survey to assess the outcomes of the R4D, ERT, and CSFA strategies and LASER's ability to reach its theory of change (see key steps above). The evaluation team can use the aforementioned fidelity of implementation criterion to develop protocols to assess implementation against adherence to its model, acceptance by the targeted audience, and cost. This option could also include an outcomes analysis (described under option 4) using MEL data over time. The second component is a cost analysis. The evaluation team will use USAID's cost guidance tools to estimate LASER activities' value for money. This component can include producing cost￾economy or cost-efficiency estimates. The evaluation team must begin working with the implementing organization at the inception of the work to gather relevant cost data and break them down in a way that examines the value for money. Data collection methods: KIIs, focus groups, document review. Data analysis: Qualitative coding and thematic analysis, cost analysis, trends analysis of administrative MEL data. Timeline for completing the evaluation: 12 months Cost: Medium- to higher-cost option Benefits of this design: This end-of-project design provides a much more in-depth understanding of how LASER strategies contributed to the program's outcomes compared to the midterm options. It also allows the evaluation team to look at value for money related to the implementation strategies and at the core versus buy-in funding structures. If USAID wanted to do both a midterm and final evaluation, this option could also add a PEA analysis that builds on the midterm evaluation option 1. By conducting two rounds of data collection, the evaluators can learn how LASER used midterm feedback to adjust programming— and the adjustments' results. Final Performance Evaluation Option 4: Assessing the LASER model. This mixed-methods end-of-project performance evaluation comprehensively assesses the LASER design. It qualitatively compares it to other research programs designed and implemented by USAID with the objective of identifying key elements of each research program that best help USAID reach its research goals. The proposed design includes three components: a qualitative comparative analysis (QCA), an outcomes analysis, and a cost analysis. The first component uses a qualitative comparative analysis to examine patterns across multiple projects and cooperative agreements. The design identifies sets of conditions that are most likely to lead to a given outcome. The data analysis process embedded in a QCA can reduce the qualitative complexity of different items, such as distinct funding programs, and compare each one's essential characteristics related to specific outcomes.1 1 Qualitative comparative analysis can be defined as “a means of analysing the causal contribution of different conditions (e.g., aspects of an intervention and the wider context) to an outcome of interest” (BetterEvaluation.org 2021). An example of qualitative comparative analysis used in development can be found here: https://journals.sagepub.com/doi/full/10.1177/10982140177 10502. 7 QCA is a case-based approach that uses qualitative data from document reviews and interviews and quantitative MEL data. The methodology is based on two assumptions: (1) that change is often the result of different combinations of factors, rather than one individual factor; and (2) that different combinations of factors can produce similar changes (Ragin [1984] as cited in INTRAC 2017). The evaluation team would develop cases for each research program (HESN, PEER, RTAC). The team would then compare and draw patterns across the cases about why some activities or approaches work and others do not. There are six key steps in the QCA process that each case study analysis should include: Step 1: Develop a theory of change. The team can develop a theory of change or use an existing one. The purpose of the analysis within this step is to identify the changes the QCA study aims to focus on and the factors that (in theory) help bring them about. Step 2: Identify the cases. The evaluation team selects a group of cases with similar outcomes, a group of cases where desired outcomes were achieved by the program, and a group of cases where the outcomes were not achieved. Step 3: Identify a set of factors or conditions. The key factors or conditions selected are those whose presence or absence may contribute to the outcomes (e.g., university-based or contractor￾based; core, buy-in, or hybrid funded; type of strategy used to engage MBIOs). It is essential that all factors in the theory of change are included in the analysis. Step 4: Score each factor. The scoring process involves looking at all factors across the selected cases and developing criteria for scoring them. For example, a factor can be scored “0 or 1” depending on whether it is present or absent in the case. Step 5: Data analysis. After scoring all the factors across the cases, the evaluation team analyzes the data for patterns. Step 6: Interpret the findings. This step is often iterative as the evaluation team goes back and forth between the data and interpretation. The strengths of QCA include a rigorous methodology for drawing comparisons across a small number of cases, the ability to address why some interventions or approaches worked while others did not, and the ability to combine with MEL data to help implementers and donors learn from the implementation process. The methodology’s weaknesses include a scoring system based on subjective judgments, an inability to contend with missing cases, and time-intensive requirements that make it costly. The second component of the evaluation is an outcomes analysis. The outcomes analysis uses secondary data to assess the extent to which key outcomes changed over the life of the project. These outcomes could be compared to HESN 1.0, PEER, and RTAC (particularly if the evaluator stacks outcome values for each program component) to provide insights into the different ways that these programs support USAID's learning agenda and uptake of research by different stakeholders. The analysis could also feed into step 5 of the QCA analysis. The outcomes analysis can be paired with a qualitative round of data collection focused on key stakeholder interviews, if resources allow. The steps for conducting an outcomes analysis include: Step 1. Gather all the MEL plan administrative data for analysis. Step 2. Determine the categories of outputs and outcomes standard across the various research programs (e.g., products produced, direct beneficiaries, utilization of research). 8 Step 3. Gather any additional outcomes data from secondary sources not available in the MEL data. Step 4. Separate outputs and outcomes according to core-funded, buy-in funded, hybrid, or other type of funding mechanism to examine trends by main funding sources.2 2 The outcomes analysis could also be stacked by type of outcome or output. Step. 5. Conduct a trends analysis by organizing data annually. Step 6. Look for patterns across the cases. The main advantage of an outcomes analysis is that it allows you to compare outcomes across programs with similar goals; flexibility is embedded in the approach in selecting key variables, and it provides quantitative trends over time. The main disadvantage of the approach is that it reduces learning to a measurable outcome. However, when paired with QCA, the evaluation overcomes this limitation. The third component of this evaluation design is a cost analysis. The cost analysis employs USAID's cost guidance tools to estimate the value for money of LASER activities. This component can include cost economy, cost-efficiency, or cost￾effectiveness analysis and can also be used to compare LASER's value to similar funding programs. Data collection methods: Document review, KIIs, administrative and secondary data. Data analysis: quantitative trends analysis; cost economy or cost-efficiency analysis; factor analysis and scoring; thematic qualitative analysis of QCA data. Timeline for completing the evaluation design: 12– 18 months. Cost: Higher cost option Benefits of this design. There are three main benefits of this design compared to options 1–3. First, this is the most comprehensive design that examines the type of implementation model, compares it to the other USAID research programs, and compares outcomes and provides value-for￾money data. Second, it analyzes factors across different models to help USAID understand which factors in research design programs help USAID reach its desired outcomes. Finally, this design looks at USAID research programs at a systems level (understanding factors of success) and at a programmatic level, which could inform how USAID designs research programs in the future. 9 Table 1. Midterm evaluation (Option 1) . Research questions Evaluation design(s) Advantages Disadvantages MIDTERM OPTIONS Option 1: Assessing LASER implementation and system dynamics Implementation 1. How well has the blend of core and buy-in activities met the needs of USAID Missions, Bureaus, and Independent Offices (MBIOs) to date? 2. Comparing university-based research models such as LASER, HESN 1.0, and PEER to other implementer-based models such as RTAC: ● What advantages do university-based implementers provide in helping USAID meet its research goals? ● What are the disadvantages or weaknesses in these models? 3. What adjustments can LASER make to its funding or implementation structure in its final two years to ensure it meets its original intermediate results (IRs)? 4. How resilient was the LASER program to unforeseen circumstances, such as the COVID-19 pandemic? How did LASER’s design support or limit its resiliency? 5. How well has LASER achieved its planned outputs in the first three years of the program? System dynamics 6. What incentives and relationships among LASER stakeholders facilitated forward progress toward the goals (IRs)? How? 7. What incentives and relationships among LASER stakeholders hindered the program’s progress toward its goals (IRs)? How? 8. How could incentives of LASER stakeholders be better aligned to improve progress toward the program’s IRs? 9. How could relationships among LASER stakeholders be improved to facilitate progress toward program IRs? Qualitative implementation study with a rapid political economy analysis (PEA) Provides actionable lessons on program design and stakeholder incentives and relationships. Helps understand power dynamics among stakeholders that help or hinder implementation. Provides feedback on how the LASER model is functioning and how to improve programming in the latter half of the project. Medium-cost option. Rapid PEA cannot capture the full scope of relationships or outcomes, particularly over time. Option does not examine project impact and cannot capture long-term achievements such as research uptake. Results from the PEA are only useful if LASER implementing partners can act on findings. Results have limited long-term use. 10 Table 2. Midterm evaluation (Option 2) . Research questions Evaluation design(s) Advantages Disadvantages MIDTERM OPTIONS Option 2: Assessing early results of LASER 1. To what extent has LASER made progress toward the following planned results? ● Increased delivery of collaborative and effective development￾focused research ● Increased synthesis, exchange, and translation of research results into useable development products and practices ● Increased dissemination of translated research results for evidence-based solutions ● Enhanced systems and structures for gender and minority considerations in the HEI network that enable women and minorities to conduct research 2. How are LASER's Comprehensive Success Factors Analysis (CSFA), Research for Development (R4D) Convenings, and Embedded Research Translation (ERT) strategies affecting: ● MBIO satisfaction and uptake of research results; and ● Academic and news media uptake of research results? 3. How do midterm results from LASER in those areas compare to mid￾project results from similar USAID funding systems such as PEER, RTAC, HELIX, and HARPNET? 4. What changes can USAID and implementing partners make to the LASER strategies in the remaining time to ensure the project meets its goals and objectives? Qualitative implementation study Allows LASER to quickly improve ongoing programming. Provides feedback on the CSFA, R4D, and ERT strategies along with lessons learned. Low-cost option. Results have limited long-term use. Results are based mainly on review of documents, M&E reporting data, and interviews providing perception data. 11 Table 3. End-of-project evaluation (Option 3) . Research questions Evaluation design(s) Advantages Disadvantages FINAL EVALUATION OPTIONS Option 3: Assessing mature results and cost-efficiency of LASER 1. How have LASER's Comprehensive Success Factors Analysis (CSFA), Research for Development (R4D) Convenings, and Embedded Research Translation (ERT) strategies affected: - MBIO satisfaction and uptake of research results; and - Academic and news media uptake of research results? ● How do results from LASER in those areas compare to other USAID funding systems such as PEER, RTAC, HELIX, and HARPNET? 2. How did LASER stimulate engagement and growth of researchers, particularly women and minorities, in international development research? 3. To what extent have other programs adopted strategies similar to CSFA and ERT as a result of LASER dissemination activities? 4. What was the cost-per-output under LASER, and did the project’s efficiency meet expectations? 5. How resilient was the LASER program to unforeseen circumstances, such as the COVID-19 pandemic? How did LASER adapt to these challenging circumstances? 6. How does LASER's value-for-money compare to similar cooperative agreements in terms of research and policy impacts? 7. What other factors—and questions—should USAID explore to gain insights into the best way to engage universities and MBIOs to further the international development research agenda? Qualitative implementation study with a cost analysis Provides feedback on the CSFA, R4D and ERT strategies along with lessons learned. Cost analysis can provide insights into the value for money for each strategy. Focused, medium￾cost option end-of￾project option. Requires strong administrative data, including detailed cost data, and interviews providing perception data. Does not examine impact. 12 Table 4. End-of-project evaluation (Option 4) . Research questions Evaluation design(s) Advantages Disadvantages FINAL EVAUATION OPTIONS Option 4: Assessing the LASER model 1. Which aspects of USAID models to fund research best contribute to the use of research for policy and decision making in LMICs? To research uptake and use for community development? ● Which funding model (e.g., cooperative agreement, contract, APS) has the best value for money? ● Within LASER, which component (core vs. buy-in) has the best value for money as it relates to the stated outcomes of the project? ● How does the LASER model compare to similar USAID cooperative agreements and funding programs, such as PEER, RTAC, HELIX, and HARPNET, in terms of meeting planned objectives and outcomes? ● Which elements of LASER's design facilitated the program reaching each of its IRs? ● Which elements of LASER's design slowed progress toward IRs? 2. Comparing university-based models such as LASER, HESN 1.0, and PEER to other implementer-based models such as RTAC: ● What advantages do university-based implementers provide in helping USAID meet its research goals? ● What are the disadvantages or weaknesses in these models? 3. How does LASER's value-for-money compare to similar cooperative agreements in terms of research and policy impacts? 4. Which cooperative agreement model (HESN 1.0, LASER, PEER) has been used the most by MBIOs? Why? 5. What type of funding design or model is best suited to helping USAID reach its objectives of research uptake, evidence-based decision making, and contributions to international development? Mixed-methods performance evaluation with 3 components: 1. Qualitative comparative analysis (QCA)3 3 Qualitative comparative analysis can be defined as “a means of analysing the causal contribution of different conditions (e.g. aspects of an intervention and the wider context) to an outcome of interest” (BetterEvaluation.org 2021). An example of qualitative comparative analysis used in development can be found here: https://journals.sagepub.com/doi/full/10.1177/1098214017710502. 2. Outcomes analysis 3. Cost analysis Analyzes patterns across multiple projects and cooperative agreements to identify sets of conditions that are most likely to lead to a given outcome. Provides life-of￾project view of results and potential contributions to development by looking at outcomes. Provides comparable value-for-money estimates, which could be broken down by core and buy-in. Requires strong administrative data, including detailed cost data. Does not examine impact. Medium- to high￾cost option.