USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Final Evaluation Report Photo: Global Devlab/USAID Photo: Global Devlab/USAID ACKNOWLEDGEMENT We would like to thank Triple Line authors, Courtney Roberts, Chelsea Keyser, Stephen Rahaim, Lorin Kavanaugh-Ulku, Katharine Sullivan, Meredith Perry, Scott Jackson, Ku McMahan, Augusta Abrahamse, Meghan Majorowski, Will Durbin, Ankunda Kariisa, Joshua Josa, Michelle Malecki, Devorah West, Meera Pranav, Molly Doyle, Maurice Kent, and Elena Walls. We are also grateful to Alex Dehgan, Wendy Taylor, Dave Ferguson, and Alexis Bonnell for sharing insights with the evaluation team. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Final Evaluation Report DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Photo: Global Devlab/USAID TABLE OF CONTENTS ACRONYMS AND ABBREVIATIONS i DEFINITIONS OF TERMS USED IN THE EVALUATION iii ABSTRACT iv EXECUTIVE SUMMARY v 1 THE GRAND CHALLENGES FOR DEVELOPMENT AND THE META-EVALUATION: INTRODUCTION, CONTEXT, AND OVERVIEW 1 1.1 Introduction 1 1.2 Context 1 1.3 Overview of the Grand Challenges 4 2 EVALUATION OBJECTIVES AND SCOPE 7 2.1 Evaluation Design and Questions 7 3 EVALUATION METHODOLOGY 11 3.1 Data Collection 11 3.2 Data Analysis and Synthesis 13 3.3 Limitations 13 4 FINDINGS 15 4.1 Retrospective Assessment of GC Results 15 4.2 Scaling Innovations 23 4.3 Acceleration 27 4.4 Investing in Ecosystems 31 4.5 Catalytic Effects 38 4.6 Procurement and Reducing Barriers 43 4.7 Engaging USAID Missions and Operating Units (OUs) 49 4.8 Measuring Impact, Results, and Uptake 53 4.9 Measuring Cost-Effectiveness 59 4.10 Governance, Partnership Models, and Operational Effectiveness 61 5 CONCLUSIONS AND RECOMMENDATIONS 67 5.1 Conclusions 67 5.2 Strategic Recommendations 69 5.3 Programmatic Recommendations 71 6 DISSEMINATION PLANS 75 6.1 Knowledge Products 75 6.2 Learning and Dissemination Workshop 76 ANNEX 1 PORTFOLIO MAPPING 77 ANNEX 2 EVALUATION MANAGEMENT 80 ANNEX 3 EVALUATION METHODOLOGY 81 ANNEX 4 EVALUATION MATRIX 88 ANNEX 5 ANALYTICAL FRAMEWORK 89 ANNEX 6 EVALUATION TOOLS 90 ANNEX 7 KEY INFORMANTS INTERVIEWED 124 ANNEX 8 LIST OF DOCUMENTS CONSULTED 128 ANNEX 9 COMPARATOR ANALYSIS 129 ANNEX 10 COST-EFFECTIVENESS ANALYSIS REVIEW 147 ANNEX 11 COST EFFECTIVENESS ANALYSIS FRAMEWORK 157 ANNEX 12 STATUS OF COST-EFFECTIVENESS ACROSS GCS 172 ANNEX 13 INSIGHT MEMO: LEARNING FROM THE INTEGRATION OF GENDER AND SOCIAL INCLUSION (GESI) INTO USAID GRAND CHALLENGES 177 ANNEX 14 CASE STUDIES 189 ANNEX 15 SURVEY FINDINGS 205 ANNEX 16 PROCUREMENT ANALYSIS 207 ANNEX 17 TERMS OF REFERENCE 209 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION i ACRONYMS AND ABBREVIATIONS ACR GCD All Children Reading: A Grand Challenge for Development AfDB African Development Bank BMGF Bill & Melinda Gates Foundation BZ Ministerie van Buitenlandse Zaken (Dutch Ministry of Foreign Affairs) CDC Center for Disease Control CDCS Country Development and Cooperation Strategies CEA Cost Effectiveness Analysis CHIC Creating Hope in Conflict: A Humanitarian Grand Challenge CII Center for Accelerating Innovation and Impact, situated within USAID’s Global Health Bureau CSO Civil Society Organization DDI Democracy, Development, and Innovation DFAT Australian Department of Foreign Affairs and Trade DIV Development Innovation Ventures DoD Department of Defense DRC Democratic Republic of the Congo DSI South African Department of Science and Innovation E3 USAID Bureau for Economic Growth, Education, and Environment EBOLA Fighting Ebola EdTech Educational Technology EEHSC Ensuring Effective Health Supply Chains EPD Energy Private Development EPIC Exploratory Programs and Innovation Competition, situated within USAID’s Development, Democracy and Innovation Bureau FCDO UK Foreign, Commonwealth & Development Office FLQ Forward-Looking Question GAC Global Affairs Canada GC Grand Challenge / Grand Challenge for Development GCC Grand Challenges Canada GESI Gender Equality and Social Inclusion GH Global Health Bureau GiZ Gesellschaft für internationale Zusammenarbeit (German Corporation for International Cooperation) HESN USAID Higher Education Solutions Network HIC High Income Country IDIA International Development Innovation Alliance IDS Institute for Development Studies IIA Innovation Investment Alliance INGO International Non-Governmental Organization KII Key Informant Interview KOICA Korea International Cooperation Agency KP Knowledge Product LMIC Low- and Middle-Income Country MAVC Making All Voices Count Grand Challenge for Development MEL Monitoring, Evaluation, and Learning MENA Middle East and North Africa MEQ Main Evaluation Question MNH Maternal and Neonatal Health Norad Norwegian Agency for Development Cooperation OEQ Overarching Evaluation Question OU Operating Unit PAEGC Powering Agriculture: An Energy Grand Challenge for Development PoC Point of Contact PPE Personal Protective Equipment R&D Research and Development SEQ Sub-Evaluation Question SHOPS Plus Sustaining Health Outcomes through the Private Sector Plus Sida Swedish International Development Cooperation Agency ii USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION SIRA Selective Integrated Reading Activity SL@B Saving Lives at Birth Grand Challenge for Development SOGE Scaling Off-Grid Energy Grand Challenge for Development SoW Statement of Work SPIS Solar Powered Irrigation System SWFF Securing Water for Food Grand Challenge for Development TA Technical Assistance TB Tuberculosis ToC Theory of Change ToR Terms of Reference TTS Transition to Scale UAV Unmanned Aerial Vehicle USAID United States Agency for International Development WE4F Water and Energy for Food Grand Challenge for Development WFP World Food Program WHO World Health Organization WMP World Mosquito Program Zika Combating Zika and Future Threats Grand Challenge for Development USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION iii DEFINITIONS OF TERMS USED IN THE EVALUATION 1 See https://www.usaid.gov/fallsemester/usaid-101/innovation-development 2 International Development Innovation Alliance (June 2017) Insights on Scaling Innovation The following terms are used throughout the evaluation. Acceleration strategies: interventions and support which speed up an innovation’s movement along the pathway to scale and therefore their growth and impact. Applicant: an organization that responds to a call for proposals for funding to develop an innovation that helps solve the problem addressed by the Grand Challenge for Development. Awardee: an organization who has been selected to receive a Grand Challenge for Development grant at the end of a competitive selection process but has not yet received the grant funding or started implementation. Grantee: an organization that is in receipt of a Grand Challenge for Development grant and is implementing their proposed project to develop, test, implement, or scale an innovation. Innovation: a novel business or organizational model, operational or production process, or product or service that leads to substantial improvements (not incremental ‘next steps’) in addressing development challenges. Innovation may incorporate science and technology, but is often broader, to include new processes or business models.1 Innovation ecosystem: the context of organizations, markets, key actors, and policies in which an innovator is developing and implementing their innovation. Innovator: an organization or individual who is developing and testing innovations who may or may not be in receipt of a Grand Challenge for Development grant. Non-traditional actor: a grantee or partner organization who has not previously accessed USAID funding to implement projects. Examples include small, locally-owned, or private sector organizations. Pathway to scale: the journey an innovation takes from being an untested idea to being implemented and used by significant numbers of people. Steps along the pathway can be characterized as follows: Ideation; Research and Development; Proof of Concept; Transition to Scale; Scaling; and Sustainable Scale.2 System: the network of actors and organizations that work together through relationships, regulations, principles, and procedures to deliver a goal – for example, a country’s health system or the international humanitarian system. iv USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ABSTRACT The meta-evaluation of nine USAID Grand Challenges for Development (GCs), undertaken by Triple Line in 2020-2021, examines what the GCs have achieved and how they have done so in order to provide USAID with an actionable evidence base. The report sets out the policy context for the introduction of Grand Challenges in 2011 as a new approach to programming and provides information about each of the nine GCs studied. Findings are presented against each of ten evaluation questions, exploring the achievements of the GCs (results, scaling, acceleration, innovation ecosystems, and catalytic effects); aspects of GC implementation (procurement and access, and the engagement of USAID Missions); and how GCs have been managed (measurement of results and cost effectiveness, and governance arrangements). The findings lead to the evaluation’s conclusions about the achievements and effectiveness of USAID’s GCs: i) that, overall, they have achieved positive results in varied sectors, many of which are likely to be sustainable, and have supported the scaling of some significant innovations; and ii) that the GC model, when implemented well, is a results-driven approach that is both effective at supporting innovations to become scale-ready and at strengthening ecosystems. The report ends with two sets of practical, actionable recommendations to strengthen GC programing: strategic recommendations for USAID policy and GC managers, and programmatic recommendations for USAID and partner GC managers. The report annexes provide additional methodological information, further supporting analysis undertaken for the evaluation on comparators, cost effectiveness, and gender and social inclusion, and evidence from the grantee survey conducted for the evaluation. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION v EXECUTIVE SUMMARY USAID’s meta-evaluation of the Grand Challenges for Development (GCs) was commissioned to enable systematic reflection on ten years of experience across nine challenges and to generate an actionable evidence base and recommendations on which to build future programming. GCs are programs that mobilize governments, companies, and foundations around specific developmental or humanitarian challenges. Through these programs, USAID and public and private partners bring in new voices to solve developmental problems. GCs source new solutions, test new ideas, and scale what works by awarding grants and using additional tools to provide targeted technical assistance support to a wide variety of actors from many countries. Their introduction signaled a shift away from large bilateral and multilateral agencies, aid organizations, and private voluntary organizations, and towards non-traditional actors such as the private sector, smaller civil society organizations, and academic institutions and local partners. GCs harness new technologies and collaborative partnerships in support of entrepreneurship, collective problem solving, and new approaches. They enable USAID to foster innovative solutions by mobilizing its convening power to leverage the funds and resources of other agencies, testing a range of solutions to identify those with highest potential to succeed at scale, supporting commercialization, operating with a higher tolerance for risk, enabling the flexible use of funds through milestone-based funding mechanisms, and supporting different stages of innovation. The GC portfolio review of the meta-evaluation represents a total investment of $463.4M from USAID and other donor partners since 2011. 767 awards were made from nearly 16,000 applications across nine challenges in the agriculture, energy, education, health, humanitarian, and governance sectors. Innovations have been supported in 83 countries in every region of the globe and 94 projects have a global or regional focus. The USAID GCs All Children Reading (ACD GCD) aimed at advancing EdTech innovation and research to improve reading outcomes for marginalized children in low resource settings. Combating Zika and Future Threats (Zika) invested in innovations designed to prevent, detect, and respond to the Zika virus and improve capacity to combat future infectious disease outbreaks. Creating Hope in Conflict (CHIC) funds innovations to save and improve the lives of the most vulnerable and hardest-to-reach people affected by humanitarian crises caused by conflict. Fighting Ebola (EBOLA) funded cost￾effective, scalable, and practical solutions to improve the provision of healthcare worker equipment and improve infection treatment and control measures. Making All Voices Count (MAVC) aimed to strengthen civil society and government engagement to strengthen accountability and policy decision-making. Powering Agriculture (PAEGC) aimed to improve agricultural practices, increase food production, and improve efficiency by increasing access to reliable, affordable, and clean energy. Saving Lives at Birth (SL@B) sought interventions to save lives around the time of childbirth and the early postnatal period for women and newborns in underserved, disadvantaged, and marginalized communities around the world. Scaling Off-Grid Energy (SOGE) supported investments in new business models, approaches, and geographies and strengthened local ecosystems to accelerate growth in the household solar energy market with the goal of connecting households in Sub-Saharan Africa. Securing Water for Food (SWFF) supported solutions enabling the production of more food with less water and/or making more water available for food production, processing, and distribution. vi USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION The meta-evaluation covered ten areas of inquiry with an overall aim of assessing i) the effectiveness of the model for delivering results and achieving development and humanitarian outcomes, and ii) its appropriateness for achieving sustainable results and strengthening the innovation ecosystem. The meta￾evaluation questions related to GC achievements, implementation, and management. The meta-evaluation makes detailed, practical recommendations grounded in the meta-evaluation evidence aimed at policymakers, GC managers within USAID, and GC implementing partners. They are designed to strengthen implementation of the GC model in the future as it enters its second decade. OVERVIEW OF FINDINGS The GC model has led to the effective delivery of results and the achievement of positive outcomes in development and humanitarian action with results stronger in some GCs than others. As a result of the evaluated GCs lives have been saved and improved in conflict settings; communities are using clean energy systems and millions of people have off-grid energy connections; farmers have produced more food, increased their incomes, and reduced water consumption; mothers’ and babies’ lives have been saved; and education has become more inclusive. There is sufficient evidence to believe that results are sustainable, particularly for the stronger performing GCs. The GC model has many distinctive features which support the results, scaling, and strengthening of innovation ecosystems. The model has supported results by leveraging substantial additional funds, bringing in a wide range of actors to tackle development challenges, and supporting innovative solutions with both funding and technical assistance. It works well across different sectors and contexts. Where performance has fallen short, it was due to weaknesses in how the model had been implemented not the model itself: some GCs faced difficulties due to weaknesses in areas such as understanding of the context, objective setting and results measurement, learning and adaptation, supporting pathways to scale, and coordinated engagement by donors. GCs are better suited to early- and mid-stage innovation and supporting innovations along the pathway to scale rather than implemention at and achieving scale, which has occurred more rarely and takes time and finances. Those comparatively few innovations which have achieved notably large￾scale results and scaling seem particularly likely to establish and progress beyond GC support. The GC model does support innovators to engage effectively with the innovation ecosystem – the context of organizations, markets, policy environments, and key actors – but has been more limited in its attempts to strengthen that ecosystem. GC ACHIEVEMENTS The GCs have achieved good results. Some innovations have scaled, but the focus has been largely on early- to mid-stage innovations which evidence indicates GCs are better placed to support. GCs have provided innovators with support for acceleration (moving along the pathway to scale) and for engaging with the innovation ecosystem which has to be tailored to context to be effective. GCs have leveraged substantial funding. Results Five of the nine GCs met or exceeded targets and there is sufficient evidence to confirm that results are sustainable. Three achieved results but could not be rated because objectives and targets had not been defined clearly enough to judge their overall success. Results for gender equality and social inclusion (GESI) were clear for just three GCs whose achievements were reported in a way that made women and other disadvantaged groups visible. GCs have been successful at surfacing innovative solutions from actors and stakeholders that are new to USAID. They are particularly effective at sharing the risk of developing and testing new solutions, approaches, techniques, and technologies. Scaling There are some stand￾out successes of GC-supported innovations that have scaled, but there is no consistent definition or data for achieving scale and very few innovations have been supported to do so. 86 percent of grants have been for early- to mid-stage innovations (ideation, research and development, and proof of concept) with a focus on setting the groundwork for scaling. Taking innovations to scale often requires significant financial support: out of 519 awards for which there is data only 20 were for $2M or more. The time required to take most early-stage innovations to scale is also considerable, whether for innovations using market solutions or public partnership engagement to attract funding. GCs which have successfully supported innovations to reach scale have considered the pathway to scale from the outset, including the needs of end users, USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION vii understood local context by developing relevant local partnerships, and deployed local expertise to provide scaling support. Success at scale: Zika and USAID assisted the World Mosquito Program to establish a model for scaling and sustainability beyond GC funding through a loan￾based buy-in model in partnership with the Brazilian government which released foundation funding and buy-ins with municipalities. Acceleration is about innovations moving along the pathway to scale to reach growth and impact. Technical assistance and innovation support are needed for this to happen. This requires a clear strategy for acceleration from the outset. For some GCs there was less appreciation of this and they relied instead on innovations themselves to choose their path to scale using specific scaling grants. Some, such as SL@B, who did not address this at all at the start, adjusted its approach and provided good acceleration support later on. Acceleration support needs to be customized to innovator needs and the local context, and can take many forms. Acceleration support providers must have a wide skill set with experience of local markets, but these services can be resource intensive and, for global GCs, logistically challenging. Successful Acceleration Support: SL@B provided tailored acceleration support to Gradian’s anesthesia program in Zambia and coaching on scaling their model in other countries. Support enabled Gradian’s partners to field test and refine a simulation-based training curriculum and methodology, and professional coaches helped them to pitch for financial support. Ecosystem investment Engaging ecosystem stakeholders is crucial to increasing the reach and impact of GC-funded innovations. All GCs have implemented some programing and investment in the ecosystems associated with their innovations, and there are numerous instances of successful interfaces which have led to increases in reach and impact for GC-funded innovations. There is no evidence of a consistent or formal approach to ecosystem engagement across the GC portfolio as a whole and only limited examples of GCs working with USAID Missions and other Operating Units and programs. Relations with USAID could be leveraged further to strengthen GC implementation. Creating a regional or local presence can focus energies and resources effectively, but geographically wider-ranging portfolios can struggle to influence systemic change. Catalytic effects The GC model is very effective at catalyzing funding and at raising awareness. $971.8M has been leveraged from external investors. The model de-risks investment by sharing the risk and supporting innovators with technical assistance. GCs also engage intentionally with investors and other actors who might bring funding and help innovators to become investment ready. To be effective, this requires a strategy tailored to GC objectives and the investment context (such as actors, markets, and policies). GCs raise awareness of both challenges and solutions effectively when they can demonstrate achievements; develop communications products; leverage the credibility of donor backing, convening power, and brand value; and showcase the distinctiveness of the GC approach. Recommendations for Action • GCs should undergo complete and rigorous design ahead of launch involving effective USAID engagement with donors and alignment of all parties around the GC objectives and intended outcomes. Understanding context and how innovations might develop along the ‘pathway to scale’ is critical, as is a good system for results measurement. • Strengthen outcomes for women and disadvantaged groups by embedding GESI principles into GC design and reporting. • Focus GCs on supporting early- to mid-stage innovations and getting them ready to scale. • Ensure that GCs have an acceleration and scaling strategy in place from the outset, that it is rooted in an understanding of the principles of acceleration, that it adapts to need over time, and that it is delivered by experienced providers with local knowledge and networks. • Include ecosystem engagement in program design, build connections with the other USAID programs or other GC partners operating in the same space, and use USAID’s and partner’s convening power to forge early links between innovators, experts, funders, and potential investors. Support this with a shared model, methodology, and practical toolkit for GCs. viii USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION GC IMPLEMENTATION GCs have taken many steps to increase their reach. The high volumes of applications received are resource intensive to process but foster wide and open competition. GCs and USAID Missions could both benefit from further mutual engagement. Procurement Reaching and funding non-traditional actors – those who would not normally access USAID funding – is an important way of meeting GC objectives. Most GCs have taken explicit and practical steps to increase their accessibility, particularly for innovators from developing countries. The low percentage of total applicants who receive awards (4.8 percent), is in part due to the GC model of open competition, making the selection process intensive and often lengthy. USAID Missions engagement GCs engage with USAID Missions inconsistently – some successfully, others not all – but where they do, they reap benefits by strengthening their relevance, sustainability, and linkages to the innovation ecosystem. Benefits are mutual: through GCs, Missions can potentially access a cost-effective and easy-to-use mechanism to further their priorities in-country. Recommendations for Action • GCs need to scope the innovation landscape and context and understand the market to attract and support diverse, good quality applications while deterring ineligible and weak applications. • GCs could engage with USAID Missions to add value to design and implementation plans, application sourcing, ecosystem strengthening, and longer-term results monitoring, and to explore mutual benefits. GC MANAGEMENT Results measurement would benefit from a consistent and rigorous approach in order to make GC achievements visible and to demonstrate the worth of the investment. Within the various models of governance and management, clarity and shared objectives are critical. Results measurement Partly due to their dispersal across USAID, each GC has its own approach to measuring impact, results, and uptake rather than operating within an overarching strategy or framework which sets out clear definitions and expectations. There are some good examples of clear, aligned objectives and expectations of results supported by systems to aggregate results from projects, but this was partial for many GCs which makes it difficult for them to demonstrate their overall achievements. For many GCs, undifferentiated results (e.g., for ‘people’) mean that outcomes for disadvantaged groups, including women, are not visible. GCs provide support to grantees in results monitoring and learning and there are positive examples of monitoring data and learning being used to strengthen both GC and project implementation, although this is not done systematically. Cost effectiveness analysis Those who are concerned with designing, managing, and implementing GCs have considerable interest in GC cost effectiveness analysis (CEA) for both evaluating the performance of the portfolio and to determine which innovations should receive acceleration support towards scale. To date, there have been no clear mandates, standards, or simple and user-friendly tools and techniques for GCs to undertake CEA. As a result, there were wide differences among GCs in their understanding and approach to it. Governance, management, and effectiveness There are different models for GC governance and management. A principal variable is whether the GC is funded by USAID alone or in partnership with other donors. There are several advantages to a multi-donor model, such as complementary expertise and wider networks, although it increases the complexity and transaction USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ix costs of governance structures. Overall management can be problematic if donors are not aligned. Both in governance and in management (which can be in￾house, external, or hybrid), clarity about objectives, clear lines of accountability, and the effective deployment of expertise are essential. Recommendations for Action • Establish a set of overarching principles and approaches for monitoring, evaluation, and learning, setting out clear expectations and practical guidance for GCs to apply to their own program. • GCs should develop clear objectives, performance indicators, monitoring systems, and evaluation approaches for the GC as a whole and support innovators to do likewise. This should make gender and other characteristics of exclusion visible. • A common framework for CEA should be applied across all GCs. (Proposals are set out in Annex 11 of this USAID GC meta-evaluation report.) • Build a vision and set of objectives for the GC which are shared by all parties, informed by a clear intervention model, supported by clear decision-making processes, adequate resourcing, and clarity about roles and responsibilities. Grand Challenges for Development CHIC Zika PAEGC SWFF SL@B ACR GCD MAVC EBOLA SOGE 2.2M people in conflict with access to improved humanitarian services Surfaced leading technologies for improved public health Reached 234,100 with clean energy to increase agricultural productivity Benefitted 7.1M farmers in 28 countries; 19B litres less water consumed Reached 3M mothers and babies. Improved over 155,000 lives and saved over 11,500 Over 600,000 early grade learners reached Improved government responsiveness to citizen requests Innovations went on to support West African responses to COVID-19 1.2M actual and 14.4M projected off-grid connections IMPACTS AND OUTCOMES x USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Photo: Global Devlab/USAID USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 1 1 THE GRAND CHALLENGES FOR DEVELOPMENT AND THE META￾EVALUATION: INTRODUCTION, CONTEXT, AND OVERVIEW 1 ‘GC’ is the selected abbreviation, rather than the also prevalent GCD – standing for Grand Challenges for Development – to reflect the fact that the GC family encompasses both development and humanitarian interventions and thus ensure that it reflects the entirety of USAID’s work. 2 Evaluation Statement of Work, Updated 2021, p. 7 3 Selected examples include the focus on GCs in the 1991 High Performance Computing Act; the DARPA sponsored Grand Challenges from 2002 onwards; and the National Academy of Engineering’s Grand Challenges for Engineering. See Pena, V and Stokes C 2019, “The Use of Grand Challenges in the Federal Government”, IDA Science and Technology Policy Institute 4 BMGF launched the Grand Challenges in Global Health program in 2003 to tackle specific diseases by bringing new entrants into the process of finding solutions, mobilizing multiple partners and pooling funding, and adopting an approach drawing on venture capital which accepted there would be failures among the innovations tested, and allowed grantees to pivot as needed. See National Academies of Sciences, Engineering and Medicine (NASEM) 2017, “The Role of Science, Technology, Innovation and Partnership in the Future of USAID” Washington D.C. The National Academies Press 5 IPE Triple Line interview with Alex Dehgan, USAID Chief Scientist in the Shah Administration, December 2020 1.1 INTRODUCTION USAID has been implementing Grand Challenges for Development (Grand Challenges or GCs) for ten years.1 As USAID looks to advance implementation and expand the use of open innovation competition methods, of which GCs are one type, USAID’s Exploratory Programs and Innovation Competition (EPIC) team (situated within the Innovation Division in the Innovation, Technology, and Research hub of the Bureau for Democracy, Development, and Innovation (DDI)) recognized the opportunity to systematically reflect on this experience to generate an actionable evidence base. As a result, EPIC contracted Triple Line to conduct a meta-evaluation across its GCs to “generate actionable strategy recommendations, useful to evaluation audiences [such as both USAID and partner GC funders and managers] and include practical measurement frameworks and strategic recommendations that can be used by USAID and partners to guide investment decisions and advance the design, management, and measurement of performance of GCs”.2 This report proceeds according to the following structure: Section 1 provides a brief overview of the context in which the USAID GC portfolio operates and the broad-stroke descriptions of the GCs themselves. Section 2 details the objectives and scope of the evaluation, and the principles underlying its design in response. Section 3 explains the methodologies utilized in data collection, analysis, and synthesis across all of the meta-evaluation’s activities. Section 4 systematically presents the findings of the meta-evaluation organized by Main Evaluation Question (MEQ). Section 5 collates the evaluation team’s conclusions and recommendations on the back of those findings. Finally, Section 6 provides a brief overview of the intended dissemination plans for both this report and other evaluation products. 1.2 CONTEXT GCs have been used by the U.S. Federal Government since the 1980s as a way of galvanizing interest around specific challenges and mobilizing a wide range of public and private sector actors to propose and test solutions.3 GCs were introduced to USAID as a model for development during the tenure of Administrator Raj Shah, formerly of the Bill & Melinda Gates Foundation (BMGF), where he had first seen the model adopted.4 The administration recognized the significant advances in science and technology within international development but acknowledged that USAID had not fully engaged with or utilized this progress. Under Administrator Shah, the Agency was given the mandate to “restore science, technology, and innovation in international development, focused on revolution rather than evolution, because while problems were scaling exponentially, solutions remained linear”.5 At the same time that technological advances were making new approaches possible, there was a growing awareness of changes in the development landscape. In 2009, a paper circulated by the USAID Office of Development Partners (a forerunner of the 2 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION U.S. Global Development Lab (the Lab) and DDI) foresaw a shift from a sector dominated by large bilateral and multilateral agencies, aid organizations, and charities to one in which a wider range of actors was engaged directly in tackling development challenges and innovation was actively encouraged and supported.6 Key components of the proposed new approach included: • Leveraging new solutions and technologies through new and collaborative approaches to problem solving involving multiple stakeholders with an emphasis on end-users. • Supporting entrepreneurship by creating an enabling environment for success, removing constraints, and providing targeted assistance to enable start-ups to become viable, sustainable economic entities. • Exploring partnerships with organizations outside of the U.S. Government in line with a greater commitment to open approaches to problem solving, including donor coordination and the sharing of tools and experiential knowledge within and beyond the Agency. • Purposefully leveraging a suite of innovative methods and programing to complement calls for innovation and building awareness of the specific challenges to be addressed, using both grants and a range of other funding mechanisms.7 • Exploring the potential to leverage funds and resources of other actors (such as non￾governmental organizations (NGOs), foundations, and private companies) focused on the same “wicked problems at the heart of USAID’s mission”.8 At USAID, GCs presented an approach which enshrined these principles in a single delivery model with the power “to disrupt traditional thinking in a sector and industry and introduce, expand and evolve what is possible in that industry”.9 From the outset, GCs incorporated an open innovation approach to “support the creation and diffusion of innovative solutions with transformational impact on major 6 “A Strategy for Innovation at USAID: Building USAID for the 21st Century”, USAID Office of Development Partners, December 2009 (draft) 7 Other funding mechanisms came to include prizes, massive open online courses, hackathons, pitch competitions, showcases, co-creations and results-based financing. 8 Ibid. 9 Alexis Bonnell, Former Division Chief of Applied Innovation at the Lab, quoted in Pena and Stokes op.cit. 10 IDA STPI 2020 “Powering Agriculture: Summative Evaluation” 11 NASEM op.cit. 12 Ibid. 13 Raetzell, L. and Seidler, M. 2016 “Program-Level Mid-Term Evaluation: Powering Agriculture: An Energy Grand Challenge for Development” quoted in IDA STPI op.cit. 14 Ferguson, D. July/August 2014 “Introduction: A Grand Challenge for Next Generation Solutions”, USAID, Frontlines 15 Dehgan later described the process as “instead of looking for the needle in the haystack, incentivize the needle to find you.” See Testimony of Dr. Alex Dehgan, now CEO of Conservation X Labs, before the House Science, Space and Technology Committee, subcommittee on Research and Technology, Wednesday, December 13th, 2017 international development problems”.10 The model was founded on the belief that science and technology, appropriately deployed, can drive transformation. Furthermore, since USAID does not have all the answers to key development questions and given that these questions are shared by many other individuals and organizations, “breakthrough progress” is possible by “engaging the world in the quest for solutions”.11 The role of public agencies would therefore be to foster the “development, adaptation, and scaling of innovations from multiple sources”.12 This thinking is clearly embodied in the Theory of Change (ToC) for one GC: “By engaging and mobilizing diverse, global solver communities, USAID and its Partners can source, select, incubate, test, and scale up science and technology innovations that will overcome critical barriers to development and accelerate the pace at which the world’s most pressing development problems can be addressed.”13 This new approach to innovation was regarded as radical at the time as it demanded changing the mindset around innovation in development and reversed the usual approach to designing development interventions by bringing a greater focus on understanding and defining the challenge to be addressed. “Typically, USAID technical experts go through a rigorous project design process that results in a specific solution for a particular development challenge. The GC approach turns this process on its head. With GCs we spend a lot of time defining the problem that needs to be solved but we don’t presuppose the solution”.14 Once the problem was defined, a GC did not specify a pathway to solutions, but focused instead on bringing together a wide pool of actors to propose innovations of their own.15 GCs allowed USAID to spotlight development challenges, foster innovative solutions, and encourage their uptake by: • Engaging a wide community of individuals and organizations around the world, not previously USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 3 engaged in the development community, to tap into their collective wisdom. • Mobilizing USAID’s convening power to focus other public agencies and donors on a particular development challenges, leveraging their funds and resources in conjunction with those of USAID. • Testing a range of solutions to identify those with the highest potential to succeed at scale; • Supporting entrepreneurs to commercialize their innovations by bringing working solutions to market, thereby seeding new markets for development solutions. • Operating with a higher tolerance for risk and allowing grantees to use funds more flexibly through milestone-based funding mechanisms. • Intervening at different stages of innovation, from initial piloting to early adoption, as well as laying the basis for future scaling and market growth.16 Above all, the GC model enabled USAID both to fund the development of new innovations by small organizations and to create the conditions for the long-term sustainability of solutions. Support was therefore not restricted to funding but extended to the provision of resources and services aimed at increasing the sustainability and impact of different innovations. This support included technical assistance (TA), consultative services to strengthen and improve business models and strategy, training, outreach services, the convening of communities of practice and other forums, knowledge dissemination through events and other media, and providing awardees with access to follow on funding either through public or private investment channels.17 Since the launch of the GC program, USAID has created designated spaces for innovation within the agency, such as the Global Health Bureau’s (GH) Center for Innovation and Impact (CII), the Lab (see below), and the newly formed Bureau for 16 IDA STPI op.cit. 17 Ibid. 18 The Lab now continues its work in the newly formed Democracy, Development and Innovation Bureau (DDI), where the Innovation, Technology and Research hub continues to advance the goals first formulated in the Lab and works across geographies and sectors to support Agency uptake of innovative program design and solutions. 19 IPE Triple Line interview with Wendy Taylor, former Director of the CII, December 2020. 20 DIV supports a range of actors to address development challenges. Unlike the GCs, however, which are focused on a specific, well defined challenge, DIV “Accepts applications year-round for innovations in every sector and country in which USAID operates”. See DIV 10 Year Brochure 2020, “Development Innovation Ventures: Turning Bright Ideas into Global Solutions’. 21 The Lab had its antecedents in the IDEA and Office of Science and Technology teams, running since 2008/2009 and subsequently merged into the Lab structure. 22 https://www.usaid.gov/GlobalDevLab 23 SL@B was launched under the leadership of Wendy Taylor, who noted key differences between the BMGF GC model, and that of SL@B: ‘Most of the [BMGF] Grand Challenges had been focused on technologies. They had their exploration, but they were pretty narrow so we opened the aperture a little wider. We were looking at service delivery solutions. We designed [the model] so there was a whole range of ways people could come in and try to solve the problem - service delivery, devices, technology: it was really agnostic as to where the solutions came from, focusing on the problem’. 24 Ibid. 25 SL@B provides an example of staged funding, with seed funds to developing new ideas with potential for positive impact on health outcomes for pregnant women and their babies; validation funds to establish proof of concept of these innovations; and transition to scale funds to test the potential for solutions to scale in a wider setting (https://www.usaid.gov/global-health/health-areas/maternal-and-child-health/projects/saving-lives-birtha-grand-challenge) Humanitarian Affairs Innovation Team, to catalyze and complement the previous solely vertical model of channeling innovation through individual Operating Units (OUs).18 The intent was for these spaces to act “as a catalyst for experimentation, ensure USAID remained in step with advances in technology, serve as a pathbreaker and ‘cheerleader’ for innovation across the Agency, and play a key role in institutionalizing new mechanisms and sharing learnings”. 19 The Lab, itself home to both Development Innovation Ventures (DIV)20 and the portfolio of some of the GCs under review in this evaluation, was set up in 2014.21 Its mandate was to “produce breakthrough development innovations” and to accelerate development by “opening development to people everywhere with good ideas”,22 working with other parts of the Agency to share lessons from the GCs, and supporting the uptake and implementation of innovative ideas emerging from the different programs. In consultation with internal and external experts, the first GC, Saving Lives at Birth (SL@B), was launched in 2011. SL@B, housed in GH, focused on maternal and newborn health and embodied key principles of open innovation:23 the problem was clearly and discretely defined (targeting the first 48 hours during and after birth) while the scope of solutions was kept open, allowing for “a whole range of ways people could come in and try to solve the problem”.24 In addition, the GC was launched with external partnerships and funding, which enabled the program to offer staged grants and help innovations transition to scale (TTS).25 Since 2011, USAID has launched eleven GCs, nine of which are covered in this evaluation. These nine GCs represent a total investment of $463.4M from USAID and other donor partners. 15,966 applications have been received from innovators with 767 awards made. The GCs fund projects in 4 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 83 countries and have 94 global projects covering seven different development sectors. In 2020, USAID launched its most recent GC, Water and Energy for Food (WE4F), building on the accumulated learning from a decade of implementing open innovation through GCs. Further detail on the scope and objectives of the individual GCs is presented in the next section with a full portfolio mapping available in Annex 1. 1.3 OVERVIEW OF THE GRAND CHALLENGES The following section provides a brief summary of each GC included in this evaluation. A more detailed portfolio analysis can be found at Annex 1. 26 Between 1995 and 2014, Australian Government international development assistance was led by the Australian Agency for International Development (AusAID). In 2014, AusAID was merged into the Australian Department for Foreign Affairs and Trade (DFAT). In this report, ‘DFAT’ refers to both DFAT and AusAID involvement. 27 As detailed in Section 2, the GC Ensuring Effective Health Supply Chains (EEHSC) has not been considered as part of this evaluation. This was due to its comparatively small size in terms of budget and grantees, a lack of data, and the fact that it was primarily operated by the Bill & Melinda Gates Foundation with only limited USAID involvement. 28 While many GCs were open to applicants from around the world, this column is intended to highlight where the majority of awardees are concentrated. 29 The figures for SOGE are likely underestimated as they do not include the number of applicants for Rounds 1 and 3 nor the number of awards for Round 3 since providers were not required to record the number of applicants and due to partial data respectively. ALL CHILDREN READING: A GRAND CHALLENGE FOR DEVELOPMENT Launched in 2011, All Children Reading: A Grand Challenge for Development (ACR GCD) is a $29.0m GC (of which USAID contributed $14.0m) in partnership with USAID’s DDI, World Vision, and the Australian Department of Foreign Affairs and Trade (DFAT).26ACR GCD aims to advance EdTech innovation and research in order to improve reading outcomes for marginalized children in low resource contexts. There have since been three rounds of grant funding and ten rounds designed for smaller, more thematically and geographically targeted prize awards for a total of 92 awardees. TABLE 1.1: GC OVERVIEW27 NAME TOTAL BUDGET (USD, MILLIONS) SECTOR START DATE END DATE NO. OF AWARDS GEOGRAPHIC CON￾CENTRATION28 All Children Reading: A Grand Challenge for Development 29.0 Education 2011 Ongoing 92 Global Combating Zika and Future Threats 30.0 Health 2016 2018 26 Latin America and Caribbean Creating Hope in Conflict 39.9 Humanitarian 2017 Ongoing 50 Sub-Saharan Africa, MENA Fighting Ebola 8.9 Health 2015 2015 14 West Africa Making All Voices Count 45.0 Civil Society Engagement and Governance 2012 2016 178 East Asia and Pacific, Sub￾Saharan Africa Powering Agriculture 51.8 Agriculture / Energy 2012 2019 24 Sub-Saharan Africa Saving Lives at Birth 100.0 Health 2011 2020 147 Global Scaling Off-Grid Energy29 124.5 Energy 2016 2020 196 Sub-Saharan Africa Securing Water for Food 34.3 Agriculture / Water 2013 2020 40 Sub-Saharan Africa, MENA, South and Southeast Asia TOTAL 463.4 - - - 767 - USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 5 COMBATING ZIKA AND FUTURE THREATS USAID launched the Combating Zika and Future Threats Grand Challenge (Zika) in April 2016 to invest up to $30.0m in innovations designed to prevent, detect, and respond to the vector-based Zika virus and build capacity to combat future infectious disease outbreaks. Due to its thematic and geographic specificity, Zika only had one funding round. It disbursed its entire budget to 26 awardees, including research institutions, commercial players, and civil society organizations, and provided funded acceleration support. CREATING HOPE IN CONFLICT Creating Hope in Conflict (CHIC) is a $39.9m GC (of which USAID has contributed $12.4m). Launched in 2017, CHIC is a jointly funded program between USAID, the Dutch Ministry of Foreign Affairs (BZ), the UK Foreign, Commonwealth, and Development Office (FCDO), and Global Affairs Canada (GAC). It is managed by Grand Challenges Canada (GCC).30 The GC calls on innovators around the world to submit ideas to save and improve the lives of the most vulnerable and hardest-to-reach people affected by humanitarian crises caused by conflict. It is thus geographically limited to fragile and conflict￾affected states such as the DRC, South Sudan, Yemen, and Syria. CHIC has also supported interventions in refugee camps in countries such as Kenya, Turkey, and Jordan. The GC has made 50 awards so far. FIGHTING EBOLA Fighting Ebola (EBOLA) was an $8.9m program funded by USAID in partnership with the Center for Disease Control (CDC), the Department of Defense (DoD), and the White House. Set up in 2014 in response to the 2014 West African Ebola epidemic, EBOLA sought to fund cost-effective, scalable, and practical solutions to improve the provision of personal protective equipment (PPE) and tools for healthcare workers in Ebola-hit contexts. EBOLA made a total of 14 awards and hosted hackathons, co￾creation sessions, and additional acceleration support. MAKING ALL VOICES COUNT Running between 2012 and 2016, Making All Voices Count (MAVC) was a $45.0m GC (of which USAID contributed $10.0m). A partnership 30 From 1997, UK Government international development assistance was led by the Department for International Development (DFID). In September 2020, DFID merged with the Foreign and Commonwealth Office (FCO) to form the UK Foreign, Commonwealth and Development Office (FCDO). In keeping with UK Government guidance, ‘FCDO’ refers to both FCDO and DFID involvement in this report. between USAID, FCDO, Omidyar Network, and the Swedish International Development Agency (Sida), MAVC aimed to 1) support civil society in-country to deliver feedback to national or local government and engage in policy decision￾making and 2) build governmental capacity to translate that feedback into action and to support government initiatives/solicit civil society engagement. The GC also had a strong learning component so as to produce an evidence-base around what works in accountable governance for policy makers and civil society actors. Receiving 2,849 applications over three rounds, MAVC supported 178 grantees. POWERING AGRICULTURE: AN ENERGY GRAND CHALLENGE FOR DEVELOPMENT Powering Agriculture: An Energy Grand Challenge for Development (PAEGC) was a $51.8m GC (of which USAID contributed $15.4m). The partnership between USAID, Sida, Duke Energy, and the German Gesellschaft für internationale Zusammenarbeit (GiZ), with USAID also running the GC, operated between 2012 and 2019. It sought to catalyze resources on addressing many farmers’ and agribusinesses’ lack of access to reliable, affordable, and clean energy in low- and middle-income countries (LMICs), which hindered their ability to adopt modern agricultural practices, increase food production, and improve efficiency. PAEGC funded 24 projects, mostly in Sub-Saharan Africa, across two funding rounds. The new Water and Energy for Food GC continues the work of both PAEGC and Securing Water for Food (SWFF) (below). SAVING LIVES AT BIRTH Launched in 2011, Saving Lives at Birth (SL@B) was a $100.0m partnership between USAID, FCDO, the Norwegian Agency for Development Cooperation (Norad), the Korea International Cooperation Agency (KOICA), the BMGF, and GCC. USAID acted as the coordinating partner in the consortium. SL@B sought interventions to save lives around the time of childbirth and the early postnatal period for women and newborns in underserved, disadvantaged, and marginalized communities around the world. SL@B made a total of 147 awards to 116 innovations over eight open innovation challenge rounds. 6 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION SCALING OFF-GRID ENERGY Launched in 2016, Scaling Off-Grid Energy (SOGE) was a $124.5m GC (of which USAID contributed $11.2m) comprising a partnership between USAID, Power Africa, FCDO, Shell Foundation, and the African Development Bank (AfDB) with USAID managing and running the GC secretariat. It sought to accelerate access to clean, affordable energy for millions of households and businesses across Sub-Saharan Africa. SOGE supported early-stage companies and financial intermediaries, drove technology and business model innovation, and grew the market ecosystem for off-grid energy. SOGE made 182 awards to 77 companies and market enablers across 23 countries, and has made 14 awards to date through the follow-on Solar E-Waste and Household Solar Workforce Development Challenges. The partnership model, now organized 31 DSI was previously known as the Department of Science and Technology (DST). under the Household Solar Funders Group representing a coalition of 36 partners, is being carried forward by Shell Foundation. SECURING WATER FOR FOOD Running between 2013 and 2020, Securing Water for Food (SWFF) was a $34.3m GC (of which USAID contributed $12.1m) comprising a partnership between USAID, Sida, the South African Department of Science and Innovation (DSI)31, and BZ with USAID running the GC. SWFF provided acceleration support to promote solutions that enable the production of more food with less water and/or make more water available for food production, processing, and distribution. Through its 40 supported projects, SWFF impacted over seven million beneficiary farmers and other customers. The new Water and Energy for Food GC continues the work of SWFF and PAEGC (above). FIGURE 1.1: TIMELINE OF THE GC PORTFOLIO UNDER STUDY 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 SL@B ACR GCD MAVC PAEGC SWFF EBOLA Zika SOGE CHIC USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 7 2 EVALUATION OBJECTIVES AND SCOPE 32 Evaluation Statement of Work, Updated 2021, p. 7 33 Other GC activities, such as prizes, hackathons, commit fairs, massive open online courses, conferences, and co-creation workshops are also a purposeful part of the GC approach, although too varied to be consistently analyzed in this meta-evaluation. 34 The WE4F GC was launched after the start of this meta-evaluation and was thus never expected to be investigated. 35 https://drive.google.com/file/d/1AEGr7t0UX0j51r-W7wC8iu4IxnJ5qJEx/view?usp=sharing 36 The prioritization process was also informed by the evaluation ‘Kick-Off Call’ which included an initial analysis of priority areas of interest, and which fed into the refinement of the evaluation questions. The evaluation objective was to ‘generate actionable strategy recommendations, useful to evaluation audiences [such as both USAID and partner GC funders and managers] and include practical measurement frameworks and strategic recommendations that can be used by USAID and partners to guide investment decisions and advance the design, management and measurement of performance of GCs’.32 Triple Line has thus taken a utilization-focused approach, that is, designed and executed the evaluation for its primary intended users and uses. The primary audience is decision makers who fund and manage GCs, including USAID staff and OUs involved in managing GCs, USAID’s EPIC team, USAID leadership and Mission staff, and GC partner and donor staff who co-fund and/or manage GCs. Secondary evaluation users include other policy and decision makers considering the use of open innovation challenges, for example, the governments of LMICs. The evaluation examined nine GCs that USAID has engaged as a partner as set out in Section 1.3. The evaluation was designed to respond to the objectives, scope, and parameters set out in the terms of reference (ToR). While GCs conducted a range of activities including prizes, hackathons, commit fairs, massive open online courses (MOOCs), conferences, results-based financing, and co-creation workshops, this report is centered on the challenge fund and acceleration aspects of the GC model.33 Following discussion with USAID and Catalyst (USAID’s managing agent for the evaluation) during the inception phase, departures from the ToR relate to the following: • Scope of GCs covered: This evaluation considers nine of USAID’s ten GCs.34 While Ensuring Effective Health Supply Chains (EEHSC) was one of the GCs set out in the evaluation Statement of Work (SoW),35 an assessment of the feasibility of responding to the evaluation questions during a prioritization exercise highlighted concerns around data availability, particularly with regards to results achieved. In follow-up discussions with USAID, further challenges were raised around the opportunities for the evaluation team to draw robust learning from this program as USAID’s participation was extremely limited and there was a lack of data around the award. The meta-evaluation has thus not included EEHSC. • Prioritization and refinement of evaluation questions: As set out in the SoW, the evaluation team reviewed the feasibility of answering all evaluation areas of interest and questions across all of the GCs based on an assessment of availability and accessibility of data. This formed the basis of our ‘Prioritization Memo’ (deliverable 2)36 which concluded that ‘across the [nine] GCs in the portfolio there will be sufficient available and usable data to answer questions related to all ten Areas of Interest identified in the original Statement of Work’. • COVID-19 adaptations: As a result of the COVID-19 pandemic and bans on international travel, the evaluation team made alterations to its proposed data collection methodologies set out in the proposal. Details are provided in Section 2.1.2. 2.1 EVALUATION DESIGN AND QUESTIONS The evaluation team adopted a mixed-methods approach to the evaluation. Data was collected and analyzed through a portfolio mapping exercise, an extensive document review of GC-specific and higher-level strategic documentation, key informant interviews (KIIs) with a wide selection of actors in the stakeholder ecosystem (see Annex 7), grantee case studies, a grantee survey, a cost effectiveness analysis (CEA) review and framework exercise, comparator analysis, a gender and social inclusion (GESI) review of the GC portfolio, several internal co-creation and validation workshops, and external expert input. Further details are provided in Sections 3.1 and 3.2. 8 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION FIGURE 2.1: GENERIC GC INTERVENTION MODEL PROGRAM DESIGN IMPLEMENTATION ACHIEVEMENTS Shared PROGRAM OBJECTIVES (among partners) ‘Ecosystem’ development support USAID/ partner Mission & other Operating Units Thematic Objectives Theory of Change Result framework Budget Governance arrangements Stages of innovation Scaling objectives Geographic focus Ecosystem support Linkage with other Program Engagement of related agencies FUND MANAGEMENT Design of calls Pipeline and launch Selection Contracting Funds and Tech support Comms & Outreach M&E and Learning SUSTAINABLE OUTCOMES AND IMPACT AT ‘SCALE’ • Raised awareness • Enabling environment PROGRAM APPRAISAL & DESIGN (including context analysis & problem identification) 37 The order of the MEQs presented in this report and in Table 2.1 has been chosen to aid accessibility and flow in accordance with the groupings determined by the OEQs. Responsibility for data collection, capture, and analysis was distributed amongst the evaluation team by GC with standardized data capture and analysis templates utilized to ensure consistency. Developed with feedback from USAID GC policy and program managers, the evaluation team further standardized its approach by utilizing a generic GC intervention model as a baseline framework for appraisals of the contextually and structurally diverse GC portfolio (see Figure 2.1). The results of those appraisals then informed the construction of descriptive profiles of each of the GCs in order to surface how design and implementation factors supported or hindered the achievement of GC objectives and to enable effective comparison between the diverse GCs. Findings were then compared and contrasted in full to draw out overarching conclusions and recommendations in consultation with Catalyst and with USAID. Over the course of this process, the evaluation team organized the ten MEQs under two Overarching Evaluation Questions (OEQs). The two OEQs are: • OEQ 1: To what extent has the implementation of the Grand Challenge approach led to the effective delivery of results and achievement of development and humanitarian outcomes? • OEQ 2: To what extent has the Grand Challenge approach been an appropriate method for achieving sustainable results at scale and strengthening innovation ecosystems? The first of these OEQs focuses on learning about the suitability of the Grand Challenge model for achieving results and sustainable impact at scale (which relates to the design stage in the intervention model), and the second focuses on learning about how the model has been implemented and the effect it has had on the delivery of results. The OEQs frame the evaluation conclusions and all MEQs were used to answer them. The evaluation team also organized the Sub￾Evaluation Questions (SEQs) into retrospective and prospective groups, the latter renamed as Forward￾Looking Questions (FLQs). The MEQs answered in this evaluation organized by OEQ can be found in Table 2.1 with a full list of SEQs and FLQs in Annex 4. TABLE 2.1: MAIN EVALUATION QUESTIONS37 MEQ NO. AREA OF INTEREST EVALUATION QUESTIONS 1 Retrospective Assessment of GC Results What results have been achieved and sustained across GCs? What factors support, inhibit, and explain the success and sustainability of GCs and their innovations? 2 Scaling Innovations Which GC innovations have reached scale (either for sustainability and/or for impact) and what strategies and factors contributed to or inhibited scaling? USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 9 MEQ NO. AREA OF INTEREST EVALUATION QUESTIONS 3 Acceleration Which GC acceleration strategies are most effective and at what stages of innovation? 4 Investing in Ecosystems How effectively have GCs invested in ecosystems strengthening and what have been the results of these investments? 5 Catalytic Effects To what extent have GCs catalyzed funding, development of other solutions, follow-on funding, and awareness? 6 Procurement and Reducing Barriers to Funding for Non-Traditional USAID Partners38 To what extent are GCs accessible to all types of innovators? 7 Engaging USAID and Partner Missions39 and OUs To what extent have GCs effectively engaged USAID and partner Missions and OUs, and what lessons can be learned from engagement efforts to date? 8 Measuring Impact, Results, and Uptake How were results, impact and program effectiveness measured across GCs? 9 Measuring Cost Effectiveness How feasible is it to measure the cost effectiveness of previous and future GCs and to compare the cost effectiveness of GCs to traditional program models? 10 Governance, Partnership Models, and Operational Effectiveness How have GCs been managed/governed differently and what models can be described? What are the advantages and disadvantages of different models? 38 For the purposes of this evaluation, a non-traditional USAID partner is defined using USAID’s definition of a ‘new partner’, i.e. “An individual or organization that has not received any funding from the U.S. Agency for International Development (USAID) as a prime partner over the last five years”. 39 The inclusion of ‘partner’ Missions and OUs follows an earlier suggestion from USAID and Catalyst to better reflect the reality of how a number of GCs are implemented. This represents a change from the initial ToR. The evaluation team also conducted a series of analytical exercises in collaboration with USAID and external experts to provide additional evidence sources for the evaluation and to serve as standalone products (insight memos) for audiences within USAID. They are included in the report annexes and comprise: • A comparator analysis of other open innovation schemes operated by USAID and other donor partners (Annex 9). • An insight memo on GESI (Annex 13). • Grantee case studies (Annex 14). A CEA framework and CEA review have also been produced and have fed into the findings in this report (Annexes 10 and 11). Some of these will form the basis of short knowledge products for dissemination by USAID as set out in Section 6. 2.1.1 GENDER EQUALITY AND SOCIAL INCLUSION (GESI) The evaluation team has adopted a GESI lens throughout data collection and analysis. The GESI insights memo (Annex 13) highlights how GESI has been considered at each stage of the GC lifecycle across the GC portfolio as a cross-cutting issue. The memo outlines successes, potential areas of improvement, and insights in relation to GESI in GC design, implementation, and results. 2.1.2 COVID-19 ADAPTATIONS The ongoing COVID-19 pandemic has required the evaluation team to adjust its approach from what was originally envisaged. Government￾imposed travel and contact restrictions did not allow for international travel and face-to-face contact, disrupting plans for case study research and preventing opportunities to meet USAID stakeholders. The following adaptations were made: • Case studies were conducted remotely and were not country specific. (See Section 3.1.4 for details). • All KIIs were conducted remotely using the informant’s choice of web-enabled conferencing tool. Questions were shared in advance and scheduling was flexible to minimize demands placed on the informant. • Remote workshops were conducted during the analysis, synthesis, and validation stages and made use of a range of virtual tools such as the Miro virtual whiteboard and MentiMeter real-time polling platform to increase engagement and interactivity. 10 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Photo: Global Devlab/USAID USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 11 3 EVALUATION METHODOLOGY 40 The GCs for which interviews were conducted included EEHSC and excluded MAVC which closed in 2017. It was partially on the basis of information from KIIs conducted for EEHSC that it was decided to exclude it from analysis due to its comparatively low amount of funding, small number of grantees for which USAID was responsible, and limited data availability. 3.1 DATA COLLECTION The evaluation team made use of a range of data sources as evidence on which its findings, conclusions, and recommendations are based. To facilitate data collection, the evaluation team compiled a data collection toolkit (see Annex 6) which ensured that the approach between evaluation team members was consistent and that the data collected was both relevant and useful. A full description of the evaluation methodology can be found at Annex 3. 3.1.1 PORTFOLIO MAPPING During the inception phase of the evaluation, a portfolio mapping exercise was conducted to inform a prioritization exercise to determine which grantees should be contacted for the grantee survey. This exercise involved reviewing grantee-level documentation to fill in data gaps in collaboration with USAID. Analysis was also conducted according to other relevant variables including call type/mechanism, applicants, awards, and funding catalyzed, and a geographical heatmap was created to visually represent the distribution and concentration of grantees around the world (Annex 1). 3.1.2 DESK REVIEW Responsibility for data collection, capture, and analysis for each GC was distributed among evaluation team members. The first stage of this process was to review an extensive set of program and strategic level documentation sent to the evaluation team by USAID GC Managers – 2,397 documents in total (see Annex 8). The team proceeded to select key documents relating to design, implementation, and results by GC and logged key findings in a standardized data capture template (one per GC reviewed) using one row per data source to enable clear traceability of evidence sources. In total, the team extracted data and evidence from 502 documents. 3.1.3 KEY INFORMANT INTERVIEWS (KIIS) KIIs were a critical plank of data collection for this meta-evaluation. During implementation, a total of 64 different people across nine GCs were interviewed.40 Interviews augmented available evidence from the document review, collected and gave voice to different perspectives, and generated qualitative insights to triangulate data against other sources of evidence. The KIIs yielded a wide range of insights into GC performance, design, implementation, lessons learned, and best practice. The detailed notes taken during each KII have collectively formed a core data set on which the evaluation team’s conclusions and recommendations are based. A full list of KIIs can be found at Annex 7. 3.1.4 CASE STUDIES Case studies were planned with the objective of digging deeper into some of the thematic priorities that emerged during the meta-evaluation. Initiated towards the end of the process, the purpose was to document key learnings from selected awards which could be used to inform both the evaluation and other challenges in the future. Three key thematic priorities were identified: scaling up and sustaining the innovation, partnering with USAID Missions and in-country governments, and using MEL effectively for acceleration and scaling. The evaluators asked USAID GC Managers to shortlist awards that would offer good learning in these three thematic areas. Eight awards were identified from five GCs. All were approached and five responded. The awards were studied in detail through a desk review of available reports and documents which was followed by an interview with the award manager(s). The qualitative data compiled from all sources was collated and analyzed to develop the case studies. 3.1.5 GRANTEE SURVEY The evaluation team conducted an anonymous online survey with current and recently completed grantees. This served as a data source and evidence base to feed into the findings and conclusions for MEQs 1, 2, 4, 7, 8, 9, and 10. Further details of the 12 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION survey questionnaire can be found in Annex 6, Section A6.4, and a summary of survey results can be found in Annex 15. Since data on grant end dates was inconsistently available, the evaluation team focused on projects that had started in 2017 or later. The projects surveyed corresponded to about 11 percent of the overall GC portfolio. Responses were received from 54 grantee organizations and at least one grantee responded from each of the nine GCs to whom the survey was sent.41 Only one response was received from ACR GCD, but there were at least five respondents from each of the other GCs. 3.1.6 COST EFFECTIVENESS ANALYSIS (CEA) REVIEW AND FRAMEWORK A review of the feasibility of conducting a portfolio￾level CEA in the future was originally planned to be disseminated as a standalone product of the meta￾evaluation in relation to the MEQ on this topic. It was determined early on in the evaluation, however, that there was insufficient data to conduct a meaningful CEA as data on activity costs and beneficiary-level outcomes had not been collected systematically across the GC portfolio. A brief review of CEA methodologies and practices by USAID and other donors was therefore undertaken to help guide the development of a forward-looking CEA framework which will be disseminated as a standalone document (see Section 6). Some of its key findings have been integrated into Section 4. The CEA review and framework, including a full methodology, can be found at Annex 10 and Annex 11. 3.1.7 COMPARATOR ANALYSIS The comparator analysis was used to address the following questions related to the rationale for selecting the GC mechanism for programing rather than other development intervention mechanisms: • What is the value of putting funds into GCs rather than other program types? • Do GCs generate a greater range of innovative solutions than other programs? • What additional value do GCs bring compared with other programs? • Are there constraints associated with GCs which other programs manage to avoid? The evaluation team identified four programs whose analysis would address these questions by focusing 41 The survey was not sent to MAVC grantees as this fund closed in 2017. on the specific differences between GCs and more traditional programs which operate in the same sectors or have comparable delivery objectives. These four comparator programs were derived from discussion at the Prioritization Workshop held at the end of July 2020 and subsequent discussions with USAID and Catalyst. The comparators were: • The Innovation Investment Alliance (IIA): A global funding and learning partnership between USAID and Skoll Foundation to enable not-for-profit social enterprises to take proven innovations to scale. Sample projects include Lista by Fundacion Capital, a $1.9M grant to build financial capabilities for those living in poverty across seven countries in Latin America, and Evidence Action, a $2M grant to provide safe drinking water dispensers in Uganda. • Partnering for Innovation: A global program aimed at selling innovative products and services to smallholder farmers at the base of the pyramid (BoP). The program targeted both small entrepreneurs and large corporations with sample partnerships, including Oiko Credit in Peru, which provides finance and business skills to smallholder coffee producers, and ATEC in Cambodia, which provides biodigesters to farmers to make their own organic fertilizer. • Local Works: A global program to make development more locally-led and locally specific, Local Works funds Missions to adapt and localize existing programs in line with their objectives, enabling them to transfer leadership to local entities or create new, locally owned programs. Sample initiatives include working with local actors in Bangladesh to address the lack of resources for Rohingya refugees and linking local actors with national policy in the Dominican Republic to address high rates of poverty and crime along the border with Haiti. • Sustaining Health Outcomes through the Private Sector Plus (SHOPS Plus): A flagship global program that aims both to build a global evidence base for private health sector engagement and strengthen local health systems through the engagement of private health actors and innovations in countries around the world. Areas of focus include family planning and reproductive health, maternal, newborn and child health, tuberculosis (TB), and HIV/ AIDS by improving service delivery models, strengthening business sustainability and access to finance, developing partnerships, and conducting USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 13 research.42 Sample “innovation” activities include developing an application called TB Star in Nigeria to aid in the diagnosis and treatment of TB and piloting a micro-insurance product for those living with HIV/AIDS in Tanzania. The rationale for selection can be found in Annex 9. The comparator analysis was conducted through a mixture of KIIs and document review. Documents which aligned with the GC desk review were prioritized. Between the KIIs and document requests, the evaluation team engaged with comparator points of contact (PoCs) over three rounds of data sourcing. 3.1.8 GENDER AND SOCIAL INCLUSION (GESI) REVIEW The GESI review was used to address questions about how GESI had been integrated into GCs at each stage of the intervention, so that evidence could be integrated into MEQ findings where appropriate. The review was structured by design, implementation, and results, the three themes of the GC intervention model. The review analyzed evidence on GESI captured for the meta-evaluation through desk review and KIIs to reach its assessment and conclusions. It used this to make a set of recommendations for how GCs might strengthen their integration of GESI into programing in the future. Further detail can be found at Annex 13. 3.2 DATA ANALYSIS AND SYNTHESIS The intervention model analytical framework was a tool which was developed to interrogate the design, implementation, and results of each of the GCs. The evaluation team developed this tool to ensure consistency in the review of the GCs against which each of the MEQs would be assessed and to guide data collection and capture. The analytical framework is set out in Annex 5. As a first step in the analytical process, the evaluation team prepared descriptive profiles of the design, implementation, and results of each GC. These were shared with relevant PoCs for validation and to ensure factual accuracy. The descriptive profiles enabled comparative analysis of the basic characteristics of each GC and the identification of commonalities and differences. 42 For detail on SHOPS Plus initiatives to combat HIV/AIDS and TB specifically, see https://www.shopsplusproject.org/healthareas/hiv and https://www. shopsplusproject.org/healthareas/tuberculosis Three analytical frameworks (organized by GC, MEQ, and SEQ/FLQ) were then used to capture findings from the various data collection exercises listed in Section 3.1. Their standardization ensured both consistency of approach across team members and preserved evidence traceability by ensuring that each judgement was justified with evidence and examples. Annex Table 5 shows how the GC analytical framework was utilized. The analytical frameworks can be found here. As well as individual analyses, two workshops were convened in order to share findings and preliminary conclusions among evaluation team members. The first was limited to evaluation team members while the second included Catalyst in order to articulate and receive early feedback on draft conclusions and recommendations. Findings and recommendations were also subsequently discussed with USAID. 3.3 LIMITATIONS While the evaluation team has gone to great lengths to ensure that all of its findings, conclusions, and recommendations are evidence￾based, there have nevertheless been limitations on the extent of the conclusions drawn. Some of these limitations are the result of unavoidable information gaps. Similarly, interviewees are able to relay only the information they can recall, a fact that betrays both a degree of recall and selection bias and the long period of time over which GCs have been implemented. The evaluation team has also encountered significant data gaps during portfolio mapping and quantitative analysis exercises. Catalyzed funding figures, for instance, are only incompletely available across the GCs, since funding catalyzation was not an explicit objective for a number of GCs (such as EBOLA and Zika). There have also been gaps in the monitoring of grantee and applicant characteristics across the GC portfolio, leading to an incomplete picture, most significantly, of: • LMIC country applicants. • Grantee and applicant organization types. • First-time grantees and applicants. • Women grantees and applicants. 14 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION The evaluation team has included appropriate qualifications and caveats where necessary to reflect these limitations since ensuring that claims are evidence-based is any evaluation’s first priority. The scope of Section 4.6 was limited to USAID Missions, and did not cover donor partner missions (such as FCDO or DFAT missions), since there was insufficient documentary or KII evidence on this topic, the latter due in part to the wide range of questions needing to be covered in limited time. The evaluation team has endeavored to fill in any gaps through other means where possible. For instance, data found through web trawls has been used to determine procurement time scales across the GC portfolio (see Section 4.6.1), the end dates for many grantee funding periods were sourced through an extensive review of grantee-level contract documentation, and two rounds of follow￾up emails have been sent to comparator program managers. Photo: Global Devlab/USAID USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 15 4 FINDINGS 4.1 RETROSPECTIVE ASSESSMENT OF GC RESULTS MEQ 1: WHAT RESULTS HAVE BEEN ACHIEVED AND SUSTAINED ACROSS GCS? WHAT FACTORS SUPPORT, INHIBIT, AND EXPLAIN THE SUCCESS AND SUSTAINABILITY OF GCS AND THEIR INNOVATIONS? Summary: Five of the nine GCs met or exceeded their target achievements. For these five GCs there is sufficient evidence to confirm that the results are sustainable.43 There are five key common success factors which contribute to the achievements: • GC design and objective setting informed by a strong understanding of context. • GC implementation includes learning and adaptation. • GCs understand and provide support for pathways to scale. • Effective acceleration and scaling support. • Coordinated engagement by all donors. There are notable successes in agriculture, water management, and health, but this does not mean that the GC model is only suited to these sectors. In some cases, the GC model offers the ability to deliver results quickly, for instance in the cases of fast moving and emergency health crises (Zika and EBOLA). Whether moving fast or slow, a sound understanding of the local context is critical for success. The success of the higher performing GCs is a result of good design elements (listed above) and a mix of GC instruments supporting a balance of early- and later-stage innovations on the pathway to scale. GESI results were clear for only three GCs. The evidence from the GCs is that early-stage innovations (i.e., the first three stages of ideation, research and development, and proof of concept) have worked particularly well in the context of testing known tools and processes and introducing them to new contexts in LMICs. This success must be considered against a general inconsistency across the GCs in the extent to which outcomes have been adequately defined and expectations set. Recommendations: GCs should undergo complete and rigorous design, involving effective USAID engagement with donors and partner alignment around the GC objectives, taking the success factors above into account. GCs should focus on achieving results by supporting early- and mid-stage innovations and supporting them on the pathway to scale, and by strengthening developmental outcomes by embedding GESI principles into design and differentiating beneficiaries when setting objectives and measuring results. 43 Meaning that outcomes are maintained after USAID funding has concluded – see findings following Table 4.1. There is considerable diversity across each GC in the size and number of innovations funded. The GC model has also been leveraged to meet different objectives (e.g., coming up with immediate, short-term solutions during the Ebola epidemic or attempting to foster system-level governance and accountability improvements under MAVC). This diversity can make it difficult to consistently and effectively measure the achievement and sustainability of results across GCs. There has also been inconsistency in how targets were set and results measured which can influence the assessment of the performance of the GCs, as can fund maturity. Five of the nine GCs have met or exceeded the targets set and there is sufficient evidence to confirm that their results are sustainable. For this evaluation, the overall achievement of results from the nine GCs has been assessed in terms of how each of these GCs has performed against the expected outcomes and targets set. There are, however, GCs (EBOLA, SOGE) where 16 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION the outcomes were not defined sufficiently, signaling a need across the GCs for an increased emphasis on setting clear, realistic targets and measuring the achievement of results. Success of the GCs has been measured in terms of whether the objectives of the challenge and planned outcomes were achieved and sustained, based on available data. Five of the nine GCs produced results that achieved or exceeded expectations (). One GC was rated as having weak performance (--). Three GCs could not be rated with confidence due to a lack of clarity about objectives and expected outcomes against which to rate them, and/or data availability, and are thus unrated (~). The sustainability of results achieved is best considered in terms of whether the specific GC’s outcomes have been maintained after USAID grant funding has concluded. For example, increased smallholder farm incomes from PAEGC or SWFF, increased literacy from ACR GCD, improved neonatal health indicators for SL@B, or improved government service delivery for MAVC. This can be difficult to measure given that projects (both GC-funded and others) rarely allocate funding for monitoring and evaluation after grants or contracts have concluded. TABLE 4.1: KEY GC RESULTS GC SECTOR RATING EXPECTED RESULTS AND COMMENT 1 CHIC Humanitarian  2.2M end users have access to improved humanitarian products or services against an outcome target of 1.07M. 639 intermediaries using or implementing improved humanitarian products or services against a target of 423. Early signs of success at scale: Hala Systems’ early warning system against airstrikes. 2 Zika Health  The program surfaced leading technologies that could be used for protection/diagnostics for improved public health from the 26 innovations funded. A high-risk innovation program that achieved what it set out to do. Significant successes at scale: The World Mosquito Program (WMP) and Premise Data both leveraged investment and have been taken to scale and expanded to countries beyond the GC funding. 3 PAEGC Agriculture/ Energy  Broad outcomes achieved but no clear quantitative targets set. There is evidence that the GC has contributed to the development and sustainable use of appropriate community scale technologies for the uptake of clean energy for agriculture. GC reached 234,100 beneficiaries of which 18,230 were trained on the use of clean energy systems and 54,440 increased their knowledge of clean energy technologies. Provided 417 instances of TA on business acceleration needs and provided 288 referrals, linkages, and partnership connections to investors, funders, donors, partners, and other stakeholders. The Investment Alliance also invested $1M in selected companies and identified 56 additional potential investments. Very effective publication and dissemination to clean energy￾agriculture nexus community members. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 17 GC SECTOR RATING EXPECTED RESULTS AND COMMENT 4 SWFF Agriculture / Water  SWFF was highly successful in meeting and exceeding all its targets. The GC’s investments are reported as having benefitted 7.1M people in 28 countries. Innovators sold nearly $15.9M worth of products to 7M farmers, enabling them to produce more than 6.8M tons of food on more than 8M hectares under improved practices. GC contributed to reducing water consumption by more than 19B liters compared to traditional practices, nearly quadrupling the program’s expectations. On average, the GC resulted in a net farmer income increase of $1,833. Innovators leveraged SWFF funding with more than $25.4M in additional funding through more than 300 partnerships. SWFF achieved an average social rate of return on investment of of 41 percent. 5 SL@B Health  Achieved its objectives in terms of seeding and accelerating innovations; supported innovations as a means to making progress in maternal and neonatal health (MNH), increasing attention on MNH, catalyzing multi-sectoral partnerships, and leveraging over $160M; and played a catalytic role in ‘de-risking investments in certain high impact innovations’. The GC adapted with a greater focus on later stage innovations and substantial results were achieved by several later stage innovations, but SL@B’s largely early-stage portfolio (with only select products on market to date) may have been at odds with expectations of near￾term ‘lives saved’. Although the portfolio is largely comprised of early-stage innovations, SL@B has already helped to save over 11,500 lives (largely as a result of four high performing innovations), has improved over 155,000 lives, and has reached over 3M beneficiaries. It is hard to assess SL@B’s potential impact on maternal and newborn mortality given the early growth stages funded, but impact models suggest SL@B innovations have the potential to save 150,000 lives by 2030. 6 EBOLA Health ~ The GC achieved its short run objective of seeking new practical and cost-effective solutions to improve PPE and infection treatment, but given its single open call round, no long-term objectives were set, nor targets estimated. A number of innovations from the GC proved very useful for tackling both Ebola and other types of outbreaks, so there was the potential of GC innovations having long-term effects. One example is MHero (a communication platform between frontline health workers and the Ministry of Health) which has been used by multiple West African governments both for Ebola and for COVID-19. 18 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION GC SECTOR RATING EXPECTED RESULTS AND COMMENT 7 ACR GCD Education ~ The objectives relating to enhancing literacy for children with disabilities can be considered achieved through the number of learners reached and the uptake of innovations beyond ACR GCD grantees. Significant achievements in terms of reach (over 600,000 early grade learners), but it is difficult to give a verdict on the overall achievement of objectives from the evidence available. The GC was able to establish much-needed focus on inclusive education and technology aid to make inclusive education a possibility. 8 SOGE Energy ~ Since its launch, SOGE’s partners have supported 77 companies and market enablers through 196 awards in 23 countries – including 17 directly supported by USAID. This has resulted in 1.2M off-grid connections and 14.4M projected connections which are estimated to reach over 71M people, but falls short of SOGE’s goal to achieve 16-20M connections. 9 MAVC Governance -- Two outcome indicator targets met – the number of citizens receiving a response from a government agency and number of changes in government policy – but not the third: a 20 percent increase in the number of citizens in contact with MAVC who believe that reporting deficiencies in public services is likely to lead to some improvement. The GC terminated six months early due to performance shortfalls and implementation weaknesses. An evaluation for Sida44 reported that there had been only limited achievements and only one of the 11 representative grant case studies had delivered at outcome level with the potential for scaling up. 44 IMC Worldwide (Feb 2018) MAVC Final Evaluation. Donor alignment around a shared vision with clear objectives and targets is critical to success. There were cases where this worked well (CHIC, PAEGC, and SWFF), but it was not the case for either MAVC or SOGE. In the case of MAVC, the partners did not have a clear, shared vision from the outset and then devoted too much time to aligning objectives. This calls for greater coordination and context analysis in the design phase. If the design process had been more rigorous in these two cases, there would have been a greater probability of the GC enabling more focused interventions and achieving greater success. The overall conclusion from our analysis for this MEQ is that factors which support the achievement of GC results and innovation success can be summarized in five key areas: 1. GC design characterized by clear objective setting and informed by a strong understanding of context, as evidenced by SL@B, ACR GCD, CHIC, and SWFF. 2. GC implementation included learning and adaptation, a success factor for a number of GCs, notably SL@B, which changed its emphasis to increase attention on scaling grants, and SWFF, where failures and adaptive pivots are documented in detail. 3. Understanding of and implementation support for pathway to scale, particularly by building robust partnerships and business networks, as in the case of PAEGC, and enabling partners to develop a shared pipeline and coordinate funding, as in the case of SOGE. 4. Effective acceleration and scaling support to innovators was highly valued and contributed to success, for example in SL@B where it led to the development of new partnerships and the ability to leverage funds from other sources. 5. Coordinated engagement by all donors, evidenced by PAEGC, SWFF, and CHIC, supported by good governance mechanisms to enable collaboration, as in the case of ACR GCD. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 19 A GC is a flexible instrument and can respond quickly to identify solutions in sudden onset emergency situations as demonstrated by EBOLA and Zika. For most other situations, the GCs need to have a clear understanding of the local context of the challenge before completing the design and need to ensure that the GC Manager has the skills needed to identify and cultivate the appropriate networks and linkages to support innovations and safeguard the legacy of the GC intervention. The factors inhibiting success all relate to any absence of the above success factors and/or not making significant enough investment in these areas. 4.1.1 SUB-EVALUATION QUESTIONS SEQ 1.1: WHAT TYPES OF OUTCOMES HAVE BEEN ACHIEVED BY GCS IN SUPPORT OF WHICH DEVELOPMENT OBJECTIVES? The objective of the GC approach is to invest in a portfolio of innovations, test them to see what works, and then support the scaling of the promising innovations. There have been successes from the GC approach across a range of different sectors and types of outcomes. The success of the higher performing GCs has therefore not been a consequence of the sector or type of outcome selected, but a result of good design, support for a balance of early- and later-stage innovations and knowing when to drop early-stage innovations that will not succeed at scale. GCs have achieved some notable successes in the agriculture and water sector and the health sector in particular. PAEGC and SWFF have both improved the incomes of rural smallholder farmers and have improved efficiency in the use of energy and water. SL@B, EBOLA, and Zika all provide good examples of success in the health sector. Zika surfaced a number of leading technologies that could be used for protection, diagnostics, wider population health, and other uses. USAID went from a place of not knowing what tools would or could work to having successful investments in public health. Zika acted as an ‘enabler’ and brought innovations to scale. For example, the results achieved by the WMP led to further funding from the MacArthur Foundation. 45 See USAID best practice at https://www.usaid.gov/project-starter/program-cycle/pmp/performance-indicator-elements/data-disaggregation The achievements in some other development areas should also be noted, in particular the access to energy through SOGE which has initiated a number of long-term investments in off-grid energy started by GC funded innovations. There is an inconsistency across the GCs in defining desired outcomes and expected levels of achievement. An outcome has to be adequately defined at the start of the challenge if its achievement is to be measured. Just three of the GCs (ACR GCD, SL@B, and CHIC) have defined theories of change (which map out the objectives of the GC and the pathways to achieving the outcomes defined), results frameworks, and realistic targets. In other cases, while the results have been exceeded (SWFF) or come reasonably close to achievement (PAEGC), the results framework was incomplete. In other cases, while there was some good practice in defining results (ACR GCD), there was a lack of clarity on what outcomes were actually expected and no clear indication of what achievements could be assessed against. GC results in terms of gender equality and social inclusion were clear for just three of the nine funds, thanks to their consistent disaggregation of results that made women and other disadvantaged groups visible.45 CHIC clearly demonstrated that the lives of men, women, adolescents, and children had been saved and improved, and SL@B saved the lives of women and newborn babies. These funds also measured results for the number of awards made to women innovators. Women and vulnerable groups have also been positively impacted by SWFF-supported innovations, through improved incomes and access to water for example, although by less than the overall average. Where results for women and other groups are not visible, this is typically because GC design did not include sufficient analysis and focus on GESI, such as defining specific target groups, considering their barriers to access, or setting objectives for achievements. SEQ 1.2: WHICH OBJECTIVES, SECTORS, AND CONTEXTS ARE GCS BEST AND WORST POSITIONED TO ACHIEVE SUCCESS IN, PARTICULARLY COMPARED TO TRADITIONAL DEVELOPMENT MODELS? 20 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION GCs have achieved success across a range of sectors and contexts which themselves are not determinants of success. Rather, GCs are best positioned to achieve success compared to a traditional approach by virtue of their flexibility in enabling the surfacing of innovative solutions and through the involvement of wider partners and potential innovators. This is demonstrated in the following examples: • Within Zika, the use of a GC model was particularly appropriate for the specific challenge faced where traditional tools and approaches to combating the Zika virus were considered inappropriate or ineffective. Vector-borne disease outbreaks such as Zika are unpredictable, often resulting in a reactive approach to containing the risk they pose and creating little incentive for investment in a new method. The GC model surfaced new and innovative solutions from new actors and stakeholders. • Within EBOLA, the participants were able to identify problems and possible solutions which a traditional aid instrument may not have been able to deliver. The GC instrument inspired quick improvements in PPE and Ebola control that improved the global response. It also enabled the bringing together of globally funded and tested innovative solutions to infection treatment and control that could transform the ability to address Ebola, and established critical public￾private partnerships to enable rapid testing and deployment of the best new solutions. • In SL@B, the GC model surfaced new approaches and partnerships in MNH that had not previously been tested. While evidence was that traditional models may still be better placed to achieve large scale impact, the GC has provided a key role in promoting the uptake of innovations. • The GC model enabled SWFF to be more flexible than a traditional development program by being able to reach out to small- and medium-sized enterprises (SMEs) and organizations with limited capacity that might not have been able to satisfy the fiduciary due diligence requirements of a traditional program. 46 A common expression to describe mid-stage innovations short of risk capital to take the innovation to the next stage. See Menon, J., Sagar, A. (2012) Prize￾Driven Innovation for Development. X PRIZE Foundation, U.S. and IIT Delhi, India. GCs seem better suited to service delivery or tangible outcomes (e.g., in relation to livelihoods). Some GCs have major systemic change as a long-term objective (e.g., ACR GCD in relation to inclusive education) and show some progress, but MAVC was distinct in focusing purely on governance and accountability and not service delivery. It was not successful. While this could be more a reflection of the poor design and coordination of the program rather than the GC instrument itself, the length of time required to effect change in governance, accountability, and state-citizen relations – and the very political nature of the work that it requires – indicates that a GC mechanism is not suitable for this type of development challenge. SEQ 1.3: WHAT IS THE RIGHT BALANCE OF EARLY- AND LATER-STAGE INNOVATIONS TO REACH WHICH TYPES OF OBJECTIVES? GCs are particularly effective at sharing the risk of developing and testing technologies, processes, and partnerships in the early- and mid-stage of the innovation stage. This ‘valley of death’ stage of innovation can have relatively high transition barriers to implementation at scale in LMICs.46 There are good examples from SL@B, PAEGC, and SWFF of innovations which have been taken through the mid-stage and then taken to scale by others. The conclusion from SWFF was that early-stage innovation funding is best suited to test whether a known technology is commercially viable. The experience of some GCs is instructive here: • 86 percent of SL@B funded awards went to early-stage innovations in order to create an early-stage pipeline that was critical at that time for filling a funding gap in early-middle growth stages. The seeding and acceleration of innovations in the MNH area developed innovator communities and ecosystems and the establishment of multi-sectoral partnerships contributed towards improving maternal and newborn lives. SL@B had some success with innovations that were taken to scale after the completion of the GC grant (e.g., Gradian, Bempu, CHW, and Dimagi), but in general was less successful at taking later-stage USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 21 innovations to scale within the timeframe of the challenge. One key factor is that this process of developing models at scale takes time, requiring different stakeholder relationships, and is arguably less suited to the time-bound GC model. • For Zika, given the specific nature of the challenge and the immediate threat posed by the Zika virus, there was an expectation that the GC should focus on mid- or later-stage innovations. In practice, early- to mid-stage innovations were funded and these proved to be successful, partly because there were some game changing early￾stage innovations that had time during the GC funding period to get to market and advance to scale. This met the objective of addressing the immediate threat of combating the Zika virus and building capacity to address future threats. • For SWFF, the approach was to start with early-stage innovations with a proven prototype and to support them step-by-step through the subsequent innovation stages, or to end support if the innovation was not performing. This approach ensured that the GC did not support innovations unless they were likely to be sustainable. Thus, there was a clear end objective of improved water/irrigation usage for agriculture which drove the innovation support and a mechanism to track the pathways of the innovations. The lesson here is that the GC needs to have a clear vision of the pathway to scale, and the appropriate skills to decide on whether and how to support early- to mid-stage projects and take these innovations to wider stakeholders to support scale-up. We conclude that GCs are best suited to supporting early- and mid-stage innovations on the pathway toward scale and other instruments are better suited to support innovations at scale. The evaluation suggests that there is no ‘right balance’ of early- and later￾stage innovations, but rather a need for each GC to develop an evidenced strategy based on good analysis at the design stage and learning from the innovations it funds, plus good plans to support innovation pathways and engagement with the ecosystem. 4.1.2 FORWARD-LOOKING QUESTIONS FLQ 1.1: HOW CAN GCS BE USED TO ADDRESS SYSTEMIC CHALLENGES GOING FORWARD? FLQ 1.2: HOW CAN GC PARTNERS BETTER TAILOR EXPECTATIONS AND ALIGN INVESTMENTS AND RESOURCES TO MEET REALISTIC OBJECTIVES? FLQ 1.3: WHICH RESOURCES/APPROACHES ARE BOTH NECESSARY AND SUFFICIENT TO REACH WHICH AIMS, AND WHICH RESOURCES/APPROACHES ARE NOT? FLQ 1.4: WHICH TYPES OF OBJECTIVES CAN BE ACHIEVED BY SUPPORTING EARLY-STAGE INNOVATIONS, AND WHICH BY LATER STAGE INNOVATIONS? The recommendations below respond to the FLQs above. Systemic change is about having an impact beyond the completion of the GC activities through the wider uptake, copying, or replication of innovation solutions or practices by individuals such as farmers, other businesses, governments, and civil society. If the GC has supported leveraging finance or the GC’s activities lead to uptake by others, then not only does this pass the sustainability test, it can also be described as having had a systemic impact on the ecosystem. GCs can achieve systemic change in a range of sectors. What matters is the presence of the success factors, set out below in the form of recommendations, concerning the quality of the design, the management of the fund, the acceleration and support skills of the GC, and effective coordination by USAID. GCs are best suited to supporting early- to mid-stage innovations and other instruments should be used to support innovation at scale. (Conceivably, such other, larger programs could include a GC as one of its components.) 22 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION The recommendations relating to MEQ1 are high level. Many aspects are revisited under sections on: • Scaling and acceleration (MEQs 2 and 3 in Sections 4.2 and 4.3), • Ecosystem development (MEQ 4 in Section 4.4), • Engagement with USAID (MEQ 7 in Section 4.7), and • Measuring results (MEQ 8 in Section 4.8). Recommendation: GCs should undergo complete and rigorous design ahead of launch involving effective USAID engagement with donors and partner alignment around the GC objectives. (Strategic recommendation for USAID Policy and USAID GC Managers) A good understanding of the pathways to scale incorporates an understanding of the payer, end- and other users, the systems, institutional or market dynamics affecting scale, and the opportunities and barriers to achieve scale in a timely way. Design of a GC should include careful consideration and definition of what pathways to scale are possible for the types of interventions it is seeking to support. This will help plan other GC components like measurement frameworks, acceleration strategies, and opportunities to identify other interested funders/investors. This should take into account the success factors identified by the evaluation (below) and the nature of the sector with quick action and solutions needed for quick onset humanitarian disaster and disease outbreaks, and deeper analysis of market systems and political economy for longer term or chronic problems. Rigorous design requires: • In-depth analysis of context and formation of realistic objectives. • Strong results measurement and learning framework and a GC Manager that adapts and learns from experience. • Understanding of the pathways to scale at the outset and implementation support in the building of robust partnerships to support this. • Good acceleration and scaling support which focuses beyond the innovator to engage the ecosystem. • Strong coordination and engagement by USAID. 47 See https://www.gsma.com/mobilefordevelopment/wp-content/uploads/2020/05/GSMA-The-Mobile-Gender-Gap-Report-2020.pdf Recommendation: Ensure that the GC has the necessary skills and experience to understand and build local partnerships and provide appropriate support to acceleration through providers who know the context. (Strategic recommendation for USAID Policy) • Ensure that these requirements are built into manager selection criteria. The requirements will vary according to the GC and need to be scoped in the design phase. Ask applicants to demonstrate their understanding of how to build partnerships and provide support, outline the skills they believe are needed and why, and provide evidence for how they have demonstrated these skills and developed and learnt from their experience. Recommendation: Focus GCs on getting innovations ready to scale rather than on implementing them at scale. (Strategic recommendation for USAID Policy) The evaluation findings show that the GCs have been particularly successful in supporting early- and mid-stage innovations and proven concepts that need to be adapted to the specific LMIC context. This reinforces the need for the GC partner to have the skills to build local partnerships and support the innovations with acceleration expertise (usually external). Recommendation: Strengthen developmental outcomes overall and for women and other disadvantaged groups specifically by embedding GESI principles and analysis into GC design (see Annex 13). (Strategic recommendation for USAID Policy; Programmatic recommendation for USAID and Partner GC Managers) • Contextual analysis should demonstrate the systemic barriers and challenges that women and disadvantaged groups face within the topic that the GC is addressing (e.g., agriculture, energy) and the barriers they might face in accessing the innovations designed to tackle them (e.g., due to the mobile gender gap).47 • The theory of change should integrate GESI, identifying clear pathways of change for women and other disadvantaged groups. • Set objectives and targets that make women, poor people, and other different target groups visible (see also Section 4.8). USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 23 4.2 SCALING INNOVATIONS MEQ 2: WHICH GC INNOVATIONS HAVE REACHED SCALE (EITHER FOR SUSTAINABILITY AND/OR FOR IMPACT) AND WHAT STRATEGIES AND FACTORS CONTRIBUTED TO OR INHIBITED SCALING? Summary: Looking across the GC portfolio, very few innovations have been supported to achieve scale despite several of the GCs having this as an objective.48 The GC portfolio has been skewed towards funding early- to mid-stage innovations with 79.6 percent of grants (for which there is data) less than $500,000. All GCs can point to examples of scaling successes, and while there is no consistent data overall on the number of GC innovations that have progressed to scale, there is positive evidence from the grantee survey. Taking innovations to scale often requires significant amounts of financial support beyond the limits provided by GCs within the portfolio. Survey feedback on acceleration and scaling support was positive. From portfolio experience, there are some major lessons on scaling support: that GCs have learned and adapted to strengthen the support they provide, that there is potential for USAID to assist in taking innovations to scale, that the focus needs to be beyond the innovators themselves, that expectations need to be explicit, and that time and expertise is needed to raise capital. GCs define stages of innovation and pathways to scale differently, but there are common characteristics. Some defined their scaling objectives and considered the potential pathways to scale at design stage while others focused more on surfacing solutions and only later considered scaling. The structures required to identify and support early-stage innovations are different to those further along the pathway to scale. GCs which have successfully supported innovations to reach scale i) understand the local context by developing local relevant partnerships; ii) consider the pathway to scale from the outset, especially from the user’s perspective; and iii) deploy local expertise that understands the context. Recommendations: See Section 4.3.2. There is no standard GC framework for scale using, for example, the six standard International Development Innovation Alliance (IDIA) stages, although PAEGC and SWFF have an adapted version of the six-point stage (see Figure 4.1).49 As a result, there is no consistent data overall on the number of GC innovations that have progressed to scale. FIGURE 4.1: PAEGC PATHWAYS TO SCALE 1. Concept Develop￾ment Articulation of the basic technical and financial feasibility of an approach and/or initial design of a product. 2. R&D Basic research, applied R&D, testing redesigning of an innovation, technology, or approach. 3. Initial Piloting Small-scale, real-world, and experimental application of an innovation, technology, or approach to evaluate feasibility, time, cost, adverse events, and other effects. 4. Early Adoption Technical validation and early proof of adoption carried out, and product/ approach has some customers/ early adopters. 5. Market Growth Innovator has proof of adoption/uptake in multiple markets, has acceleration partnerships established, and is moving toward a growing user/ customer base. 6. Wide-scale Adoption Innovation has proven its ability to reach a large customer base. 48 See Figure 4.1 Stage 6. 49 Or the six stages: Ideation; R&D; Proof of Concept; Transition to Scale; Scaling; Sustainable Scale, recognized by International Development Innovation Alliance (IDIA). This is similar to the six-point scale adapted by PAEGC – see Figure 4.1. 24 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION BOX 4.1: SUCCESS AT SCALING: ZIKA – WORLD MOSQUITO PROGRAM (WMP) Zika and USAID assisted the WMP to develop new relationships and expand networks including a scale-up in partnership with the Brazilian government through loan-based buy-in.This mechanism enabled funding to be received from BMGF and theWellcome Trust. WMP won a number of buy-ins with municipalities in Brazil who were able to take out loans under guarantee by the Ministry of Health. This provided WMP with a model for scaling and sustainability beyond GC funding. WMP’s small-scale pilot project in Colombia, covering about 40,000 people living in about 3-4 km2, grew to a large-scale project covering two major cities and a population of 2.5M people. ‘Without the USAID money, we probably wouldn’t have scaled … it allowed us … to learn and scale in a small amount of time and to do scale-up on community engagement’ – World Mosquito Program interviewee 50 Stage 6, Figure 4.1. 51 While the GC portfolio has made a total of 750 awards, there is data on the amount awarded for only 519 of them. 52 USAID distinguishes between smaller grants made to early-stage innovators and larger, milestone-based grants to later stage enterprises here. The boundary between the corresponding grant sizes is set at $500,000. 53 Survey respondents were asked to select the category which they felt best described the innovation stage they were at, out of a) early stage or start-up (e.g. ideation or research); b) proof of concept / piloting / prototyping; c) testing and transitioning to scale; and d) expanding / scaling. The extent to which innovations achieve scale in part depends on what ‘scale’ means for each of the individual innovators, how this is defined, and what is expected. Scale might mean sourcing additional funds/transitioning to scale for an existing early-stage innovation or expanding geographies for a later-stage innovation. A very clear definition is provided by SWFF: reaching at least 1M people. Some innovations, such as those addressing challenges in the agriculture sector, take much longer to reach scale than a quick solution to a humanitarian problem. For example, SWFF asked innovators to report along six scaling dimensions, including changes in areas such as the policy environment, plans to scale, and the affordability of products in their target markets, reflecting the time taken to change agricultural policies and gain market acceptance. Some GCs set a specific target on the proportion that will reach scale. For example, Zika expected that 15 percent of funded innovations would scale. The common expectation is that the innovator will take successful or proven innovations to scale, sometimes supported by a specific grant for that purpose (e.g., CHIC’s TTS grant). In the case of some health GCs, there was an expectation that the innovation may be taken forward by the government. Comparatively few interventions have been supported to achieve scale despite this being an objective of the GC model.50 Rather, the GC portfolio has been skewed towards funding early- to mid-stage innovations. As of August 2020, out of 519 awards for which there is data, 79.6 percent (413) were awarded less than $500,000 (which USAID take as an indication of being an early-stage grant) and only 20 were awarded $2M or more.51,52 In spite of the long duration of the SL@B program (2011-18), which provided the opportunity for follow-on grants to further develop the more promising innovations, 86 percent of awards went to innovations in the first three stages of growth (ideation, research and development, and proof of concept). It should be noted that timelines for scaling are significant, even in relation to a longer￾running fund like SL@B. Nine percent of survey respondents described their innovation as being at the expanding / scaling stage at the point of application, while 46 percent said that it was currently at that stage, meaning that 37 percent of innovations surveyed had progressed along the innovation pathway to reach the scaling stage. 53 Further, there is some evidence that GCs have been successful in getting innovations to the point of scaling readiness: that they identify innovations, surface and test them, move them along the pathway to scale, provide a laboratory for learning and finding pathways, and provide a platform for others to support the innovation to achieve scale (e.g., SL@B). Taking innovations to scale often requires significant amounts of financial support beyond the limits provided by GCs. This is reflected in innovators’ responses to the grantee survey: ‘While the scaling support was useful in letting us know how to bring our Aquaponics innovation to scale, it was not accompanied with the financial support needed to take the innovation to another level. For example, enabling us to manufacture many Aquaponics units and put them on the market as a ready product for sale. Also, the support identified the need for us to have sales and marketing agents, but it was not accompanied with the funds required to sustain such structure….’ SWFF grantee ‘Availability of scale up funding would speed up the development and implementation of projects that were initially supported by USAID’. Zika grantee USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 25 There is a range of views as to how GCs should support the innovation pathway. Some of the GCs consider their role in scaling support to be primarily financial through scaling grants or investment facilitation. The need for funding for scaling was a common theme amongst comments supplied by surveyed grantees. Others go beyond financial assistance. For example, SOGE was able to contribute to the establishment of new off-grid connections for the innovations by funding financial intermediaries rather than facilitating investment in the individual companies. Three of the comparator programs (Partnering for Innovation, IIA, and SHOPS Plus) focus on the scale and mainstreaming end of the innovation pathway. All three programs were successful in supporting future scaling through building partnerships to commercialize innovations (Partnering for Innovation) or encouraging their take up by government and private actors (IIA). SHOPS Plus helps projects to scale by building public-private partnerships around global health challenges (like access to family planning products and services). These non-GC programs are active in countries where many GCs operate and provide coordination points for GC-supported innovations to be further supported or benefit from policy- and ecosystem￾level work. They also demonstrate the importance of engaging with the ecosystem and building partnership rather than just acceleration through the innovation pathway. Survey feedback on GC acceleration and scaling support was positive. 80 percent of survey respondents said that it was sufficient, although 40-48 percent of respondents said they did not receive any scaling support as set out in Table 4.2. The average rating out of 10 for each of the five forms of support itemized in the survey was adequate, ranging from 6.3 to 7.4. The experience of the GC portfolio in supporting innovations to scale is mixed and some major lessons have been learned from the portfolio. 54 1 = Not at all useful and 10 = Extremely useful Shifting to focus on later-stage innovators may increase the likelihood of achieving scale. SL@B successfully ran acceleration and scaling support which was highly valued by the innovators and had an ambition for scaling throughout implementation. Over time, this evolved from supporting early- and mid- stage innovation to putting more emphasis on supporting later-stage innovations. CHIC has increased the upper limit of its TTS grants from 1M to 3M Canadian dollars in its third funding round, recognizing that more funds were needed in some cases. There is potential for USAID to assist in taking innovation to scale. ACR GCD, reflecting on their experience, noted that there was potentially a very powerful role for USAID and Missions to use their unique position to involve governments and other key actors in the country or region to scale innovations, especially in those cases where the “buyer” of the innovation is a public entity such as the Ministry of Education. GCs could choose to cease funding innovations which are not progressing or release milestone payments based on progress achieved as was introduced for SL@B, EBOLA, and Zika. SWFF reviewed their innovation portfolio regularly and only continued to support those that showed the potential for scale. Set clear expectations for reaching scale across the portfolio from the outset. Zika was explicit in this regard. It had an expectation that 15 percent of the portfolio would advance through the product development pipeline, progressing towards ‘impact’. Progress would look different for each innovation funded, i.e., early-stage innovations would progress through to proof of concept and mid- and later-stage innovations would progress through to scale. All awardees were asked to state on their monitoring, evaluation, and learning (MEL) plan what plans for scale were in place, and projects were required to report on their position along the scaling pathway. SWFF’s design document set out an ambition to bring eight innovations to scale and/ or be commercialized by businesses in at least eight LMICs. TABLE 4.2: GRANTEE SATISFACTION OF SCALING SUPPORT BY GC SUPPORT TO SCALING LEARNING AND SHARING IDENTIFYING PARTNERS IDENTIFYING FUNDS SUPPORT ON SYSTEMS SUPPORT TO MODEL Percent Receiving Support 61 52 57 52 57 Satisfaction (out of 10)54 7.4 6.3 6.4 6.7 7.1 26 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION It takes time and effort to provide the expertise and capital needed to support scaling. For PAEGC, there was a mixed experience on how best to support innovations to scale. While TTS grants were welcome, some GCs emphasized that there was sometimes an unrealistic assessment of how long it would take for an innovation to scale. One PAEGC grantee noted that most innovations were not ready for impact investors, so those introductions were less useful, and that advisors had limited practical experience in building or scaling a viable enterprise in an LMIC context. This learning has shaped WE4F, the follow-on GC for both PAEGC and SWFF. Taking the evidence as a whole, our conclusion is that GC experience has shown that the GC model is better suited for getting innovations ready to scale, but less suited to supporting innovations to achieve implementation at scale. 4.2.1 SUB-EVALUATION QUESTIONS SEQ 2.1: HOW AND TO WHAT EXTENT HAVE SCALE AND PATHWAYS TO SCALE BEEN DEFINED EX ANTE VERSUS EX POST? The GCs defined the stages of innovation and pathways to scale differently, but there are common characteristics. PAEGC provides a good example, presented in Figure 4.1, of a framework for describing the stages of innovation supported and measuring how innovations progressed. Some GCs, such as Zika, SWFF, and SOGE, clearly defined their scaling objectives from the outset and gave consideration to the potential pathways to scale at the program design stage. This enabled the GC to provide all the key elements of support considered to be required for achievement at scale. This included identifying and defining the stages of innovation to support, designing funding rounds to attract the kind of organizations that have the potential to scale, and making appropriate provision for the technical support required. Other GCs such as PAEGC, SL@B, and ACR GCD initially focused more on surfacing solutions and only later considered the kind of support required to take innovations further along the pathways to scale. For PAEGC, the second of the program’s two calls sought later￾stage innovations from organizations identified as having the potential to scale, introduced more comprehensive acceleration support, and later developed a linked financing mechanism to support investment-ready innovations. Similarly, the first two ACR GCD funding rounds were focused on identifying and testing innovative solutions to defined problems, but the third round started to consider how to support promising innovations identified in earlier rounds along pathways to scale. SL@B initially saw scaling as a ‘concept’ that would be facilitated at a later point, meaning that the program funded a large proportion of early- to mid-stage innovations in the early years. As the ambition for scaling increased, more attention was paid to TTS grants and the requirement for innovators to demonstrate a sustainable scaling plan. Over time, there was a realization at SL@B that there was significant variation in the amount of time and funding required to take different kinds of innovations to scale. The time required to take most early￾stage innovations to scale should not be underestimated. This applies to market solutions needing to attract funding and, for some GCs, especially in health and education, innovations requiring public partnership engagement. The CHIC design envisaged that promising, early-stage innovations supported by seed grants would be able to apply for TTS grants to enable them to progress along the pathway to scale, thus creating a ‘funnel’ of innovations. None of the innovations developed through seed grants have yet matured sufficiently to apply for transition grants and, unless current funding timescales and donor commitments are extended, they might not be able to in the future. The lesson on the time taken to raise capital was raised in a PAEGC KII: ‘a few months of support at the end is nowhere near sufficient, capital raising takes 1-2 years so needs to start a lot earlier’. Most GCs’ periods of performance (for the program writ large) are not long enough to allow early-stage innovations time to achieve scale. The new WE4F design provides a good example of how pathways to scale can be developed. The PAEGC and SWFF programs achieved some success in brokering partnerships with potential investors to support the scaling of innovations that were considered to be ‘investment￾ready’, but other promising innovations that require further transition support have been picked up by the follow-on WE4F program. WE4F is explicitly focused on both supporting promising innovations that were identified and nurtured during PAEGC and USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 27 SWFF ‘to support their next level of scaling’ while also opening up new calls for innovations within the water-agriculture-food nexus. Additionally, the PAEGC grantee quoted above went on to say, in relation to timescales for scaling support, that ‘the new project, WE4F, recognizes this much better’. The program structures and mechanisms required to identify and support early-stage innovations are quite different to those necessary to support those further along the pathway to scale. Most GCs (including CHIC, PAEGC, SL@B, SOGE, SWFF, and Zika) supported multiple stages of innovation, striking a balance between early-stage and mid- to later￾stage innovations. This leads to relatively balanced and diverse portfolios, but also requires complex management structures to provide all the different kinds of support required for innovations at different stages. SEQ 2.2: WHAT HAVE BEEN THE CHARACTERISTICS OF THE INNOVATIONS THAT REACH SCALE? GCs that have successfully supported innovations to reach scale share some common features: • Considering the pathway to scale from the outset, especially from the end-user’s perspective; • Understanding the local context by developing local relevant partnerships; and • Deploying local expertise that is familiar with the context to provide scaling support. These common features were shared by the following GCs: • PAEGC, EBOLA, SWFF, SL@B, and Zika: Innovators who made significant progress toward becoming sustainable developed strong local partnerships and intimate knowledge of the local market and context to ensure the innovation was contextually relevant, including customer willingness and ability to pay. • PAEGC: Innovators with prior experience of working in LMICs generally succeeded more than those that had none. This should be a prerequisite moving forward. • Zika and SL@B: User-centric approaches and the involvement of end-users in design and roll out (particularly if this is the government) are more likely to achieve scale. • PAEGC: Projects that developed complex, novel technologies were less likely to reach commercial scale than simpler or more established technologies, implying that a focus on user needs is important. Recommendations on scaling and acceleration are given together at the end of Section 4.3.2. 4.3 ACCELERATION MEQ 3: WHICH GC ACCELERATION STRATEGIES ARE MOST EFFECTIVE AND AT WHAT STAGES OF INNOVATION? Summary: A key learning from the portfolio is the need to have a clear strategy for acceleration support from the beginning of the GC. Customized acceleration support appears to have been an effective strategy for a number of GCs, notably SL@B, Zika, SWFF, and PAEGC. There are examples of innovative approaches to acceleration. Multi-faceted and tailored approaches to acceleration support are effective, but resource intensive and logistically challenging in global GCs. External accelerator services (the model for all GCs) provide added value to innovators. 65 percent of innovations receiving support at the beginning of their grants found it useful or very useful, and only 18 percent were dissatisfied. The critical factor in quality is relevance to context. Recommendations: The focus should be on supporting early- to mid-stage innovations. The critical issue is for the GC to have the right strategic understanding of how to support the portfolio of innovations for acceleration and scaling, and to plan for this from the outset. Apply tailored acceleration support only to those innovations that have the prospect of reaching scale. Acceleration strategies encompass the various types of support provided by USAID to grantees to help them move along the pathway to scale (see Figure 4.1). The term ‘acceleration’ suggests that these strategies will speed up growth and impact. The evaluation revealed that clear acceleration strategies should be built into GC design. This was set out clearly for Zika and SWFF, but, for other GCs, there 28 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION was a reliance on the funded innovation to choose the growth pathway using a TTS grant and little appreciation of the need for specific support. There was some adjustment: in SL@B there was no strategy, funding, partner, or understanding of what ‘designing with acceleration and scale’ really meant when the GC began in 2011, but good learning about the need to ensure that acceleration support is contextualized led to changes in the approach and support being provided by accelerator services.55 The evidence from the portfolio is that the GC should be less of an ‘acceleration doer’, but should rather have the skills to identify those innovations that have the best potential and then facilitate the necessary engagement with specialist and tailored support, as was the case for SWFF. Although resource intensive, effective acceleration support should be customized to innovator needs and local contexts. Customized support was used successfully for SL@B, Zika, SWFF, and PAEGC. This includes 1:1 support, workshops, mentorships, and the facilitation of linkages. Such personalized support ensures that it is ‘right sized’. Innovators also find opportunities for peer-to-peer exchange of experiences and learning particularly beneficial. SWFF used a ‘Needs Diagnostic Tool’ to ensure that expectations were aligned to improve the success of acceleration support.56 SWFF also had some success with a vendor approach which evolved to include locally based vendors, making it easier for innovators to continue to access support after their participation in the GC had ended. SWFF measured the results of its acceleration support and learned from the operational partnerships which continued after the end of GC funding, providing greater confidence in the sustainability of the outcomes. Effective acceleration support is also contextualized to the specifics of the innovation context and requires a wide skillset with experience of the local market in order to be able to build local partnerships and mobilize finance. PAEGC is an example of where greater attention to the local market context improved results: because the innovators were pursuing different business models in different regions and were at different points of the innovation life cycle, individual support was considered a critical component of the program’s success. 55 SL@B Final Evaluation Report (2020) 56 https://securingwaterforfood.org/wp-content/uploads/2020/10/SWFF_FinalAnnualReport_Oct2014-March2020_4-15-2020-update.pdf Multi-faceted and tailored approaches to acceleration support are effective, but resource intensive and logistically challenging in global GCs. The PAEGC model of acceleration support provided by VentureWell, working in collaboration with TetraTech, was based on an initial participatory diagnostic that considered five dimensions of a venture development framework, i.e., business model, technology innovation, market, resources, and team and venture structures. The same framework was then used to track progress within each dimension and analysis of progress identified factors leading to successful venture development. This venture development approach was not part of the original design but rather added when the need for more comprehensive acceleration support became clear. Apart from group webinars, individual PAEGC venture support was provided mainly through remote mentoring with occasional opportunities for face-to-face workshops. VentureWell’s PAEGC team identified that the remote mentoring was restrictive but worked well when linked to impending face-to-face workshops. ‘…it is important to move a venture’s technology, market, and business model forward in parallel. Working to develop a technology in the absence of movement on the other two can be a “road to nowhere” where the technology exists without a viable path to market’. PAEGC KII On the basis of learning from its initial support to innovators, in 2018, SL@B also introduced a new, tailored package of more intensive acceleration support. This resulted from a recognition that the time and cost of taking innovations to scale and to better secure or accelerate sustainable impact was greater than anticipated. Tiered support provided more intensive mentoring for the most promising grantees in the portfolio. Innovators were in both early- and later-stage, but more in-depth support was provided for later-stage grantees or those early-stage ideas that appeared likely to be ‘game￾changing’. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 29 BOX 4.2: ACCELERATION SUPPORT: SL@B – EXPANSION OF GRADIAN’S ANESTHESIA PROGRAM IN TANZANIA AND UGANDA SL@B provided tailored acceleration support to Gradian’s anesthesia and critical care concepts for maternal health and coaching on core techniques for scaling their model within Zambia and across other countries such as Tanzania and Uganda. The acceleration support enabled Gradian’s local partners the opportunity to field test and refine a simulation-based product training curriculum and methodology. SL@B also supported Gradian to pitch support from professional coaches in communications pieces and promotions through media channels which were reported to be successful in the evaluation.57 57 Duke Global Health Innovation Center (May 2020) Evaluating SL@B 58 This could be based on PAEGC, SWFF or the USAID supported International Development Innovation Alliance (IDIA). Other aspects of acceleration were less successful, especially attracting investment. For example, PAEGC and SWFF attempted to bring in the investor community through investor events, but this matchmaking method had only limited success. In the case of Zika, individual innovators took their own steps (with support from USAID and TA providers) to improve the chances of their innovations being sustainable in-country, e.g., by establishing agreements with local stakeholders. Support to the enabling environment (see Section 4.4) was a key aspect of the acceleration support under SOGE, but a clearer implementation plan of this wider ecosystem support was needed in order to be effective. 4.3.1 SUB-EVALUATION QUESTIONS SEQ 3.1: WHAT ARE THE ADVANTAGES AND DISADVANTAGES OF PROVIDING IN￾HOUSE VERSUS EXTERNAL ACCELERATION SUPPORT? While some GCs (e.g., SWFF and PAEGC) were actively involved in managing acceleration services, all GCs relied on external acceleration service providers. ‘Having an external party come in to challenge innovators to think more holistically about the business model and their proposed market is essential, as this is something which innovators tend to miss out (they focus more on the technology/product). Advising them to ‘look around the corner’ is an important aspect of the support. This could be said to be part of wider “systems thinking” (and resonates with wider development thinking, i.e., don’t just think of the emergency at hand, think about the causal problems)’. ZIKA KII The key issue for acceleration services is not whether this is in-house or external, but whether the GC knows how best to support the innovations in the local context. External acceleration services can provide real added value to innovators. They challenge them to think about their business model and proposed market in the round, not just about the product/technology. There have undoubtedly also been successful cases of internally managed acceleration services (with external experts). 4.3.2 FORWARD-LOOKING QUESTIONS FLQ 2.1: HOW CAN WE CATEGORIZE DIFFERENT PATHWAYS TO SCALE AND APPROPRIATELY INTEGRATE PATHWAYS TO SCALE INTO PROGRAMING? FLQ 3.1: WHAT ACCELERATION STRATEGIES MIGHT BE CONSIDERED IN FUTURE GC PROGRAMING TO BEST SUPPORT INNOVATORS? GCs have categorized the pathways to scale in different ways on a three-stage or six-stage scale (see Figure 4.1) and the nature of these pathways to scale will follow different timescales depending on the sector. While there would be merit in having a standard categorization based on the IDIA approach, a greater priority is the need to establish a clear focus on scaling objectives and consider pathways to scale from the outset.58 This will enable all the elements required to support progress on the pathway to scale. A realistic time scale and rate of adoption (and failure) needs to be established for the GC and reviewed periodically, and the GC needs to take a strategic approach in selecting and supporting only those innovations which continue to have the potential to progress along a successful pathway to scale. 30 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION When introducing any innovation, local context must be considered. How an innovation is implemented can often determine its impact as much as the properties of the innovation itself. Introducing innovative tools or approaches will often require aligning stakeholders. The GC manager, USAID, and USAID Missions have a key role to leverage their convening power to encourage adoption or facilitation by the government. This is especially true for public health, education, and energy GCs. Recommendation: GCs will always be hoping to identify and surface game changing innovations. These can emerge in an unpredictable way and cannot always be planned for, but it is recommended that the GCs have an acceleration and scaling strategy in place from the outset and focus them on supporting early- to mid-stage innovations. (Strategic recommendation for USAID Policy) Develop strategies based on the following: • Ensure acceleration and scaling support is in the design of the program, that the GC Manager has an understanding of the principles of acceleration, and that the GC Manager knows the types of tailored support that can be provided to the innovation. Ensure that there are clear definitions of ‘scale’ and ‘acceleration’ from the start of the program and associated criteria for providing acceleration support.59 (Strategic recommendation for USAID GC Managers) • At the design stage of a GC, map pathways to scale conceptually, recognizing that there are multiple pathways to scale including investment by impact and other private sector investors; commercialization of innovations by SMEs, franchises, or large-scale corporations; incorporation into government programs and systems; and adoption and implementation by local communities. GC applicants should be required to map the pathway 59 Criteria could include having an identified market, having positive user feedback on innovation desirability and utility; potential investors in place; developed business plan; demonstrated technical feasibility; plausible marketing strategy. These could be applied at application/selection, after inception and at a mid-point, within a strategy for acceleration and scaling. to scale for their own innovation using this conceptual framework. (Strategic recommendation for USAID GC Managers) • Monitor and measure the performance of acceleration services to enable learning within and across GCs. Baseline diagnostics and acceleration monitoring frameworks should be multi-dimensional, i.e., consider the development of the business administration, access to resources, market engagement, and business model, as well as the development of the technology or service. Ask grantees to assess progress towards scale as a key component of milestone reporting. (Programmatic recommendation for USAID and Partner GC Managers) • Adapt the approach to acceleration over time, particularly with a program spanning a long period, in order to enable learning to be applied and to give room to adapt the way acceleration support is provided. (Programmatic recommendation for USAID GC Managers) • Ensure that acceleration support enables the development of the innovators’ technology, market, business administration, and business model progress in parallel in order to ensure that they have a viable path to market and longer-term sustainability. (Programmatic recommendation for Partner GC Managers) • Provide tailored 1:1 acceleration support to innovators combined with a mentorship approach and the ability to facilitate networking and connections, including for public sector uptake (for health and education innovations). Ensure that there is sufficient time and resources for acceleration (which will follow different rates of growth and uptake depending on the sector of the GC). (Programmatic recommendation for Partner GC Managers) • Ensure that the acceleration support provider, likely to be external, has the necessary local knowledge, networks, and experience of the local context. (Programmatic recommendation for Partner GC Managers) USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 31 4.4 INVESTING IN ECOSYSTEMS MEQ 4: HOW EFFECTIVELY HAVE GCS INVESTED IN ECOSYSTEMS STRENGTHENING, AND WHAT HAVE BEEN THE RESULTS OF THESE INVESTMENTS? Summary: All GCs have made investments in the ecosystems associated with their innovation and there are numerous instances of successful interfaces which have led to increases in reach and impact for GC-funded innovations. Engagement with the wider ecosystem in which an innovation is situated is crucial to place grantees on the pathway to scale and sustainability. A number of GCs built engagement into the initial design of the GC as an explicit objective or introduced system-building activities into the application phase. For others, a focus on the wider ecosystem occurred through structured investments during implementation. Ecosystems comprise both institutional and policy environments, as well as a wide range of actors – innovators and experts working in the same field, donors and investors, policy makers and public bodies, suppliers of market and supply chain services, intermediaries, and end-users – all of whom contribute to the testing and adoption of different innovations. GCs worked with these groups – as well as with intermediaries such as market accelerators, expert panels, coaches, and consultants – to foster enabling environments for grantees and their innovations. Despite individual success stories, there is no evidence of a consistent or formal approach to ecosystem engagement across the GC portfolio as a whole. Similarly, there are limited examples of GCs working with USAID Missions, other OUs, or programs to explore synergies and the potential to build links with their own ecosystems. These networks could be leveraged further to integrate GC-funded innovations into existing systems, particularly in the case of regionally focused GCs or territories where a global GC is supporting a cluster of grantees. Innovations that have a direct bearing on government-led programs in education or public health are supportive of wider ecosystems as there are opportunities to find synergies with and strengthen existing policies and programs and enable these innovations to scale and achieve development impact. The experience of the PAEGC East Africa Hub demonstrates that creating a regional or local presence for GCs can focus energies and resources to build a multi-pronged enabling environment for innovators. Interventions include working with networks of innovators, building partnerships, advising on regulatory requirements, creating and sharing toolkits, and building awareness of new solutions. Wider ranging portfolios implemented in multiple geographies, however, can struggle to influence systemic change due to the breadth, rather than depth, of their investments in ecosystems. GCs benefit from engaging with ecosystems at different points along the program timeline: early engagement widens the pool of potential applicants, targeted partnerships can strengthen trialing of innovations, and network building creates different pathways to sustainability. GCs need to work with governments and other decision makers to address regulatory barriers facing innovators; creating an enabling environment for trialing innovations can be as important as the solutions themselves when it comes to their adoption. Recommendations: GCs should invest more effectively in ecosystem strengthening in the future by creating a standard approach to engagement across the portfolio; by building awareness and inviting different actors into the process from the outset; by engaging directly with USAID Missions and other OUs to take greater advantage of existing USAID and partner funded networks in relevant sectors and geographies; and by identifying opportunities for high-impact, targeted interventions that have a direct and material benefit to grantees and the wider community of innovators. Ecosystem engagement is crucial to placing grantees and their innovations on the pathway to scale and sustainability. Innovations may originate in laboratory conditions, but must be integrated into wider systems for testing and trialing and must create the demonstration effects necessary for uptake and adoption. EBOLA grantees visited ‘hot zones’ to field test products, enabling solutions to move from laboratory to full manufacture. For SL@B, an article outlining the GC’s retrospective ToC (in the case of TTS grants) stated that ‘forming MNH partnerships with governments, health professionals, communities, researchers, academics, for-profit industries, and other potential funders for whom a project’s scalability and sustainability is a priority is essential for these awards to succeed’ and noted that 32 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION the GC brought an increased focus and understanding to ecosystem engagement over time.60 Other GCs, as detailed below, have also engaged with their wider ecosystems to lay the foundation for growth and impact beyond the lifespan of the program itself. All GCs interact with and invest time, effort, and financial and non-financial resources in ecosystems to increase awareness and uptake of funded innovations and there are numerous examples of successful interactions with different ecosystem actors. Engagement can occur at different points along the GC timeline and take many forms. These include sharing ideas and experiences among a wider community of innovators, experts, and funders; raising awareness, gaining political traction, and addressing the regulatory environment through interaction with regional bodies, government representatives at different levels, and public bodies; securing long-term funding to ensure sustainability beyond the duration of the GC; and integrating into existing systems and supply chains to drive commercialization and scaling opportunities. Examples of different ecosystem actors who GCs interact with include the following: • Innovators working in the same or related areas as GC grantees who share ideas and experiences and enhance the GC-funded portfolio of innovations. Examples include SL@B’s investment in a community of practice around the GC centered on the DevelopmentXChange platform, an annual event attended by GC and other innovators, and a range of actors including private sector investors and government officials, and SOGE’s participation in global forums tackling the challenge of off-grid energy provision. • USAID OUs and programs which intersect in terms of objectives or geographic/sectoral focus and which offer opportunities to build awareness, deploy, test, and increase the reach of individual innovations. Examples include ACR GCD combining forces with other USAID-funded programs in the Philippines, and SWFF engaging with Missions as a core operating principle of the GC.61 • Funding bodies such as other donor agencies, foundations, or NGOs that can help to secure long-term funding for innovations which need longer to demonstrate impact and achieve scale than is possible in the timeframe of the GC alone. Examples include the partnerships that ACR GCD was able to build with DFAT and World Vision; Zika’s links 60 Lalli, M., Ruysen, H., Blencowe, H. et al. Saving Lives at Birth; development of a retrospective theory of change, impact framework and prioritized metrics. Global Health 14, 13 (2018). https://doi.org/10.1186/s12992-018-0327-z 61 Note that this level of engagement - finding program synergies between grantees and Mission priorities and focused activities - goes further than the requirement that all GCs (with the exception of CHIC) engage with Missions for approval to deliver awards in their territories. to the World Health Organization (WHO); CHIC’s engagement with a range of humanitarian networks convened with the assistance of the World Food Program (WFP); and PAEGC’s links with the AfDB. • Ecosystem intermediaries such as market accelerators, consultants, and coaches who help create enabling environments for innovations to take root in different local contexts. Examples include SOGE’s funding of market advocacy organizations to drive policy support for growth in off-grid energy in Nigeria, Uganda, and Ethiopia, and boot camps run by CHIC and EBOLA to build synergies across cohorts of grantees. • Other investors or commercial sources of finance such as impact investors, venture capital funds, banks, and other financial institutions that can provide investment for market-building and commercializing innovations as a key pathway to scale. Examples include SWFF building market linkages in the different countries in which the GC operates, SL@B and Zika connecting innovators to commercial partners, and PAEGC’s links with impact investor Alpha Mundi and venture capital firm Factor(e). • Policy makers and key public, private, or civil society organizations (CSOs) operating in the same space or tackling similar development challenges. These can be enablers or inhibitors of GC-funded innovations, depending on the policy climate in which innovators are operating and the scope of opportunities for integration with existing systems. Examples include ACR GCD engaging directly with education policymakers in Morocco, India, and the Philippines; EBOLA and Zika working with the Ministries of Health in Liberia and Colombia respectively; and PAEGC, SOGE, and SWFF providing grantees with advice on navigating the regulatory environments affecting grantees in different countries. • Supply chain actors who provide the enabling infrastructure necessary for the long-term viability of specific innovations. Examples include EBOLA utilizing the assets of other supply chain actors in Monrovia, SL@B linking with licensing partners, SOGE engaging supply chain actors via results-based financing to reduce cost via bulk purchasing agreements and running competitions and training programs to support solar household system installation and services, and PAEGC building public-private partnerships along the tea supply chain with a range of partners in Kenya. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 33 FIGURE 4.2: ECOSYSTEM ACTORS GC Grantee Community of Innovators Subject Matter Experts USAID Missions and Operating Units Policy Makers Investors Donor Agencies, Foundations, and NGOs Supply Chain Actors Market Intermediaries and Advisers Public, Private, and Civil Society Organizations • Partners with shared objectives • Opportunities to link to existing programs and systems • Synergies and shared learning • Opportunities to collaborate • Regulatory environment for new solutions • Erect or remove barriers to adoption • Advice and guidance • Experience-based insight • Commercialization partners • Long-term financial sustainability • Upstream and downstream infrastructure • Key partners in testing innovators in market conditions • Networks and connections • Contextual understanding • Opportunities for future funding • Staged funding to grow to scale • Enabling environment for new solutions • Advocacy and coaching Grantees cited ecosystem engagement activities as contributing to the success of their innovations. A third of respondents to the grantee survey (33 percent) strongly agreed with the proposition that ‘support received through GCs has catalyzed the development of other solutions by our organization or other players’, with a further 27 percent agreeing with the statement (see Figure 4.3). 21 percent of survey respondents were neutral and 18 percent either disagreed or strongly disagreed, suggesting that there were occasions where support was either lacking or not effective. One respondent, referencing EBOLA, stated that the program could have been more effective if infrastructure around solution standards had been set in parallel with other organizations such as the WHO. FIGURE 4.3: GRANTEE SURVEY RESPONSES 33 percent 23 percent 21 percent 3 percent 15 percent “The support we have received through the GC has catalyzed the development of other solutions by our organization or other players” Strongly agree Agree Neutral Disagree Strongly disagree Ecosystem engagement may be structured into the initial design of the GC as an explicit objective or introduced through system-building activities in the application phase. Applicants for funding through CHIC, for example, were required to demonstrate both the relevance of their innovations and the potential for collaboration with private-sector partners as part of their funding submissions. (Engagement continued into implementation as CHIC worked directly with broader humanitarian sector networks in different geographies and created opportunities for innovators to meet with and influence leaders representing different parts of the wider ecosystem.) Similarly, SL@B and SOGE called out the importance of developing links with wider ecosystems as a core component of the GC design and SWFF had the explicit objective of aligning and finding synergies with other investments in water systems development. Funds which may not have prioritized ecosystem investment at the outset have nevertheless created structures or forums for innovators to have greater engagement with ecosystem actors during implementation. This engagement occurs as grantees start to pilot and test innovations in different countries and contexts and run up against the need to build awareness and trust; as they start to secure acceptance, endorsement, and longer-term funding; and as they start to demonstrate the place of different innovations in wider systems and prove their efficacy. These systemic and infrastructural requirements drive both formal and informal interaction with different actors in the ecosystem. Examples of 34 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION structural engagement include finding opportunities to integrate with existing information systems (EBOLA and Zika); building a community of practice around a common platform (SL@B); and, most notably, creating a regional hub that enables multiple points of interaction with a wide range of actors (PAEGC). GCs have worked with intermediaries to create support networks within grantee cohorts and to assist them in engaging with different actors in the wider ecosystem. Intermediaries provide coaching, expert advice, and consultancy and market acceleration services to build enabling environments for innovators. Both CHIC and EBOLA organized boot camps for grantees run by intermediaries who provided targeted coaching for participants. For CHIC, Brink and the WFP helped the cohort of grantees to find synergies between their projects and engage in the peer-to-peer sharing of learning and expertise.62 The EBOLA boot camp was run by IDEO to align grantees on the core objectives of the GC and to standardize activities in line with CDC guidelines.63 This made it easier for innovators to meet the stringent protocols established by health authorities in affected countries. SOGE funded market accelerators in Nigeria and Uganda, building on the success of these engagements to extend funding to accelerators in Ethiopia as well.64 The accelerators provided market intelligence, policy advocacy, TA (including building awareness of the sector with local banks and financial institutions), convening activities among innovators, research into local off-grid energy policy environments, activities targeted at overcoming market barriers (such as enabling access to foreign currency for local projects), building consumer awareness, and, in the case of the Rwandan Market Accelerator via Energy Private Development (EPD), forming a trade association of off-grid enterprises in Rwanda. Despite individual success stories, there is no evidence of a formalized approach to ecosystem engagement across the GC portfolio as a whole. While grantees may have opportunities to meet and share experiences, there is no formal process in place for program managers 62 Brink is a UK-based innovation consultancy, focused on ideation and experimentation, with experience working with frontier technologies in the humanitarian and educational sectors. 63 IDEO is a global design company with a wide portfolio of private and public sector clients. 64 SOGE worked with the Nigeria Off-Grid Market Accelerator (NOMAP): the Uganda Off-Grid Energy Market Accelerator (UOMA); the Ethiopian Access to Energy Market Accelerator (EMA); and the Rwandan Market Accelerator via Energy Private Development (EPD). 65 For access to these materials, see the following links: Scaling Pathways: https://scalingpathways.globalinnovationexchange.org/; Partnering for Innovation: https://www.partneringforinnovation.org/our-knowledge-2; Local Works: https://www.usaid.gov/local-faith-and-transformative-partnerships/local-works 66 See CII’s Pathways to Scale and Ideas to Impact series. of different GCs to do the same and learn from each other’s successes and failures in integrating with different system actors or structures. Other USAID programs committed to open innovation have all produced toolkits associated with their specific program objectives. These include the Pathways to Scale series which emerged from the work done by IIA; a practitioners’ guide for serving the smallholder market effectively, created by Partnering for Innovation; and advice to Missions in driving locally￾owned development in their countries, produced by Local Works.65 Some GCs have also produced and shared toolkits for grantees, other sector actors, and GC fund managers. These include a SWFF workbook on reaching women smallholder farmers, as well as a report on how to learn from failure, pivot, and adjust program focus as a result. SL@B produced guidance materials for grantees engaging with local health systems (including introductory and scale up materials applicable specifically to GH innovations),66 and training in the use of workbooks and toolkits. PAEGC created a toolkit on solar powered irrigation systems (SPIS) for innovators and investors. Within the GCs themselves, there may not be a common definition of what constitutes the relevant ecosystem or with whom, why, and when to engage. SOGE defined ecosystem engagement as a key objective in the original design of the GC, but this in itself was not sufficient to ensure effective engagement. There needs to be alignment among the funding partners regarding broader implications for implementation of such a focus. In the case of SOGE, funding partners agreed on the need to strengthen ecosystems as a key aspect of the design of the GC, but there were different expectations regarding the scale and scope of engagement and the nature of the support required from funders, fund managers, and the grantees themselves. The absence of an agreed-upon, formalized approach may also be seen in the limited degree to which GCs made use of other USAID (and partner) OUs, programs, and networks. SWFF worked directly with USAID Missions to align and find synergies with other investments in water systems. As well as having some involvement in the grantee selection process, USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 35 Missions helped create an enabling environment for SWFF innovators by facilitating discussions between them and local authorities. Zika also helped grantees work through other U.S. government channels and connect with multilaterals such as the WHO, and ACR GCD sought co-funding from USAID Missions and timed and targeted prize competitions around Ministry of Education needs as identified by Missions. These examples, however, appear to be the exceptions rather than the rule. This has led to an overall perception that these ‘in-house’ assets have been underutilized and could be leveraged more effectively to broker introductions, make connections between programs and actors in the same sector, and provide a deeper understanding of local systems and the particular contexts in which GC-funded innovations must operate.67 Other comparator programs at USAID that have focused on open innovation or promoting innovative approaches have built ecosystem mapping and engagement planning into program design from the outset. Partnering for Innovation adopts a portfolio, or ecosystem, approach to funding grantees, looking at not just the results and impact of individual innovators but at the impact of all grantees working in tandem to meet program objectives. It engages with different actors depending on the focus of different funding rounds (e.g., when funding vaccine development, the program worked with governments and research institutions to overcome financial barriers facing entrepreneurs looking to serve the smallholder farmer market). Partnering for Innovation also engaged with financial institutions or logistics firms when addressing supply chain bottlenecks. Local Works and SHOPS Plus are focused on finding systemic solutions to development needs and ensuring that a full complement of stakeholders is involved in agreeing and co-designing these needs: • Local Works program officers travel to different communities on ‘listening tours’ to hold dialogues with local actors on their experiences and priorities. Another tool for building relationships across ecosystems is the use of ‘Whole System in the Room’ workshops where USAID convenes all 67 Section 4.7 examines GC engagement with USAID Missions and operating units in further detail. 68 E.g., in Myanmar, USAID/Burma used the funding from Local Works to address the heroin epidemic in Kachin State. They spent three months scoping the challenge listening to local actors, and then convened a workshop of 111 people from civil society, faith-based organizations, government, charities and academia. These stakeholders later coordinated their efforts to create a system-wide approach, with initiatives such as women’s rehabilitation centers, employment schemes, awareness raising and expanded access to health care. 69 E.g., in Nigeria, initiatives focused on integrating private facilities into national Health Management Information Systems, working with local governments to ensure local area data was being integrated into national dashboards. Similarly, an initiative aimed at combating TB was a public-private partnership, with the government providing the drugs and the private facilities providing the services – the whole overseen by clinical associations. actors affected by a common issue to jointly lead the development of solutions.68 • SHOPS Plus engages with players across the public health ecosystem when designing and implementing projects in specific countries. It works with civil society organizations and media companies to spread healthcare messaging, with national governments to influence policy and support the contracting of NGOs or other non-state healthcare providers, and with local governance institutions to support public-private partnerships.69 4.4.1 SUB-EVALUATION QUESTIONS SEQ 4.1: WHAT TYPES OF PARTNERS/ INNOVATORS WERE BEST SUITED TO SUPPORT WHICH TYPE OF ECOSYSTEM? Innovations that have a direct bearing on government-led programs in education or public health are supportive of wider national ecosystems as there are opportunities to find synergies and strengthen existing policies, programs, and systems. Where innovations add to and enhance existing government initiatives or policy areas, they have the potential to achieve systemic impact at regional or national level. In the education sector, ACR GCD has secured successful system change in education in multiple geographies on the back of specific initiatives. In Morocco, the government changed education policy to cater for inclusion and accessibility for children with disabilities, while in the Philippines the program secured government consent to have the curriculum printed in braille while at the same time entering into data-sharing with the Department of Education through the USAID-funded Baa Pilipinas project in Mindanao. In India, an ACR GCD grantee, Pratham Education Foundation, was a key partner in a state￾wide reading improvement project involving 3,400 schools across 35 districts in Maharashtra state. Similar examples of uptake occur in the health sector, where collaboration between EBOLA, the CDC, and the DoD ensured that PPE improvements designed by grantees were tested to ensure that they met the latest protocols for use and therefore 36 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION met health system procurement requirements in disease-affected countries. Similarly, the mHero innovation, funded by EBOLA, was taken up by and received additional funding from the Liberian Ministry of Health as part of an electronic disease and surveillance response system (which later proved its efficacy and versatility by sending out early information on the COVID-19 pandemic to local health workers). The Zika program team helped grantees access other channels of support in U.S. government departments and the WHO.70 In parallel with these program-sponsored activities, some innovators, such as Premise Data, working with the Public Health Department in Cali, Colombia took the initiative in creating their own agreements with local governments. Creating a regional or local presence for GCs around a focused set of solutions concentrates energy and resources and delivers greater system impact. The PAEGC East Africa Hub, established in Nairobi (illustrated at Figure 4.4), allowed the GC to support both grantees and other innovators in the sector through multiple avenues of engagement. These included brokering long-term sources of funding through the Investment Alliance; establishing public￾private partnerships (notably with GiZ, the Kenyan Tea Development Agency, the Ethical Tea Partnership, Mars Drinks, and Bettys & Taylors of Harrogate); 70 For example to help grantees source testing specimens to validate the diagnostic tool (which had been a significant constraint); and to provide networking and ecosystem strengthening support in-country in some cases. 71 Note that CHIC operates in complex humanitarian environments where aid is often not reaching the people who need it due to government standing in the way of humanitarian action. In these situations, CHIC works with humanitarian actors and tries to drive regulatory change through these relationships. raising awareness of innovations by providing access to pilot site installations and by ensuring uptake of its SPIS toolkit by the AfDB; producing policy briefs on the benefits of clean energy technologies; and guiding innovators through policy and regulations in the clean energy and agriculture space. Against this example of targeted, locally-focused engagement, wider-ranging portfolios implemented in multiple geographies can struggle to influence systemic change due to the breadth, rather than depth, of their investments in ecosystems. CHIC’s portfolio, for example, was too broad to tackle regulatory change in any single country and it has not been easy for the program to achieve traction with policy makers in different countries, made especially difficult given the fact that these countries were all in the midst of humanitarian crises.71 SEQ 4.2: WHAT OBJECTIVES AND COMPONENTS, SUCH AS SOURCING, TESTING, AND SCALING, ARE ECOSYSTEMS (AND RELATED VALUE-CHAIN AND CLUSTER SYSTEMS) MOST RELEVANT TO SUPPORT? GCs benefit from engaging with ecosystems at different points along the program timeline. Early engagement can widen the pool of potential applicants, such as SL@B’s GC Grantee Public-Private Partnerships Training Programs Toolkits and Technical Assistance Awareness Building Community of Practice Market Intermediation • Partnership between GiZ, Ethical Tea Partnership, Kenyan Tea Agency, Private Sector Companies • Guides to policy and regulation in clean energy space • Policy briefs and advocacy • Solar Powered Irrigation System (SPIS) toolkit adopted by AfDB • Advice and coaching on business model development and pitching to investors • Training and coaching for GC innovator cohorts • Technical advice and expertise • Building links between other innovators operating in the clean energy sector • Opportunities for collaboration • Raising awareness among different actors • Visits to installations at pilot sites FIGURE 4.4: THE PAEGC INNOVATION HUB USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 37 DevelopmentXChange platform and Zika’s initiative to create a collaborative space for innovators and other partners in the unmanned aerial vehicle (UAV) sector during the application process. Targeted partnerships can strengthen the trialing of innovations with examples including ACR GCD’s partnership with Pratham in India, EBOLA’s collaboration with other U.S. government bodies to test PPE improvements, and PAEGC’s public-private partnerships detailed above. Finally, as has been shown with regard to all the GCs in the portfolio, network building creates different pathways to sustainability whether through government or multi￾lateral uptake and endorsement, private investment, or partnership with existing systems and initiatives. Creating an enabling environment for trialing innovations can be as important as the solutions themselves when it comes to their adoption. GCs need to work with governments and other decision-makers to address regulatory barriers facing innovators and, as already illustrated, there are many examples of GCs that have provided assistance to grantees in negotiating the complexities of local regulatory requirements. In addition, finding synergies with existing programs, partnerships, and systems increases the likelihood that other system actors will be receptive to and see the value of specific innovations. This, in turn, creates more opportunities for trialing innovations in different contexts and increases the likelihood of adoption either by commercial partners or policymakers. 4.4.2 FORWARD-LOOKING QUESTIONS FLQ 4.1: HOW CAN GCS MORE EFFECTIVELY INVEST IN LONGER-TERM ECOSYSTEMS STRENGTHENING GOING FORWARD? All GCs face similar challenges in engaging different ecosystem actors. A shared, portfolio￾wide approach will benefit donors, program managers, and grantees alike with the explicit understanding that different types of innovations have different pathways to scale (e.g., via public or private uptake), meaning that the precise mode and focus of ecosystem engagement will adjust to fit the relevant pathway. This has been noted in other evaluations: the 2020 Evaluation of SL@B highlighted the GC’s success 72 EVALUATING SAVING LIVES AT BIRTH – Evaluation Report: Rounds One to Eight (2011-2020) 73 A global movement launched during the United Nations Millennium Development Goals Summit in September 2010 that mobilizes and intensifies international and national action to address the major health challenges facing women, children and adolescents around the world. See https://www. everywomaneverychild.org/every-woman-every-child-innovation-marketplace/ in supporting innovators, but also pointed to areas of improvement.72 These include more intentional focus on engaging local public and private sectors early on in the design of a program, particularly in priority LMICs, and making more targeted connections. Specifically, the evaluation recommended early engagement with local government ministries to determine joint priorities, strengthen demand-driven innovation sourcing, and leverage existing networks (such as Every Woman Every Child73). These recommendations can be applied more generally across the GC portfolio as a whole. GCs can invest more effectively in ecosystem strengthening in the future by creating a shared and systematic approach to engagement across the portfolio, by building awareness and inviting different actors into the process from the outset, by taking greater advantage of existing USAID and donor partner networks in relevant sectors and geographies, and by identifying opportunities for high-impact, targeted interventions that have a direct and material benefit to grantees and the wider community of innovators. Examples drawn from other programs at USAID highlight the benefits of building clear ecosystem engagement plans and milestones to create opportunities for greater program impact. Identifying and supporting key enablers and growing the pool of actors invested in the success of innovations increases the potential for uptake and adoption and the likelihood of long-term sustainability beyond the term of each individual GC. Recommendation: Create a shared model, methodology, and practical toolkit for ecosystem engagement to be used by all GCs, building on a decade’s worth of experience among GCs and other programs committed to open innovation, providing a clear-sighted view of the costs of effective engagement and examples of how other GCs have budgeted for these costs. (Strategic recommendation for USAID Policy) • Create a portfolio-wide approach to ecosystem engagement at the strategic level. Create tools to assist ecosystem mapping such as identifying key touchpoints and engagement plans for connecting with different actors and identifying high-impact, targeted interventions. (Strategic recommendation for USAID Policy) 38 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION • Set aside human and financial resources to guide implementation of the engagement plans and to measure and gather lessons about their implementation to inform the design of future programs. As the pool of experience and insight grows, the toolkit should be updated and revised to reflect growing GC expertise in leveraging ecosystems to drive and support innovation. (Strategic recommendation for USAID Policy) • Dedicate time and attention at the outset to agree a common definition of what constitutes the relevant ecosystems for their GCs and an approach to engaging with wider system actors. (Programmatic recommendation for USAID GC Managers) • Apply the methodology and toolkit to the specific context of each GC. (Programmatic recommendation for Partner GC Managers) Recommendation: Include ecosystem engagement in program design from the outset, building connections with other USAID programs operating in the same space and with similar objectives. Use USAID’s convening power to forge early links between a wider network of innovators, experts, funders, and potential investors. (Strategic recommendation for USAID Policy; Programmatic recommendation for USAID GC Managers) • Engagement planning should both look for synergies with the objectives and priorities of other USAID programs, and draw on existing relationships which USAID Missions and other OUs have built up in sectors and geographies that intersect with the focus of GCs. These relationships should be leveraged to identify innovation needs and gaps, vet innovations, build awareness of GC innovations, identify potential partnerships with relevant system actors, and lay the foundations for adoption and future scale up. (Programmatic recommendation for USAID GC Managers) • Build cross-cutting portfolio-wide links between GCs, Missions, and other USAID OUs to encourage cross-program communication and shared learning. (Strategic recommendation for USAID Policy) • Leverage existing USAID relationships in sectors and geographies relevant to individual GCs. (Programmatic recommendation for USAID and Partner GC Managers) 4.5 CATALYTIC EFFECTS MEQ 5: TO WHAT EXTENT HAVE GCS CATALYZED FUNDING, DEVELOPMENT OF OTHER SOLUTIONS, FOLLOW-ON FUNDING, AND AWARENESS? Summary: There is good evidence that the GC model is effective at catalyzing funding and awareness. The model means that investment is de-risked and low-cost and therefore more attractive to investors. Catalyzing investment requires a clear strategy which is tailored to GC objectives and the investor ecosystem and adjusted as needed. Many factors support GCs’ catalytic effects. For the leveraging of funding, supporting factors are investments being de-risked; the provision of TA to innovators; the intentional engagement with investors; application criteria including match funding requirements; bringing in new actors likely to contribute funding; understanding and involving investors in the process; and taking steps to improve the investor readiness of innovators. All of these factors, many of which involve USAID working in partnership, contribute to positioning GCs better to catalyze funding than more traditional approaches. For awareness raising, supporting factors are demonstrating achievements, including through the use of results data and communications products; the credibility of donor backing; the convening power and brand value of GCs; and showcasing the distinctiveness of the GC approach. Recommendations: Apply learning about what has worked well in catalyzing funding and raising awareness to both the design and implementation of GCs, tailoring chosen approaches to the specific challenge and context. Establish a clear strategy for how a GC will achieve catalytic effects: establish requirements, such as securing match funding; provide TA on securing investment and becoming investor ready; set targets for catalytic results; support innovators to evidence and showcase achievements; leverage the brand value of donor funding; map the investor ecosystem (the context of investment actors, markets and policies) and involve investors in the GC; and showcase the distinctiveness and strengths of the GC model. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 39 FIGURE 4.5: FUNDING FROM EXTERNAL INVESTORS CATALYZED BY GCS, IN USD M74 74 Based on the evaluation team’s document review, catalyzing funding from other sources was not a specific project objective for ACR GCD, Zika, EBOLA and MAVC. 75 MAVC was not included in the survey. There is evidence that six GCs have leveraged substantial additional funds across their innovations. The most successful are SOGE, which has leveraged $645M through its partnership with other donors and seeding investment funds like Sima Fund in which $1M from USAID helped stimulate co-investment of $95M from private investors, and SL@B, whose innovators are estimated to have leveraged $160M of external funding during the award period and subsequently. See Figure 4.5 for further details. There is also evidence across GCs of funds leveraged by individual innovators. At the individual innovator level, over two-thirds of survey respondents agreed or strongly agreed that engagement in the GC had enabled them to access or generate further funding for their innovation (see Figure 4.6). These positive responses were received from all eight GCs surveyed.75 Additionally, almost every GC, including the less mature ones established more recently, provide examples of individual innovations which have secured additional funds (sometimes significantly so) and through this achieved wider impact as explored in Section 4.2. Several GCs built catalytic criteria into fund design. SOGE has “catalyzed investments” as one of its four stated pillars of the fund, PAEGC had the stated objective of catalyzing at least $25M in private sector finance for innovators (from two specific impact investors), and SWFF aimed to increase investment. SL@B did not have catalyzing investment as a key element of its intervention logic at the outset, but was nevertheless still successful in this. Requiring match funding and private sector partnerships are other examples of how GC design decisions support the achievement of catalytic effects (e.g., CHIC). The other GCs (ACR GCD, EBOLA, MAVC, and Zika) did not have catalyzing funding as a specific objective. FIGURE 4.6: GRANTEE SURVEY RESPONSES “Engagement in the GC has enabled us to access or generate further funding for our innovation” Strongly agree Agree Neutral Disagree Strongly disagree 37 percent 33 percent 14 percent 10 percent 6 percent 40 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Four of the nine GCs have required (or very strongly encouraged) successful applicants to use at least some of their funding rounds to secure match funding for their innovation. For example, applicants to CHIC’s TTS grants are required to demonstrate commitments from partners and stakeholders and, if awarded funding, are required to secure match funds. SOGE’s selection criteria includes the amount of private capital which USAID grant funding would unlock, at a minimum 2:1 ratio, and sought partnerships with the potential to attract a much higher ratio (10:1). This approach ensures that innovators are thinking about sustainability and the practicalities of scaling from the outset. All GCs provide TA to innovators, often including specific support for securing investment. Alongside support for the innovation itself (design, marketing, scaling its reach), TA extends to facilitating investment. SOGE provided investment preparedness workshops and convened investor circles to bring together investors and select innovators for structured pitch presentations, and PAEGC had a model for linking innovators to investors. Referrals from USAID/partners and networking or investment opportunities were cited by 41 percent of survey respondents as ways in which the GC had helped them to raise further funds for their innovation. Catalyzing investment requires a clear strategy that is tailored to GC objectives and the investor ecosystem and that it be adjusted as needed. To illustrate, the SOGE team pivoted their model to investing in financial intermediaries rather than in individual companies to catalyze investments, blending small amounts of donor capital with various financial instruments to decrease the risk for commercial investors. This contributed to their success. Evidence from interviews, supported by results, was that SWFF was less successful in catalyzing private funding and had not developed a clear strategy to identify investors, investment firms and foundations with funding strategies which aligned with SWFF goals. Funding is also catalyzed by building investment readiness so that, for example, innovators can meet investor due diligence requirements. 33 percent of survey respondents said that their increased organizational capacity, through the GC, had helped them raise further funds for their innovation. The converse is true as well: only a handful of PAEGC innovators were investment worthy on paper, and fewer actually attracted investment, suggesting that insufficient attention had been paid to enabling innovators to meet investment criteria. Investor readiness TA was also rated poorly by SWFF participants, a fund which was also less successful in catalyzing private funding. GCs can play a catalytic role in raising awareness about funded innovations both locally and internationally. Just over 90 percent of survey respondents agreed (27 percent) or strongly agreed (63.5 percent) that the engagement in the GC had contributed to increased awareness about their innovation in the context in which they worked. Awareness outside the immediate innovation context has also been raised: SL@B used an innovation platform to raise awareness of the challenge and crowdsource solutions which led to 4,444 applications being received across its eight funding calls. SWFF dedicated considerable resources to communications throughout the program. Zika surfaced a number of leading technologies that could be used for protection, diagnostics, and wider population health. A wide range of marketing activities showcased this work, successfully demonstrating impact to actors within the public health community. By contrast, partners in SOGE did not all conceive of SOGE itself as a vehicle to engage in external communication and awareness raising, rather as an internal coordination platform. Most communication took place under partners’ own branding as it took some time for SOGE to develop a clear brand which satisfied all parties. Several GCs have results indicators that measure catalytic effects. Evidence is important to demonstrate the effectiveness of GCs for catalyzing funding. Both SWFF and CHIC for example track and report the number of innovators establishing or increasing partnerships to support scaling, the amount of additional funding secured and also, at output level, communications materials produced and shared. 41 percent of survey respondents had used evidence (e.g., impact data, evaluations, case studies) and communication materials generated by the GC in fundraising. 4.5.1 SUB-EVALUATION QUESTIONS SEQ 5.1: DO GCS GENERATE OR CATALYZE MORE FUNDING THAN USAID CAN WORKING ALONE? Catalyzation of funding is a key feature of and is supported by several aspects of the USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 41 GC model and has been achieved. Further, KII evidence is that GCs can catalyze more funding than USAID’s more traditional implementation approaches. The achievement of impact at scale by GCs is largely predicated on ‘crowding in’ the private sector and leveraging additional funds for innovations that are tested and developed through GC funding, and which are thus ‘de-risked’ for investors. This ‘de-risking’ aspect of GCs is particularly important for those that, like Zika, are consciously open to potentially game-changing innovations but which might also be higher-risk investments and therefore less attractive investments in their own right. SL@B’s convening power and networks enabled it to catalyze multi￾sectoral collaborations and partnerships for several high-impact innovations (e.g., Rice University’s NEST 360 Partnership; Gradian), in the process both de-risking investment in innovations and driving advancement in the MNH field more widely. The innovation mindset of GCs could also mean that they are better placed to explore new approaches that then prove successful (such as the innovative blended finance facility introduced to SOGE’s second funding round in late 2016 through which they partnered with three debt-based capital actors in the energy space to bring in investment). There is evidence that the ‘brand value’ of donor funding (USAID and partners) through a GC and partnership working has a catalytic effect. There is evidence from interviews that the brand power of some GCs (e.g., SL@B) enhances partner reputation and visibility and has enabled some innovators to attract new partners and additional funds. One GC Manager observed that donor support provides innovators with “real credibility when seeking funds from elsewhere” and a 76 USAID Meta-Evaluation Scheme of Work p. 14 “Fund raising and partner engagement [were particularly successful]”. SL@B grantee in responding to survey questions on catalyzing effects CHIC grantee also commented that the “GC gave us a lot of credibility. The innovation week and networking events are very useful”. GC programs can leverage the brand value of the donor partnership to draw attention to both the challenge it is addressing and the portfolio of solutions being developed. There is strong evidence from SOGE that working with funding partners has an amplifying effect on the seed finance available. All the comparator programs aimed to play a catalytic role, leveraging additional resources through match funding or co-investment by funding partners. The most successful at catalyzing funding, IIA, was the one where USAID worked with Skoll rather than alone ($160M). Strong collaboration and partnerships which underpin implementation also contribute to success. The GC model, through its TA, brings together a range of implementing partners specifically to help innovators access investment and financing and to connect with potential customers and partners.76 SEQ 5.2: WHAT ARE THE KEY FACTORS THAT ENABLE GCS TO ATTRACT FUNDS FROM OTHER SOURCES, LEVERAGE ADDITIONAL DONOR FUNDS, AND RAISE AWARENESS? Raising awareness is itself a factor that can lead to leveraged funding, so the two are closely linked. From evidence across the GCs, there are several key factors as outlined in Box 4.3 and Box 4.4. BOX 4.3: GC ABILITY TO ATTRACT FUNDS – SUPPORTING FACTORS De-risking investment: Private investment for scaling is de-risked, and therefore a more attractive proposition, by typically being made alongside donor funding into innovations made ready for scaling through donor funding, into an innovation which has been through USAID due diligence processes, and which is being supported by technical assistance. Shorter, innovation-level funding commitment: Investors typically invest in an individual innovation established through GC funding rather than invest in the GC as a whole. Evidence suggests that this is more attractive. Technical assistance to innovators: TA to build organizational capacity and support innovations increases the likelihood of success. Intentional engagement with investors: TA can include bringing innovators and potential investors together in one-to-one discussions and wider convenings. 42 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Application requirements: Requirements to submit scaling and sustainability plans and secure match funding commitments embed catalytic effects. Bringing in new actors to the agenda being pursued by the GC: New actor engagement likely contributes to additional private and public investment in innovation. Understanding and involving investors: Approaches include thoroughly mapping the investor ecosystem and the early involvement of investors in the program (e.g., at design stage, through brokering relationships, in the provision of TA, etc.). Steps to improve investor readiness of innovators: These steps include aligning innovator selection criteria and TA for organizational capacity building with investor expectations and objectives. BOX 4.4: GC ABILITY TO RAISE AWARENESS – SUPPORTING FACTORS Demonstrating achievements: Collection and use of results data, learning and production of other communications products, and engagements to demonstrate innovation effectiveness (by both innovators and at program level) is effective. Credibility provided by donor backing: Donor branding assures potential investors. Convening power and brand value of the GCs: The GC brand both supports individual innovators and builds the profile of the challenge as a whole. The distinctiveness of the GC approach as a whole: The approach itself is of interest to a wider audience and a good pitch to potential investors. Key features include how the GC model can de-risk investment and take success stories further with additional funding. 4.5.2 FORWARD-LOOKING QUESTIONS FLQ 5.1: HOW CAN GCS OPTIMIZE THE CATALYTIC EFFECTS ON FUNDING AND RAISING AWARENESS? GCs can optimize catalytic effects by building on the above success factors and lessons learned. Recommendation: Apply learning from the evaluation about what has worked well to catalyze funding and awareness to both the design and implementation of GCs, tailoring chosen approaches to the specific challenge and context. (Strategic recommendation for USAID Policy; Programmatic recommendation for USAID and Partner GC Managers) • Establish a clear strategy for how the GC will achieve catalytic effects, both at design and in implementation. • Establish requirements, e.g., that applicants have secured match funding or commitments, that they include scaling plans, or that applicants communicate their innovations through, for example, externally facing material backed up by dissemination plans. • Provide TA to innovators on securing investment (e.g., business plans, investor networks) and measures required to become an ‘investment ready’ organization. • Set targets for catalytic results (e.g., investment secured, communications materials produced and shared), measure progress, and course correct if needed. • Support innovators to generate and use MEL data to evidence and showcase achievements to promote investment and raise awareness. • Leverage the brand value of donor funding and the GC itself to convene external actors locally and internationally, raise awareness, and attract investment. • Map the investor ecosystem for a given context (e.g., who are the potential investors?, what are their key considerations and requirements?) and involve investors in the program (e.g., at design, in brokering relationships, in TA provision). Importantly, ensure that the development objectives of an innovation (e.g., tackling poverty) are not diluted in an attempt to attract certain investors. • Showcase the distinctiveness and strengths of the GC model to attract investment (e.g., rigorous selection processes, TA support, that it de-risks investments). USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 43 4.6 PROCUREMENT AND REDUCING BARRIERS MEQ 6: TO WHAT EXTENT ARE GCS ACCESSIBLE TO ALL TYPES OF INNOVATORS? Summary: Reaching and funding ‘non-traditional development actors’ – those who would not normally access USAID funding – is an important way of achieving GC objectives and was a feature of seven out of nine GCs. GCs have been designed in several ways to reach non-traditional actors, as itemized, and several GCs took explicit steps to increase accessibility. For all five funds for which there is data, the percentage of awardees from LMICs is below 55 percent and between 7 percent and 17 percent for three of these. Positively, over three quarters of grantees responding to the evaluation survey had not received funding from USAID in the past five years. The length and complexity of the application process – and grant set-up for successful candidates – can be difficult (and disproportionately so for small and grassroots organizations). Political economy analysis is important for generating applications and selecting stronger ones. Open, well-publicized calls with broad eligibility criteria generate a very high volume of applications: the GCs had 15,966 applications as of April 2021 between them. The average success rate is low (4.8 percent of all applicants receive an award) which means that substantial effort is expended by fund managers and by applicants for every successful application, but efficiency is arguably a trade off with the core GC principle of open innovation. Looking at data on applications versus awards, there is evidence that applicants from LMICs or who have not previously applied for USAID funding find it harder to access GC funding. It is reasonable to assume that the converse is true: that developed country applicants and previous awardees are better positioned to be awarded GC funding, although the former might not always be best placed to develop an innovation in context. Private sector actors and local innovators can also find accessibility a challenge due to unfamiliarity with the language, concepts, and processes around development and innovation which are used by high income country GC donors and implementers. The use of innovative methods to reach applicants needs to be carefully appraised to avoid being counterproductive. Recommendations: GCs can increase efficiency by taking steps to deter ineligible or weak applications and funnel applications effectively through a staged application process. They should define target grantees carefully in relation to fund objectives and tailor application processes to increase accessibility. They should conduct research to understand and support the local market and generate local applications, and undertake local analysis of the context and political economy to inform local outreach and selection criteria. 77 Not MAVC or EBOLA. GCs are established to surface and support innovations to address a specific development challenge. Reaching and funding ‘non￾traditional development actors’ – those who would not usually access USAID funding – is seen as an important way of achieving this and was a feature of seven out of nine GCs studied (and three of the four comparator programs).77 Non-traditional actors include organizations based in the communities who would potentially benefit from the innovation, social enterprises and private sector innovators and partners, and research and academic institutions in target (i.e., LMIC) countries. For CHIC, the focus on local (non-traditional) partners is intended both to ensure the relevance of solutions to need and to source solutions which are potentially much cheaper – a critical point when humanitarian need outstrips available funding. GCs have been designed in several ways to reach non-traditional actors, including: • Mandating that at least 50 percent of the awards would go to local organizations (ACR GCD in Round 2). • Conscious intent to target the private sector (SOGE and SHOPS Plus comparator program). • Having funding explicitly available to for-profit companies, NGOs, academic/medical research institutions, CSOs, etc. with no restrictions on the type of organization or disease focus (SL@B). • Encouraging applications from affected populations (CHIC – communities affected by conflict) and a requirement to at least engage affected populations in the proposal (CHIC and Local Works comparator program). 44 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION • Innovator-specific training and assistance, a focus on applying using storytelling, and pitching not relying solely on written forms (SWFF). As another example, Local Works specifies that applicants must be NGOs, educational institutions, or another kind of entity which has accessed less than $5M of USAID funding in the last 5 years in order to be eligible. Several GCs took steps to increase accessibility or explicitly focus on non-traditional actors after their initial funding rounds resulted in a portfolio which was not particularly diverse: having an open call for proposals was not in itself sufficient to attract non-traditional actors who faced several barriers to applying or being selected. To illustrate, in SL@B, which aimed to source innovations from diverse geographies including LMICs, only 17 percent of awards (19 percent of the funding total) went to organizations with headquarters in LMICs. The fact that several GCs struggled to reach their desired audience in early rounds indicates that they had not undertaken good research into the potential applicant market at the outset. Despite this, steps taken subsequently by GCs to increase accessibility included the following initiatives: • Making application information available in local languages (ACR GCD). Allowing applications to be made in French, Swahili, and Arabic, as well as English, and simplifying the application process, to increase reach in more conflict-affected areas (CHIC). 78 Analysis which Triple Line has undertaken for Sida on integrating cross-cutting issues (such as gender equality) into Challenges highlights the correlation between diversity and inclusive practices within implementing organizations and positive project outcomes for equality. • Widening diversity by including projects in francophone countries and projects led by women as output-level results framework indicators (CHIC) and producing a workbook on reaching women smallholder farmers (SWFF).78 • Engaging outreach consultants to raise awareness of the fund and reach out to potential investees (SOGE, MAVC, SWFF, and CHIC, as well as Partnering for Innovation, a comparator fund). • Using learnings to change the type of innovator targeted to better fit needs and reduce the burden of applying directly to USAID (e.g., SOGE who moved from funding energy companies to funding financial intermediaries through which applicants could apply). • Encouraging interdisciplinary collaborations, some of which brought in first-time entrants to the maternal and neonatal health arena (SL@B). • Support to applicants by the GC Manager and local agents, e.g., via information sessions or email responses to questions. • Reducing legal complexity and associated administrative costs of initial cooperative agreements with grantees (SWFF). • Introduction of “state of innovation analysis” (SWFF), a learning point subsequently applied by other GCs. There is data on the percentage of awardees from LMICs (which likely overlaps quite strongly with non-traditional actors) for five GCs. The percentage ranges from 55 percent to 7 percent, with three GCs at 17 percent or below (see Figure 4.7). FIGURE 4.7: PERCENTAGE OF LMIC AWARDEES USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 45 TABLE 4.3: GRANTEE SURVEY RESPONSES – PROCUREMENT QUESTION PERCENT OF RESPONDENTS WHO AGREED OR STRONGLY AGREED The language in the call for proposals was easy to understand 94 percent Information about the application process was easy to understand 90 percent The application selection criteria were clear 68 percent Support from USAID/the GC Manager was available if needed 68 percent The time required to complete the application process was reasonable 80 percent 79 Analyzed further in Section 4.6.1. Positively, over three quarters of grantees responding to the evaluation survey had not received funding from USAID in the past five years. The percentage of respondents who agreed or strongly agreed about a number of topics are highlighted in Table 4.3. There were also several comments from survey respondents about the support provided: “We found the USAID team extremely responsive, rigorous and helpful in guiding us through the set-up process” – SOGE grantee. The length and complexity of the application process and of grant set-up for successful candidates can pose real difficulties for some applicants.79 • One PAEGC applicant compared the application process favorably to other grant funds, but another said that the vetting time between being shortlisted and being awarded was very drawn out: “This eliminated much of what we had planned to use for the cost share requirement, creating additional stress in coming up with additional funds”. • Similarly, a SL@B grantee whose funding round took eight months said that “the application process became prolonged, and this disallowed matching funding that we had hoped would apply. This created significant stress after we had already passed through many hoops”. • A CHIC grantee found that the paperwork and due diligence between award and signing the grant agreement required significant investment of staff time across many months. Long and demanding processes are likely to be disproportionately difficult for small and grassroots organizations to manage and comply with due to resource constraints and cash flow pressures and could act as a deterrent or burden for them. The survey was only of grantees (i.e., successful, not unsuccessful applicants), but evidence from KIIs is that donor compliance and due diligence requirements are too high a hurdle for many applicants. One CHIC grantee noted that they were “still learning about financial reporting and my team is still learning. I’m glad that the GC is helping us through this process as the learning curve is somewhat steep”. Non-traditional actors are not necessarily the most suitable. For example, there is a potential tension between wanting to reach non￾traditional partners and wanting to achieve high levels of scale, for which certain partners might be ill-equipped. SL@B’s willingness to invest in ‘high risk’ but potentially game-changing early-stage innovations and achieve scale quickly might have contributed to the higher proportion of innovators from high-income countries (HICs) (including from non-traditional actors), some of whom knew very little about their proposed lower-income market or intended country context, contributing to challenges in achieving scale or sustainability. The high number of young U.S. university-based teams in EBOLA likely also faced this challenge. From Zika, a key interviewee’s analysis was that while funding a large number of non-traditional actors was necessary for the intensive scientific research required, there were mismatches: academic institutions and universities sometimes do not have the incentives to scale that a business would and there could be a lack of demand in LMIC markets for products developed by U.S.- based innovators lacking knowledge of the context. Political economy analysis is important to generating applications and selecting stronger ones. One of MAVC’s problems was identified as a lack of effective political economy analysis (which identifies where power lies and who is positioned to influence it) to inform the encouragement and selection of suitable applications to a fund seeking to enhance voice and accountability. In SL@B, broader bottlenecks for addressing MNH problems in most countries – e.g., systemic issues relating to financing, human resources for health issues in LMICs, and market fragmentation – have to be taken into account if an innovation is to scale 46 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION effectively. Innovators with strong local links (e.g., to government partners and the private sector) may be more likely to achieve scaling success, but this might not be enough if they do not also know how to work within systemic and political constraints. 4.6.1 SUB-EVALUATION QUESTIONS SEQ 6.1: TO WHAT EXTENT ARE GCS ACTUALLY MORE EFFICIENT THAN OTHER USAID FUNDING MECHANISMS? Open, well-publicized calls with broad eligibility criteria generate a very high volume of applications: the GCs had 15,966 applications as of April 2021 between them. After receiving over 400 proposals for its first funding round, ACR GCD introduced a two-step application process consisting of a short concept note followed up by proposals from selected grantees to make the selection process more manageable. Most GCs also funnel applications, for example by undertaking an initial screening exercise to quickly weed out ineligible applications (of which there can be a large number – 55 percent for CHIC’s first funding round), undertaking an innovation screen, and having a shortlisting process. These steps increase efficiency, but more curated funding mechanisms, with their targeted approaches avoiding open calls, have a much higher applicant-to-grantee ratio and are therefore arguably more efficient. VentureWell, TA providers for PAEGC, outlined how an alternative ‘intake model’ might operate by giving a ‘small amount of finance to test, find out about strategy, business model, coachability [i.e., not for achievement of results], then provide further financial 80 WE4F is an example of a dispersed model which is successful; MAVC of one which was not. How well it is implemented is key. 81 DFID, Year 1 Annual Review of CHIC, 2019. support for the next stage’ in order to be sure that the right applicants were supported and increase efficiency overall by reducing the rate of failure. One interviewee questioned the effectiveness, efficiency, and reach of GCC’s centralized hub in Canada as opposed to a more dispersed model in geographies where projects are being implemented.80 It takes some GCs many months to progress from launch to award. As well as creating real problems for applicants, as noted in Section 4.6, lengthy timescales can be difficult for donors who have financial schedules and political pressure to disburse funding. The eight months from CHIC’s launch of Round 1 (February 2018) to award (September 2018) was a concern for FCDO, for example,81 but Round 2 was actually slower (mid￾2019 to August 2020). Table 4.4 shows the time elapsed between the closure of funding calls and the announcement of awards. There is considerable variation, with the shortest time elapsed being three and a half months (ACR GCD) and the longest being over 15 months (SL@B). The average application success rate is low. This means a substantial effort is expended by fund managers and by applicants for every successful application. The 15,966 applications resulted in 767 awards, an average success rate across all GCs of 4.8 percent. Success rates in different funds ranged from 0.9 percent (EBOLA) to 7.6 percent (ACR GCD). Efficiency is a trade-off with a core principle of GCs – open innovation – whereby the net is cast wide in order to identify new innovation actors and new ideas. TABLE 4.4: TIME ELAPSED FROM CLOSURE OF FUNDING CALL TO ANNOUNCEMENT OF AWARDS ACROSS ALL GC FUNDING ROUNDS FOR WHICH DATA IS AVAILABLE METRIC TIME ELAPSED Mean 8 months and 13 days Median 7 months and 7 days Maximum (Saving Lives at Birth) 1 year, 3 months, and 6 days Minimum (All Children Reading) 3 months and 15 days USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 47 SEQ 6.2: ARE CERTAIN TYPES OF INNOVATORS BETTER POSITIONED TO SUCCEED IN GCS THAN OTHERS? WHY? There is evidence that applicants from LMICs or who have not previously applied for USAID funding find it harder to access GC funding. • CHIC and SL@B have both taken steps specifically to make their funds more accessible to LMIC applicants, as noted in Section 4.6. In EBOLA, 54 percent of total applicants were from LMICs, but only 14.3 percent of awardees (2 out of 14) were, the rest being for multi-country projects run by organizations from HICs. • In SWFF, a high 76.5 percent applicants had not previously accessed USAID funding, but the proportion of awards to this group was 57.5 percent. It is reasonable to assume that the converse is true as well: that HIC applicants and previous awardees are better positioned to be awarded GC funding, although the former might not always be best placed to develop an innovation in context. Whether HIC organizations have accessed funding before or not, they are more likely to be able to meet donor compliance and due diligence requirements, for example in relation to financial management and governance. A SL@B grantee reported, for instance, that “the number of person-hours to stay compliant was unnecessarily large and typically required senior finance or C-level involvement, distracting from the project objectives”. Western innovators are also better able to work comfortably in English which is the working language of GC donors and managers. A successful applicant is not necessarily, however, a successful innovator: PAEGC, for example, found that innovators not based in markets they are targeting (e.g., U.S.-based companies or HIC universities) can lack understanding of context and need support to make their innovations relevant. Private sector actors and local innovators can also find accessibility a challenge due to unfamiliarity with the language, concepts, and processes around development and innovation which are used by HIC GC donors and implementers. For example, SOGE worked mostly with private sector actors and with some policy makers and market enablers, but USAID mechanisms are very different from those for venture capital and it takes companies a significant amount of time to become acquainted 82 Evaluation of Amplify: http://iati.dfid.gov.uk/iati_documents/45213885.pdf with different expectations and to understand the legal implications. (This was one of the reasons why SOGE pivoted to working through financial intermediaries.) Accessibility difficulties for the private sector are significant given that its increased involvement was one of the original intentions of the GC model (see Section 1.2) and is still important, as confirmed through strategic USAID KIIs. In EBOLA, interviewees believed that the centralized U.S. administration and low engagement by Missions contributed directly to low participation by local innovators. The use of innovative methods to reach applicants needs to be carefully appraised to avoid being costly and counterproductive. EBOLA used the OpenIDEO platform to generate and develop applications, but it might not have helped to attract local innovators. While the platform has global reach and creates an online community, another evaluation undertaken by Triple Line found that many LMIC innovators found it highly resource intensive to engage with and that those in Africa found its deadlines in U.S. time zones very difficult to manage.82 4.6.2 FORWARD-LOOKING QUESTIONS FLQ 6.1: HOW CAN USAID AND GC PARTNERS FURTHER REDUCE BARRIERS TO PROCUREMENT EFFICIENCY AND TO THE INCLUSION OF NON-TRADITIONAL ACTORS/INNOVATORS? Generating enough good quality applications from target organizations is difficult. GCs need to strike a balance between deterring weaker applications in order to increase fund efficiency and lowering barriers to entry to increase the inclusion of non￾traditional actors. GCs must understand who non-traditional actors are, what they can offer, and the nature of target markets and the innovation landscapes in which they want to fund projects in order to inform the many steps they can take to attract the right applicants and reduce barriers to their inclusion. Recommendation: Take steps to deter ineligible or weak applications and funnel applications effectively through a staged application process after initial eligibility screening has taken place. 48 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION (Programmatic recommendation for USAID and Partner GC Managers) Building on steps taken by some GCs, as above: • Have very clear and specific eligibility criteria to deter applications from ineligible organizations. Communicate these criteria effectively. Consider application processes which include automated online screening mechanisms so that ineligible applications cannot progress. • Have very clear selection criteria so that applicants know what is wanted and what is likely to succeed. (Over a quarter of survey respondents said that it was not clear to them how their application would be reviewed or the criteria for selection.) Include checklists. Provide examples, Q&As, frequently asked questions (FAQs) with answers, and helplines. (30 percent of respondents gave a neutral or negative response when asked whether they could contact the fund manager easily and ask for help applying.) • Request a short concept note which enables weak or ineligible applications to be screened out, followed by a full proposal from a shortlist (like ACR GCD). Determine a suitable ratio of applicants-to-awards so that a manageable number are shortlisted.83 • Review each funding round (its statistics, processes, and outcomes), identify lessons, and apply them to subsequent rounds. Recommendation: Define target grantees carefully in relation to fund objectives and tailor application processes to make them accessible to these organizations. (Programmatic recommendation for USAID and Partner GC Managers) • Undertake innovator landscape research in context and internationally to determine which types of organization (or innovation) should be targeted: is there a cluster of innovations in certain fields? What types of organizations have successfully developed innovations? Where are they located? Are there local research hubs to tap into? • If non-traditional actors are to be targeted, be clear about which types of actors and why. Do not assume that non-traditional is the same as most suitable. 83 The evaluators suggest a ratio of shortlisted applicants-to-awards of no more than 8:1. A ratio of 3:1 has been used for other donor funds (e.g. FCDO’s Global Poverty Action Fund) without compromising quality. • Set eligibility and selection criteria to match the type of actors you are targeting (for example, their size, locality, or previous receipt of donor funding). • Tailor communications and processes to match the actors you are targeting (such as language, use of terms and concepts, time demands, and choice of platform). • Ask successful applicants what improvements to make to the process. Recommendation: Increase the quality and quantity of applications from local innovators by conducting research to understand and support the local market. (Programmatic recommendation for USAID and Partner GC Managers) • Scope the local innovator landscape to engage with potential applicants on the ground in order to market the fund effectively in context, identify and encourage applications from promising innovators, and determine how they can best be supported to apply. Involve USAID Missions. • Provide support to applicants, for example by holding events where the selection criteria and how to respond to the questions in the application are explained, by providing online Q&As and FAQs, or by finding local champions. Recommendation: Undertake local and regional analysis of context and political economy to inform local outreach and selection criteria. (Programmatic recommendation for USAID and Partner GC Managers) • Use context and political economy analysis to understand local enablers and inhibitors of innovation and its scaling. These might include systemic issues such as financing for health systems or market fragmentation. • Use understanding of these issues to inform how local innovators are targeted, selected, and supported, particularly in relation to scaling and ambitions to positively impact the wider system (e.g., health, humanitarian). USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 49 4.7 ENGAGING USAID MISSIONS AND OPERATING UNITS (OUS) MEQ 7: TO WHAT EXTENT HAVE GCS EFFECTIVELY ENGAGED USAID AND OUS, AND WHAT LESSONS CAN BE LEARNED FROM ENGAGEMENT EFFORTS TO DATE? Summary: GCs engage with USAID Missions to a limited extent – some successfully, and some not at all, depending on the approach and geographical focus of the challenge – but it was commonly understood that GCs can and do benefit in many ways from engagement, including in relation to relevance, sustainability, and ecosystem linkages. The benefits of engagement are mutual: through GCs, Missions can potentially access a cost-effective and easy-to-use funding mechanism to further their priorities in-country. There are some good examples of other USAID programs successfully engaging with USAID Missions that GCs could learn from. Recommendations: Plan how to engage USAID Missions upfront and use engagement to inform plans for designing and implementing the GC at different stages, depending on the GC’s geographical scope. A GC with clear geographic focus could consider collaboration at the outset while GCs with global scope may prioritize and reach out to particular Missions after awards are made. Recommendations include suggestions on how this might be done. GCs should engage with USAID Missions to add value, recognizing that they are independent agencies which differ markedly in priorities, resources, programing, and ability to engage. Engage Missions to strengthen ecosystem change and longer-term MEL. GC engagement with USAID Missions has been limited. SWFF had engaging Missions as a stated priority and worked directly with them: Missions were kept informed about projects, were involved in the selection process through Mission concurrence forms (as they are on every GC in fact), and provided information used in due diligence analysis. They also helped to create an enabling environment (as outlined in Section 4.4) Other GCs did not have this degree of engagement. Some GCs did involve Missions from the outset to some extent, including in advertising the GC through the Missions [ACR GCD, SL@B, SOGE (particularly in Nigeria and Uganda), and Zika for some awards], but others involved them far less or not at all (CHIC, EBOLA, MAVC, and PAEGC). Mission engagement depended in part on the nature, sector, and geographic coverage of the GC. Where GC coverage was global, experience showed that it was much more feasible to engage Missions later, once awards were declared and projects got underway. For example, ACR GCD ensured that some of their grantees, such as eKitabu and Little Thinking Minds, remained in touch with Missions throughout the award period. It was commonly understood that GCs can and do benefit in many ways from engaging with Missions and that not engaging was a missed opportunity. • Missions can help GCs to be more relevant to the needs in-country. They can facilitate better understanding of the context to inform on-the-ground engagement with innovators. For SL@B and Zika, USAID Missions helped to make the innovations contextually relevant by providing inputs or suggestions on proposals coming from their country (although not consistently so in light of some Zika grantee feedback). By contrast, a USAID official was of the opinion that lack of engagement of Missions in the EBOLA GC (which was run directly from Washington, D.C.) not only resulted in fewer applications from local and in-country actors, but meant that both the Missions and the GC missed the opportunity to collaborate and learn together. “I remember that we had to push country Missions to include some of our projects… GCs are often funded out of D.C. and fail to work closely with country Missions. Missions are also very busy. In the case of our GC, Missions weren’t necessarily as on board as we’d like for a lot of the different activities. As a result, when Missions went to create their next version of their Country Development Cooperation Strategy, they did not incorporate learnings from projects funded by the GC. We were treated as something quite separate”. KII with a GC representative who supervised a scaled project in a region 50 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION • Where Missions can see the value of innovator activities, they can contribute to their sustainability after the end of GC funding. ACR GCD was successful in this way, securing funding from USAID Mali Selective Integrated Reading Activity (SIRA) to establish 42 additional community libraries and provide household training on reading games and activities in more than 300 communities. One of the EBOLA awards that focused on Liberia, mHero, collaborated with the USAID Liberia Mission which contributed to it being scaled up countrywide. • Missions can facilitate local relationships with innovators and effect linkages to important local stakeholders. EBOLA suffered from innovators from outside the country not having local connections with the Mission and in-country government to affect system change. These innovators needed a lot of scaling support from the GC as a result. • Missions can assist with tackling regulation challenges in a country or investing in local innovation hubs (which are particularly helpful if the portfolio has a strong presence in the country) which could strengthen the local innovation ecosystem. • Missions could provide assurance on alignment of GC activities with the country or regional Country Development and Cooperation Strategies (CDCS). The benefits of partnership can be mutual: those Missions which choose to engage with a GC have potential access to a cost-effective and easy-to-use funding mechanism to further their priorities in-country. This is significant given that Mission resources are limited and that by tapping into a central procurement mechanism they do not have to invest time and resources in setting up their own programs. The active engagement between ACR GCD and Missions in Jordan and Morocco benefited the Missions by expanding their collaboration with local ministries and initiatives. For global programs operating across multiple territories, engagement with Missions may be different from those with a strong regional focus. For both, engagement is contingent on the degree to which program objectives match Mission priorities. Regionally focused GCs such as Zika and EBOLA may be better placed to create links with Missions than global GCs, although a key factor for both regional and global GCs is the number of grantees located in a single country. Where a GC is supporting a cluster of innovations in a single territory, there are increased opportunities to build these connections and collaborate, and the potential for economies of scale in time, effort, and resources for the GC and the Mission alike. Conversely, Missions have less incentive to engage where there are only a handful of GC-funded projects in-country and the totality of the program is not evident to a Mission. In these instances, and where Missions are designing large projects in-country to tackle systemic problems, GC presence can appear insignificant and therefore hamper effective working relationships. Mission engagement will always be determined by the degree of GC alignment with the CDCS. There are some good examples of other USAID programs successfully engaging with USAID Missions which GCs could learn from. These programs often evaluated the feasibility of the Mission’s engagement, developed programs jointly with the Missions, and supported their local initiatives. For example, USAID managers from Feed the Future Partnering for Innovation spent time at the outset with Missions to do this. Some Missions, such as Mozambique, have chosen to collaborate on multiple occasions while others have engaged just once. Conversely, a number of Missions chose not to buy into the program either because its objectives were not aligned with their own development priorities or because the program was perceived as duplicative of existing initiatives. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 51 BOX 4.5: USAID MISSION ENGAGEMENT COMPARATOR EXAMPLE USAID’s Local Works program was specifically designed to work with Missions wanting to trial or take locally-led development initiatives further.84 The program provides funding, flexibility, and technical assistance, and works with a minimum of three Missions per annum, connecting them through a community of practice supported by a newsletter and a quarterly call.85 Missions take part in these funds and opt in by demonstrating that they too are facing the particular problem that the program focuses on and would welcome GC-funded projects in their own territory. SHOPS Plus, USAID’s flagship initiative in private sector health, is able to run 8-10 projects simultaneously (current projects span the globe from the Caribbean to Sub-Saharan Africa and Southeast Asia). SHOPS Plus provides a field support mechanism for Missions buying into the program and supports Missions to increase their capacity and understanding of the requirements of private sector engagement and market coordination. 84 A portion of funding was set aside for unsolicited proposals, but these would still have to be directed at specific, locally led initiatives. 85 Mission funding goes to TA in the form of MEL, communications, policy, context-specific research and funding for listening tours to understand the nature of local needs. 4.7.1 FORWARD-LOOKING QUESTIONS FLQ 7.1: HOW CAN GCS BE INTEGRATED INTO USAID AND GC PARTNER MISSION PROGRAMING GOING FORWARD? Interview evidence with USAID gave pointers to how GCs might engage effectively with Missions and the evaluation has developed a model informed by the comparator analysis which could prove fruitful. Engagement of USAID Missions into GC programing is also considered. Recommendation: Plan how to engage USAID Missions from the start and use engagement to inform plans for designing and implementing the GC depending on its geographical scope. (Strategic recommendation for USAID GC Managers) For GCs with a specific geographic focus (or when targeted countries are known), such as EBOLA: • Research the geographical distribution of the problem the GC is addressing and how the problem varies by location in order to identify a sample of Missions who could be approached to join design activities for the GC. • Reach out to these selected Missions to better understand their needs and priorities (based on their CDCS) and discuss future engagements, including mutual roles and expectations. • Consider reserving a portion of funding for Missions to allocate, building on the examples from other USAID programs. The funding of such country-specific projects could take place through the same grant process used for all awards or through a buy-in mechanism to ensure that the Missions are directly involved in the program. • Consider how country-specific projects could be best funded: through the same grant process used for all awards or through a buy-in mechanism that enables direct Mission involvement. • Use the Mission understanding of political context in-country to identify the strategic issues that will likely be faced in implementation and scale-up of GC projects, and to inform the careful tailoring of activities to the particular challenges and opportunities. For a global GC: • Identify a few countries where the problem (that the GC is aiming to solve) is severe and where it is likely that there will be some awards made. • Reach out to these selected Missions to better understand their needs and priorities (based on their CDCS) and discuss future engagements including mutual roles and expectations. • Once the awards have been made, reach out to the country Missions that will have a cluster of grantees and identify opportunities for mutual, beneficial learning between the GC and Mission around a set of jointly-developed learning questions. Activities could include grantee and Mission learning roundtables and events, and learning visits to inform case studies or engagement of Mission staff in innovator events. • Consider adopting a model constructed by the evaluation team, as a result of the comparator analysis, to formalize Mission and GC engagement. Under this model, the Missions would use the GC as a source of innovative solutions to difficult 52 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION FIGURE 4.8: DIAGRAMMATIC REPRESENTATION OF HOW A LOCAL WORKS TYPE PROCESS TO IDENTIFY INTRACTABLE LOCAL PROBLEMS MIGHT BE COMBINED WITH A GC MECHANISM TO SURFACE INNOVATIVE SOLUTIONS Local Works￾type process through US or partner Mission/ OU identifies local intractable problem 1 2 6 7 8 4 5 3 US/Partner Mission asks Central OUs if there is a known solution/proven innovation Local OUs Support adaption of proven innovations to address problem in local context Local Mission/OU/Program supports local dissemination/ scaling Locally identified problem addressed Global GC challenge designed and launched Global GC surfaces solution and supports piloting Central OUs check – are there GC potential solutions or proven innovations that could be adapted? Central GC Program Unit No Yes development problems and would liaise with a central GC program unit to discover whether there are existing solutions in the pipeline or to prompt the design of a funding call to address the problem in question. This model is set out diagrammatically in Figure 4.8. Recommendation: GCs should engage with USAID Missions to add value, recognizing that they are independent agencies which differ markedly in priorities, resources, programing, and ability to engage. (Programmatic recommendation for USAID and Partner GC Managers) • GCs could engage effectively with Missions by taking a shared value approach guided by respective CDCS and demonstrating to USAID Mission staff (who may be non-technical) how the GC could support the CDCS. • GCs should reach out to the Mission on a regular basis; establish a focal point to help navigate; maintain regular communication; and ensure that they fit with their day-to-day priorities, schedule, and capacity to engage. Recommendation: Engage Missions to strengthen ecosystem change and longer￾term MEL. (Programmatic recommendation for USAID and Partner GC Managers) • Leverage Missions’ sustained presence around the globe to support GCs, making the most of partnerships between Mission staff and stakeholders across the development spectrum – including local CSOs, the private sector, other public and private donors and investors, and (in particular) government – in order to strengthen the innovation ecosystem. • Involve Missions in advertising the GC to wider audiences, especially non-traditional actors embedded in the local ecosystem. Engage with Mission foreign service nationals who are from the local community and likely to have longer tenure in office than foreign service officers and good local connections. • While the GC focuses on the development and testing of innovations, use Missions to support adoption, replication. or scale-up of innovations in-country, e.g., by government, through strategic introductions, public support, or facilitating technical and managerial support for innovation uptake. • Engage Missions in tracking the progress of innovations once GC support ends and maintaining connections with grantees to provide evidence of longer-term impact. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 53 4.8 MEASURING IMPACT, RESULTS, AND UPTAKE MEQ 8: HOW WERE RESULTS, IMPACT, AND PROGRAM EFFECTIVENESS MEASURED ACROSS GCS? Summary: Each GC has its own MEL approach: there is not an overarching GC MEL strategy or framework or set of defined expectations, e.g., on resource levels or capacity. The GCs took a wide variety of approaches to MEL. The grantee perspective on MEL, obtained through survey responses, indicates that MEL is aligned with innovation objectives for two thirds of them, and that some find reporting a real burden, although there is good MEL support. As is typical in donor programs, none of the GCs were actively collecting data on the long-term outcome and sustainability of innovations after the grant had ended, so longer-term impact is not known. Results measurement in terms of gender equality and social inclusion was strong for three GCs but much less so for others, and there are gaps in grantee support for this. The lack of emphasis on monitoring ecosystem investments and measuring ecosystem change meant that systemic changes were not documented systematically. Recommendations: GC MEL could be strengthened by an overarching approach, principles, and expectations for MEL (covering key aspects like theory of change, MEL plans and systems, objectives and indicator setting, and GESI considerations) that all GCs should apply to their own tailored MEL approaches. At the GC level, there should be a strong alignment of MEL to objectives and innovation stages; visibility of gender and other excluded groups in targets and reporting; structured learning and use of data for adaptation; and consideration of post-project follow up and visibility of ecosystem strengthening achievements. There is no standardization in MEL approaches across the GCs (reflecting their dispersal across USAID). Each GC has its own MEL approach. Five out of the nine GCs had a theory of change and overarching MEL Plan (ACR GCD, CHIC, MAVC, SL@B and SWFF). ACR GCD had a results framework based on common and harmonized result areas that were anchored in USAID standard indicators, that was developed in its third phase. CHIC’s theory of change was fully developed as Round 3 was launched. SOGE and PAEGC had MEL plans but not a visible, well￾developed theory of change, although SOGE did have a hypothesis which went some way towards this. The evaluation could find neither a theory of change nor MEL plan for EBOLA and Zika, although a MEL plan for each individual award was developed. All GCs except for EBOLA have commissioned external evaluations or plan to commission one. Five out of nine GCs (ACR GCD, CHIC, PAEGC, SOGE and SWFF) were found to have prioritized learning (i.e., produced learning documents, and/or developed a learning plan or schedule). GCs took a wide variety of approaches to monitoring and evaluation, as set out in Table 4.5. This table demonstrates that not all GCs consistently set expectations or define outcomes, as observed in MEQ 1. TABLE 4.5: GC MONITORING, EVALUATION, AND LEARNING (MEL) OVERVIEW GC MONITORING, EVALUATION AND LEARNING STATUS ACR GCD ACR GCD developed a MEL system for its third stage which defined observable, program-specific performance indicators, drawn from USAID’s F indicators, such as the total number of trained individuals/ stakeholders, and outcome and impact indicators that needed rigorous assessment. Evaluations at the grantee level were undertaken to determine whether the grantees met defined targets. This provided a sound framework for evaluating the ACR GCD grants, although not for generating performance data that could be compared or aggregated across them. A MEL partner was recruited throughout all phases to streamline and standardize reporting and build capacity of the awardees. Two independent evaluations were conducted: one at the end of the first phase, and the other at the second phase (report not available). ACR GCD also produced some learning documents. 54 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION GC MONITORING, EVALUATION AND LEARNING STATUS CHIC CHIC has a clear MEL Plan (results framework) setting out clear, disaggregated indicators at all levels (impacts, outcomes, and outputs) with a clear ultimate emphasis on the focus population (men and women in humanitarian need due to conflict) and on the aim of the program (lives saved, lives improved). It measures both innovation progress (number of projects that establish proof of concept) and scaling (funding leveraged and partnerships formed). The theory of change was developed recently. The MEL plan prioritizes learning and sharing. CHIC have been working to simplify the reporting requirements and are recruiting an independent MEL partner for developmental evaluation. EBOLA Awardees worked with USAID to establish a series of milestones for their projects during a co-creation stage. USAID tracked those milestones and made grant payments dependent upon meeting them. While individual awardees developed MEL plans, there was no overarching MEL plan for the GC. Fund objectives did not translate into clear indicators at fund level to enable achievement of objectives to be assessed. The fund was looking for solutions that would enable longer work periods, make less use of PPE, mean fewer transitions, generate less infectious waste, and/or offer enhanced protection, to help health care workers tackle Ebola. The grants were individually measured for achievement of their award￾specific goals. MAVC Results framework and theory of change were in place. A MEL manager was hired for a process and impact evaluation. PAEGC Implicit high level theory of change stated, although the final independent evaluation noted that this was not validated and recommended having a robust theory of change taking into account all key assumptions (some of which had been identified by the evaluation). Gaps in performance indicators were also noted. Targets were set for grantees which they were expected to meet. Some learning papers were published/compiled. SL@B Outline of the theory of change and results framework were available (for FCDO component). There were individual MEL plan for each award and outcome-based milestones were tracked throughout the grant period. Independent evaluations were completed in 2015 and 2020 that recommended SL@B revisit the theory of change to clearly flesh out objectives and outcome indicators. Evaluation also recommended SL@B to improve their data capture mechanism throughout the program cycle. SOGE Hypothesis and assumptions were set out, although not developed into a theory of change. The MEL plan depended on a value proposition and key performance indicator document that listed eight key performance indicators, although the SOGE team concluded that it was not reasonable or cost effective to expect awardees to report separately on GC-wide indicators; the GC managers believed that there were not adequate financial resources to compensate awardees for the additional work needed if they were to report on indicators beyond their own results frameworks. SOGE commissioned a developmental evaluation of the SOGE uptake model. The GC prioritized learning and had a learning schedule or calendar that required awardees to share papers/publications/stories from their work. SWFF A theory of change is available with four key performance indicators and evidence of results. There was also a Program Monitoring and Evaluation Plan. Targets set for grantees, such as number of beneficiaries reached, that they were expected to meet. As observed in interview, this can be counterproductive and skew efforts, as innovators work to hit payment-related imposed milestones which can divert the focus away from the organic growth of the businesses Some recent independent evaluations available. Zika No evidence of a MEL plan for the overall GC, which had a single call, but an independent evaluation is underway. Impact measures were incorporated into GC design and selection criteria, and grant payments were milestone based against targets. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 55 Results measurement in terms of gender equality and social inclusion was strong for three GCs but much less so for others, and there are gaps in grantee support for this. Only three of the nine GCs (CHIC, SL@B and SWFF) report results clearly in terms of gender equality and social inclusion by presenting results disaggregated by gender and other disadvantaged groups. In other GCs, outcome-level results are not disaggregated, rather beneficiaries are described in general terms such as “people”, “communities” or “innovators”. In these cases, GC impact on women and other disadvantaged groups – and thus on gender equality and social inclusion, which is fundamental to meeting development challenges – is not known. 65 percent of survey respondents said that they had been asked to report on inclusiveness (e.g. age, gender), but only 28 percent of survey respondents said that they had received training on collecting and presenting results by gender, age, or disability. (The GESI insight memo at Annex 13 provides further detail.) “The reporting kept us up to date with the project progress, performance and successes/failures – which allowed us to make remedial changes quickly before things became worse. It allowed us to continually interface with our beneficiaries/ customers and learn their interests in respect to innovation design and setup, and thus enable us to make design and setup adjustments.” SWFF Grantee There are positive examples of learning from monitoring data and experience, although from our review in Table 4.5, this approach has not been prioritized systematically across GCs. There is evidence that the GCs that have adapted in response to findings from monitoring data have sharpened their strategy. For example, SL@B added validation grants and undertook more outreach to grantees over time as a result of learning at fund level. In some of the GCs (e.g., ACR GCD, EBOLA, Zika), although learnings were not formally recorded, they were exchanged informally between grantees, and also used to adapt the program over time. CHIC specifically asked grantees to report on learning from innovations that do not succeed. On the other hand, a commonly expressed concern is that a slow feedback loop between the GC managers and grantees on their reporting limits the feedback’s usefulness. Grantees indicate that their MEL arrangements are useful, but there is room for improvement. Many find reporting a costly burden. • 64 percent of respondents agreed or strongly agreed that their reporting mechanisms adequately captured their results and impact, while 14 percent disagreed. There is therefore some mismatch in the utility of MEL, and sometimes it is not well aligned with innovation objectives. • There is a reporting burden. 69 percent said that the tools for reporting were easy to use, but 19 percent disagreed. Changes to requirements are also difficult. “The reporting for USAID and other donors leveraged during the project totaled around 600,000 words, about the equivalent of ten PhD theses, in 4 years, so the total reporting load across multiple donors, each of whom have their own needs and formats, is still staggeringly huge. The most obvious improvement is that donors agree that one format of reporting is sufficient.” KII, Zika “Reporting requirements and guidelines changed over the course of the grant - it would be most useful to have these fully explained before the applications are submitted so the associated costs of gathering the data from remote beneficiaries in rural Africa can be properly budgeted for.” Survey, PAEGC While there were no guidelines for GCs on allocating resources for MEL, all provided MEL support. Grantees said that good MEL supports their innovation. 87 percent of survey respondents said that they had received support on MEL from the GC manager. By contrast, only 35 percent said they had received orientation or training at the outset on collecting and presenting evidence of results. Grantees welcomed MEL support: regular MEL partners, particularly at the local level, were felt to be helpful for building capacity (more so than inputs from occasional external evaluators) and ACR GCD awardees spoke highly of the support they had received from the MEL partner in standardizing their tools and data collection practices. In another example, an ACR GCD awardee collaborated with a local MEL partner 56 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION right from proposal stage and was able to obtain regular data and insights that helped to improve and scale up their innovation. One ACR GCD awardee, who was successful in scaling their innovation, felt that putting impact and learning at the center of implementation aided scaling up more than anything. This demonstrates the importance of good MEL systems to supporting innovation: it should be about far more than reporting upwards. None of the GCs were actively collecting data from the awardees and reporting on the long-term outcome and sustainability of the innovation after the grant had ended. Efforts were made by two GCs to continue obtaining information from the innovators, although arrangements were small scale and informal, and data collected was anecdotal and based on case studies rather than being more systematic. CHIC includes ‘post-project reporting’ requirements. SWFF set up a mechanism to continue monitoring project outcomes and continues to interact with alumni who are invited to convenings and post-award surveys. Here, evidence is anecdotal and mostly from successful innovations. Part of the challenge is the difficulty of maintaining institutional knowledge within USAID.86 Immediately after a Grand Challenge is over, the team managing it disperses, as was observed for EBOLA, and no structure remains to track the innovation or its users. (Although after the challenge ended, the officials managing the EBOLA GC have informally stayed in touch with many of the innovators, and this was also observed in ACR GCD, SL@B and Zika). Moreover, even if the GCs are ongoing, they are not staffed for longer-term tracking of innovations once grants have ended; the focus is very much on monitoring current awards. Post-project monitoring may be possible only when additional funds are given to selected innovators for acceleration support. One GC staff member commented that obligations for awardees to report for the next five years may not be feasible for participants if there are no incentives. GCs have used evaluations, both early on to inform programing and at the challenge end. Six out of nine GCs commissioned external evaluations of the whole program or its parts. CHIC, in its third year, is in the process of selecting a MEL partner to support them with developmental evaluation. ACR GCD and MAVC hired a MEL partner to implement their MEL plan. The MEL partner supported the grantees to draw some inferences on the attributable effects of the 86 An example of intentional capture of learning to avoid knowledge dissipating was the way in which design for Water and Energy for Food was directly and extensively informed by learning from PAEGC and SWFF, its predecessor programs. innovations (although evaluation design did not include any cost effectiveness assessments), and also contributed to build capacity of awardees to evaluate their innovation. PAEGC, SWFF and SL@B also commissioned final independent evaluations that offer useful insights for the future. Further, most GCs reviewed performance after each funding round and used learning to adapt approaches for subsequent rounds. ACR GCD, for example, commissioned an independent evaluation of its first phase and used findings to improve its operating structure, MEL requirements, and scaling priorities. The lack of emphasis on monitoring ecosystem investments and measuring system-level change meant that such changes were not documented methodically. Although there were notable systemic changes documented as anecdotes, there were not programmatic efforts to measure and track these over a longer period. The measurement of ecosystem change was limited to the amount of external funds leveraged by the challenge, but some GC innovators were able to achieve broader impacts on the wider system beyond the innovation ecosystem and beyond original expectations. For example, EBOLA’s mHero was used by the Ministry of Health in Liberia to send out an alert to health workers about what at the time was an unknown outbreak. The innovation was integrated into the Government’s new electronic disease and surveillance response system and subsequently used to push out information about COVID-19. In ACR GCD, a Morocco based EdTech organization was able to create a systemic change in Morocco by influencing the government to change education policy to become more inclusive and accessible for children with disability. 4.8.1 FORWARD-LOOKING QUESTIONS FLQ 8.1: HOW SHOULD GC STAKEHOLDERS DEFINE AND MEASURE RESULTS, IMPACT, AND PROGRAM EFFECTIVENESS ACROSS GCS? FLQ 8.2: WHAT FRAMEWORKS, METHODS, AND LEARNING STRATEGIES CAN BEST SUPPORT MEASUREMENT GOING FORWARD? USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 57 MEQ 1 noted the practical difficulties of stating achievements when objectives and measurement systems have not been established sufficiently well. These recommendations address the forward looking questions by setting out how strong MEL systems can be established. Recommendation: Establish and implement an overarching approach and principles for MEL in GCs, which sets out clear expectations and guidance for each (new) GC to apply and adapt to their particular dimensions (e.g., sector, geographies, timescales, innovation stages supported, scaling ambitions, target beneficiaries by type etc.). This will help to ensure that suitable MEL is in place for each GC, that results can be captured at individual GC innovator portfolio level, and (potentially) that results can also be aggregated across GCs. (Strategic recommendation for USAID Policy and USAID GC Managers) • Each GC should apply the overarching MEL framework to establish their own clear theory of change, MEL framework, and evaluation plan at the outset, thus ensuring consistency and quality of approach across GCs. • Ensure that GCs develop clear objectives, performance indicators, monitoring systems, and an approach to evaluation both for the GC as a whole and for individual innovators, from whom they can aggregate results at program level. Build on good practices identified in the evaluation findings (e.g., the CHIC results framework, ACR GCD’s key performance indicators). • Consider establishing a set of standard indicators as part of this relating to wider USAID region- or sector-specific goals, cost effectiveness, scaling, catalytic effects like funding leveraged, types of organizations funded (including women-led, local and first-time applicants), end users reached (disaggregated), ecosystem strengthening, and stakeholder engagement, against which all GCs collect comparable data from their grantees to build up an overall picture of GC impact. This is currently absent, as noted in the evaluation limitations in Section 3.3. Recommendation: Give early consideration to evaluation plans, defining evaluation needs and the methodology best suited to meeting them (Programmatic recommendation for USAID and Partner GC Managers) Budget and resource evaluation appropriately and plan procurement timescales, which can be lengthy. 87 More information can be found at https://www.betterevaluation.org/en/approaches. Options include experimental evaluation, cost effectiveness evaluation, post-implementation external evaluation, and other evaluation methodologies,87 all of which must be driven by the key evaluation questions and GC nature and scope. For example: • Early-stage innovation success could be evaluated in terms of its ability to prove concepts in a smaller lab-type setting and its cost effectiveness. • Later-stage innovation success could be evaluated in terms of whether it is readily acceptable and scalable to the broader public, inclusiveness of design, and viability. • An impact evaluation using randomized control trials could be useful for a GC that is implemented over a fairly long period in a targeted region or area with a fixed set of objectives, with scope for wider impacts. • Retrospective studies could be commissioned to look at attribution/contribution in relation to system changes, where an innovation has sought to influence this. • An evaluation or review at the end of each funding round for a cohort of innovations is useful to draw out learning (e.g., about barriers and enablers), identify good practice (what has worked well and why), and inform future improvements (e.g., in TA support for scaling and acceleration). Recommendation: Ensure that GC objectives and MEL frameworks make gender and other characteristics of exclusion (e.g., age, disability) visible so that the impact of GCs on equality and inclusion (which is strongly linked to impact on poverty and other developmental outcomes) can be seen and understood. (Programmatic recommendation for USAID and partner GC managers) • Express expected outcomes in disaggregated terms (e.g. men, women, girls, boys, elderly, or disabled men and women) rather than general terms (e.g. citizens, communities, innovators); define the terms used (e.g. ‘marginalized’) and ensure that data is collected and reported accordingly. • Gauge grantee awareness and capacity on GESI and why it matters for the GC as a whole and for innovations. Provide training on how to build GESI into projects and tools to support innovators to do this and to measure results. Recommendation: GCs and projects should undertake structured learning and use data and learning for adaptation and improvement. 58 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION (Programmatic recommendation for USAID and Partner GC managers) • Develop a culture of review, reflection, and learning at all levels (by and between GCs and innovators) and introduce systems and processes to support this, including directing MEL towards learning (i.e., encouraging awardees to document what they learned instead of what they achieved) and forums for sharing learning. • Establish learning objectives. Document learning from ‘failing’ awards as well as successes. This could help to turn things around, transforming the solutions to achieve other priorities, or help other innovators not to repeat mistakes. • Ensure that feedback loops are prompt to avoid evaluations being irrelevant to organizations whose priorities might have been moved on by the time an intervention has been implemented and evaluated. FLQ 8.3: HOW SHOULD SUCCESS BE MEASURED ACROSS STAGES OF INNOVATION, PARTICULARLY EARLY-STAGE INNOVATIONS AND R&D INVESTMENTS? Recommendation: Ensure that MEL is aligned to objectives at each innovation stage. (Programmatic recommendation for USAID and Partner GC Managers) • Define success for early-stage innovation and R&D investment and measure against this. Success is not necessarily that the innovation is successful – it is expected that many will not be. Rather, success is that proof of concept testing has been completed against defined criteria and generated good evidence which can be acted upon. Criteria should include: – The user perspective (e.g., on the innovation’s desirability and usability, for different user groups including women). – Commercial dimensions (market assessment, commercial viability). – Technical robustness (feasibility). • For example, an objective of scaling could be evidenced by indicators about the scaling pathway: testing, technology dissemination, stakeholder engagement, marketing strategy, and investment secured. These can be quantified and measured over the project cycle.88 88 As used in health GCs and in SWFF (which developed the Solver Scale score methodology). 89 Individual project MEL budgets should be 5-10 percent of grant value, but GC MEL budgets will vary depending on the type of MEL required, e.g., whether it includes embedded, ongoing MEL from an external provider, or light-touch, in-house capacity. FLQ 8.4: PRACTICALLY AND OPERATIONALLY, HOW CAN GC PARTNERS (AND USAID) BETTER STRUCTURE AND COORDINATE DATA COLLECTION AND MEASUREMENT EFFORTS GOING FORWARD? WHAT SYSTEMS AND PROCESSES WILL BEST SUPPORT SUCH EFFORTS? Recommendation: GC managers (USAID and other donors) should specify their data requirements and ensure that GCs have a dedicated MEL team and an efficient MEL mechanism focused on collection of relevant and useful data against defined indicators and targets. (Strategic recommendation for USAID GC Managers) • Align innovator and program reporting with the monitoring and reporting requirements of the donor agencies (USAID and others), • Work with donor partners to make reporting requirements as consistent and streamlined as possible. • Dedicate a sufficient percentage of GC budget to MEL at both fund level and for individual grants.89 Build resources for MEL into both GC management budget and grantee budgets. Recommendation: Ensure the utility of MEL and that it is streamlined to minimize the burden on grantees while maintaining clear and regular reporting requirements. (Programmatic recommendation for Partner GC Managers) • Ensure that data collection and reporting is proportionate and useful; keep reporting light touch and focused; minimize indicators and ensure utility; measure what you want to see change; use feedback mechanisms so that reporting is not one way. • Ensure common standards, including responsible data approaches, and quality of data collection across grantees. Aggregate grantee data to build GC results. • GCs could consider co-creating or consulting on MEL systems (indicators, reporting frequencies, and challenges) with the grantees. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 59 FLQ 8.5: HOW CAN GCS MEASURE LONG￾TERM OUTCOMES AND INCENTIVIZE POST￾AWARD REPORTING? FLQ 8.6: HOW CAN THE EFFECTIVENESS OF ECOSYSTEM INVESTMENTS BE MEASURED? Recommendation: Design for and incentivize longer-term reporting to provide valuable post-grant evidence of innovation success. Tie resources to post-award reporting. (Strategic recommendation for USAID GC Managers) • Plan to track innovation status and scale over a period of several years, beyond GC funding, and resource this. Innovators could be incentivized to provide updates through USAID-supported visibility, marketing and even acceleration support. • Ask grantees what would incentivize them to engage. Seek ideas from other programs who have done this successfully. Recommendation: Ensure that ecosystem strengthening goals are incorporated within GC theories of change and the overarching MEL framework to make it visible and incentivize innovators to focus on it. (Programmatic recommendation for USAID and Partner GC Managers) • Change pathways and indicators could include stakeholder engagement, collaboration, research, and convening activities, and evaluations of ecosystem investments could encompass assessments of policy advocacy using tools like network mapping and bellwether interviews. • Include ecosystem strengthening proposals as a selection criterion for awards to ensure it is present. 4.9 MEASURING COST-EFFECTIVENESS MEQ 9: HOW FEASIBLE IS IT TO MEASURE THE COST EFFECTIVENESS OF PREVIOUS AND FUTURE GCS AND TO COMPARE THE COST EFFECTIVENESS OF GCS TO TRADITIONAL PROGRAM MODELS? Summary: There is a considerable interest by all concerned with GCs that CEA is useful for GCs, not only to evaluate the performance of the portfolio but also to assist in deciding which innovations should receive acceleration support toward scale. There has been no clear mandate, standards, or availability of simple and user-friendly tools and techniques for supporting GCs to undertake CEA. As a result, there were wide differences among GCs in terms of their understanding and approach to CEA. Only SL@B and SWFF conducted extensive CEA analysis. Recommendations: A common CEA approach should be applied across all GCs. Five key recommendations are proposed in the forward-looking CEA framework (Annex 11) produced for this evaluation. The measurement of cost effectiveness of the GC is not feasible if there has been no systematic collection of cost data relating to the activity costs of each innovation or the costs of the GC to support the innovations. With the exception of SWFF, which completed a social rate of return analysis on the portfolio of interventions, and SL@B, which completed some in-depth case studies, other GCs have only undertaken some limited analysis. Annex 10 provides a summary of how each GC has addressed cost effectiveness to the extent that it was undertaken. It was agreed with USAID that it would be useful for the evaluation team to develop a forward-looking cost effectiveness framework which could be applied to all future GCs and to new innovations for existing GCs. It is only when such CEA is undertaken systematically that it will be possible to compare the cost effectiveness of GCs with more traditional program models. “Let’s agree some broad principles which will be unifying across the grand challenges. Don’t let perfect be the enemy of the good, let’s start with some simple cost effectiveness models and if we do this it will be a huge win for the industry as the bar is low.” USAID KII Cost Effectiveness Workshop 22 December 2020 60 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 4.9.1 SUB-EVALUATION QUESTIONS SEQ 9.1: HOW AND TO WHAT EXTENT HAS THE COST EFFECTIVENESS OF GCS BEEN MEASURED? The evaluation found that there was considerable interest by USAID GC managers, partner GC managers, and grantees in cost effectiveness but, as highlighted above, CEA has not been conducted across the portfolio of innovations for each GC. Furthermore, there is very limited information on the cost efficiency of managing the GC in terms of the cost of identifying, selecting, and supporting the innovations. 4.9.2 FORWARD-LOOKING QUESTIONS FLQ 9.1: METHODOLOGICALLY AND TECHNICALLY, HOW CAN WE MEASURE THE COST EFFECTIVENESS OF GCS GOING FORWARD? The CEA framework at Annex 11 highlights the key activities that would be undertaken at each stage of the GC cycle, and the varying degree of emphasis given to measuring the cost efficiency of the management of GCs for early-stage innovations compared to the cost effectiveness of the portfolio of outcomes for late-stage innovations. The first step is for USAID to build on the considerable interest expressed by the USAID Exploratory Programs and Innovation Competitions team and GC Managers during the evaluation and develop a cross-sectoral working group to develop a set of standard indicators of outcomes and activity cost headings for each key sector. This would be following the lead that has already been taken by USAID in the education sector. USAID then needs to adopt the principles laid out in the CEA framework and ensure that CEA is undertaken for all future GCs, with particular emphasis given to designing and applying a CEA methodology during the inception phase. The CEA framework sets out the technical analysis that would be needed to be undertaken during the four stages of the GC cycle: (i) design; (ii) inception; (iii) implementation; and (iv) ex post. The purpose of CEA is not to be able to compare the economic value of, for example, health versus education GCs, but to start a process of improving an understanding of the cost effectiveness of the GC instrument compared to other types of intervention. USAID would then be learning from the CEA results generated from each GC and begin to benchmark costs for a range of activities undertaken by the GCs. USAID should build a cadre or network of expertise to validate the CEA undertaken by the GCs and provide advice to new GCs. In this way, a community of practice would emerge from the GCs and USAID that would support decision making on improving the allocation of resources to GCs by sector and stage of innovation, as well as improving the effectiveness of GC implementation. Recommendation: Move forward on measuring cost effectiveness across the range of different GCs by sector and stage of innovation by implementing the following five steps: (Strategic recommendation for USAID Policy) 1 Agree in principle that CEA becomes an integral part of undertaking a GC and that an initial CEA becomes part of the design of all future GCs before implementation. This is then followed up with CEA activities at the inception, implementation, and ex post stages of the GC as set out in the proposed CEA Framework. 2 Establish a cross-sectoral working group to expand the current standard indicators of outcomes and standard activity costs for each of the key sectors covered by the GCs, following the lead taken in the education sector. 3 Ensure that CEA is undertaken at all stages of the GC cycle with particular attention to agreeing CEA priorities in the inception phase of the GC. 4 Develop a database of benchmark unit costs at activity and outcome level for each sector and for GC Management costs. 5 Develop its own cadre or network of expertise to validate CEA analysis from the GCs and consolidate learning on best practice. FLQ 9.2: WHAT PRACTICAL FRAMEWORKS/ APPROACHES CAN BE DEVELOPED TO MEASURE GCS? • Please refer to CEA Review (Annex 10) and CEA Framework (Annex 11). USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 61 4.10 GOVERNANCE, PARTNERSHIP MODELS, AND OPERATIONAL EFFECTIVENESS MEQ 10: HOW HAVE GCS BEEN MANAGED/GOVERNED DIFFERENTLY AND WHAT MODELS CAN BE DESCRIBED? WHAT ARE THE ADVANTAGES AND DISADVANTAGES OF DIFFERENT MODELS? Summary: Within the overarching GC model there are key differences in management and governance, principally relating to the different models for donor involvement (single or multi-donor), fund management (in-house, external or hybrid), and overarching governance. Each has advantages and disadvantages, which are examined. There is no single “best” model for these complex instruments, but healthy partnerships between all actors promote success. Partners’ comparative strengths can be deployed to strengthen all stages of the GC fund management cycle, from design through grant management to MEL. The key to success is to deploy them constructively and in partnership. In-house grant management can reduce transaction costs and bring donors closer to their projects, but overall reach, networks, and depth of expertise is more likely found in an external provider. Either model requires sufficient resourcing. Recommendations: To ensure effectiveness of a GC governance model and decision-making processes, design them in detail at the outset and ensure that they are adequately resourced. Build a vision and objectives for the GC which is shared by all parties involved in governing and managing it. Define, document, and communicate partners’ roles and responsibilities, reflecting their strengths. Establish and maintain trust between all parties throughout. 90 Only IIA uses a co-funding model similar to the GCs. 91 SWFF had two pools of funds, one managed by USAID and one by Grand Challenges Canada. SOGE had a decentralized model whereby each partner was responsible for leading and implementing their own program, but was able to contribute to the design, funding and rollout of interventions led by others. 92 There is also evidence that SWFF’s achievements would have been significantly lower if it had not been able to circumvent some USAID processes, by virtue of being a multi-donor fund. Different models are described below, along with evidence for their advantages and disadvantages. Single or Multi-Donor Funding Model One principal difference is whether the GC has a single-or multi-donor funding model. This influences how the GC is managed and the complexity of the governance structures required. Two GCs do not have donor partners from outside the U.S. Government – Zika and EBOLA – while the other seven are multi-donor funded and managed. The multi-donor models all involve at least three bilateral donors and foundations. (Multi-donor funding only applied in one of the four comparator programs90 – GCs perhaps lend themselves better to the multi-donor model due to their focus on global challenges). For all multi-donor GCs except SWFF and SOGE,91 donors contributed to a single pooled fund which they oversaw collectively. There are several advantages to a multi￾donor model: • A multi-donor funding arrangement provides a larger funding pool overall plus potential crowding in of further funding through the addition of new partners (e.g., SOGE, SL@B). It can also better absorb the impact on the fund as a whole if one donor withdraws. Having several donors also provides more exposure and connections for innovators and thus greater possibility of further funding. • Multi-donor engagement brings access to wider networks and a larger pool of potential applicants, if partners publicize GCs through their own communication platforms. • Institutional funding brings significant credibility to innovators (see Section 4.5). By extension, a multi￾donor fund brings further cachet and prestige. • Multi-donor funding enables the fund to leverage the comparative strengths and interests of different partners to enhance overall performance of the GC (explored in Section 4.10.1). • It enables risk sharing of higher risk investments (e.g., SL@B) and enables the fund to overcome restrictions that might limit what a fund could otherwise support (e.g., CHIC and SL@B can fund projects which USAID itself could not).92 • The multi-donor model can therefore be a strong one. 62 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION TABLE 4.6: GC DONOR PARTNERS AND FUNDING NAME DONOR PARTNERS TOTAL FUNDING (USD, MILLIONS) PERCENT OF FUNDING FROM USAID All Children Reading: A Grand Challenge for Development USAID, World Vision, DFAT 29.0 48 percent Combating Zika and Future Threats USAID 30.0 100 percent Creating Hope in Conflict USAID, FCDO, Dutch Ministry of Foreign Affairs 39.9 31 percent Fighting Ebola USAID plus CDC, Department of Defense, The White House 8.9 100 percent Making All Voices Count USAID, FCDO, Omidyar Network, Sida 45.0 22 percent Powering Agriculture USAID, Sida, Duke Energy, GIZ 51.8 30 percent Saving Lives at Birth93 USAID, FCDO, Norad, KOICA, BMGF, GCC 100.0 - Scaling Off-Grid Energy USAID, Power Africa, FCDO, Shell Foundation, AfDB 124.5 9 percent Securing Water for Food USAID, South African Department of Science and Technology, Dutch Ministry of Foreign Affairs 34.3 35 percent 93 The USAID/partner funding breakdown for SL@B is not available. 94 Technical assistance is managed and procured by fund managers but delivered by external providers. The multi-donor model also has some disadvantages, principally arising from conflicting objectives and transaction costs of managing them. Multi-donor models can bring together donors who have divergent and even conflicting objectives for the fund, which can also change as donor priorities shift, requiring revisions to the program approach. Where these differences cannot be overcome through compromise and are not underpinned by strong and trusting relationships, there can be real tensions and difficulties for the implementing organization that hamper its effectiveness. There are several examples: • In PAEGC, there was a tension between surfacing new technology and achieving impact for poor people, which reduced its developmental effectiveness. • MAVC struggled to attract sufficient numbers of suitable applicants. The three donor governments disagreed over whether to cast the net wide in the search for projects to fund, or to adopt more targeted approaches. This delayed and hampered an effective response to the problem. • For SOGE, the level of effort required by the fund manager to co-ordinate partners and develop a common approach was underestimated and detracted from its resourcing for grant management. From KIIs, SOGE’s partnership was undermined by the dominance of USAID resulting from their funding of the secretariat, which was not therefore seen as independent by the other partners. Interviewees also felt the partnership was undermined by the absence of a pooled funding mechanism, which would have helped to create greater donor ownership of the fund as a whole. • Challenges can also arise from different funding priorities, for example within PAEGC where the GIZ donor team ‘stood apart and made their own decisions on who they would support with their own funds’ (KII), effectively becoming a fund within a fund. • PAEGC also struggled to meet different donor reporting requirements, for example in relation to financial accounting. In-house, External, or Hybrid Models of GC Management A key feature of any GC model is the relationship between its donors and the fund management team responsible for all aspects of seeking, funding and managing a network of innovators.94 In six of the GCs, USAID itself managed the fund, either in its entirety (SWFF and USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 63 Zika) or one part of it (EBOLA, PAEGC, SL@B, and SOGE). In three, the GC was managed in its entirety by external managing agents, either engaged through a competitive procurement process (ACR GCD and MAVC), or as a natural outcome of proposals to establish the fund (CHIC). A fund might have a single fund manager (e.g., Grand Challenges Canada for CHIC and World Vision for ACR GCD), or several, each responsible for different aspects of the fund (e.g., SOGE, where USAID’s in-house secretariat coordinated several managing parties, and MAVC). Whichever model applies, clearly having capacity and resourcing to manage a complex fund working with multiple implementing partners is critical and where this is in doubt, funding is put at risk (MAVC, for instance, was closed early largely due to donor concerns about fund performance and fund management capacity). Please see Section 4.10.1 for a discussion of the advantages and disadvantages of these models. Governance Processes in GC Management GCs typically have a steering committee for high level governance that meets no more than monthly and often nearer six-monthly, bringing all donors together in a decision￾making forum to direct the fund. The degree of involvement which donors have in day-to-day operations, and even in the selection process, is much more limited, although some donor staff serve on technical evaluation committees (e.g., PAEGC). In SWFF, donors are very engaged in the selection process and regular, monthly reviews of innovator performance. Whatever the exact boundary of delegated decision-making, it needs to be clear and acceptable to all involved what those boundaries are. Funds which purposefully include selected subject matter experts and people from affected populations or target beneficiaries in their design and decision￾making processes (e.g., CHIC) provide assurance about the quality of the process to donor officials at Steering Committee level, who might not have technical expertise or on-the-ground experience. KIIs did not surface advantages and disadvantages of particular models. 4.10.1 SUB-EVALUATION QUESTIONS SEQ 10.1: HOW HAVE DIFFERENT GC GOVERNANCE STRUCTURES, PARTNERSHIP AND OPERATIONAL MODELS INFLUENCED THE ACHIEVEMENT OF OBJECTIVES AND RESULTS? Models involving multiple parties (be they donors or in the fund management team or both) involve trade-offs. As explored above, such models introduce complexity and potential for disagreement, can be prone to delays as a result and demand time and resources to manage. This can impact adversely on achievement of objectives. On the other hand, involvement of several parties can leverage investment, provide complementary areas of expertise, provide solutions to specific donor-related procurement issues, and broaden the scope and reach of the fund. Whatever the model, clarity about objectives, clear lines of accountability, strong relationships, and a sound process for decision making and resolving disagreements are fundamental to the smooth operation of a GC and, by extension, to the achievements of the fund. It often takes time for these positive working practices to evolve (e.g., ACR GCD formalized governance and strategic documents after their third funding round) and an investment of time and effort is needed, but where they remain absent there can be real difficulties (as in MAVC). A ‘hands on’ approach to fund management, and a culture which enables opportunities to be seized and partnerships to be brokered, can lead to results. This is illustrated in EBOLA by USAID convening a workshop to improve Health Information System interoperability in West Africa, involving private partners to field test innovations and stepping in to form partnerships to take successful innovations wider. SEQ 10.2: HOW HAVE DIFFERENT TYPES OF GC PARTNERS USED THEIR COMPARATIVE STRENGTHS TO MAXIMIZE OPERATIONAL EFFECTIVENESS AND DEVELOPMENT IMPACT OF GCS? Multiple donor partners can bring complementary areas of expertise or focus that together enhance the overall performance of the GC. For example, the engagement of Sida as a donor partner to USAID on the PAEGC and SWFF led to an increased focus on gender equality and reaching the poor, with Sida leading on the provision of additional gender expert support and the development of gender guidelines. An infectious disease OU in USAID brought experience on broader infectious disease control priorities to Zika. FCDO’s MEL requirements, which are more rigorous than USAID’s, appear to have contributed to CHIC’s strong MEL processes. As noted earlier, having multiple donors can also 64 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION address gaps of difficulties otherwise caused by individual donor restrictions relating to policy, geography, or compliance (e.g., CHIC, SL@B). The existence of comparative strengths alone is not sufficient: they must be deployed effectively and collaboratively. MAVC suffered from donors with different comparative strengths: rather than being complementary, they became ineffective or even conflictual. For example, the Institute for Development Studies (IDS), a partner in the fund management consortium, deployed its expertise in research to lead grant-making for research, evidence, and learning, but did not effectively feed valuable research outputs into the other aspects of the fund so that learning could be acted upon by the fund manager and other grantees. The evidence suggests that involvement of technical experts at key points in the fund management cycle also enhances GC effectiveness. The relevance, appropriateness, and priority of both CHIC funding rounds and selected projects appear greater due to GCC’s involvement of subject matter and humanitarian experts as well as voices from affected communities. As we have seen, involving technical experts in TA provision for innovators is also significant. SEQ 10.3: WHAT ARE THE ADVANTAGES AND DISADVANTAGES OF IN￾HOUSE VERSUS EXTERNAL GRANTS MANAGEMENT? In-house management means that the donor has the advantage of being more closely connected to projects, can develop a good understanding of projects and the overall portfolio, and is better able to feed learning into wider programing. From the evaluation team’s experience, in-house management may be more appropriate for grant management support – as opposed to acceleration and scaling support – particularly for single donor situations, as in-house teams can be better placed to guide adherence to terms and conditions and compliance. Grantees may also prefer to communicate directly with donors rather than via a third-party managing agent. 95 It is notable that SWFF fund management costs were 26.2 percent of the total budget and 40.9 percent of fund grant value (SWFF Final Performance Evaluation April 2020). This is high, compared to other funds the evaluation team is familiar with, but demonstrates the level of resourcing that can be required for a fund that chooses to really invest in high quality launch, robust vetting processes, strong relationships between management and awardees, and robust monitoring and reporting systems, as well as a wide range of technical assistance. This model was also found to be very resource intensive for the donor to engage with. Management costs are included in the cost effectiveness framework at Annex 11, which would provide a means of tracking and comparing them across GCs. 96 DFID Project Completion Report for SL@B, 2020. In-house management can also be more efficient overall, by reducing the transaction costs of managing donors, but must be sufficiently resourced with a team of sufficient size, quality,95 and flexibility of resourcing required to manage demand peaks, for example in selection processes and annual progress reporting. In-house management also avoids the real problems which can arise from imposing an inadequate inception period on an external provider, as was the case in MAVC. External management can more easily provide teams with in-depth expertise in both fund management and the subject matter concerned. For example, GCC runs many funds, shares systems and processes across them, and has an expert workforce, making staff turnover easier to address. CHIC is headed by a humanitarian professional recruited into the role who brings expertise that might not be available in-house. The relatively unusual hybrid model employed by SL@B, whereby a multi-donor fund was managed operationally by two donor agencies (USAID and GCC) offered some advantages: it “fostered economies of scale and likely enabled donors to achieve more collectively than would have been possible alone,” noting that it “allowed for wide reach in sourcing innovations, with the program able to leverage each partner’s networks.”96 Models involving several external organizations in a single fund management team do carry an inherent risk of inefficiency and a failure to share learning across grants. This is particularly likely where there are differences in organizational cultures involved, where they are geographically distant, or do not have a history of working together. These were amongst the reasons for the real difficulties encountered in MAVC, where fund management was split between an INGO in the Netherlands, a university in the UK and a social enterprise in Kenya, each with different design and grant management processes, and very weak coordination between them. Coordination is essential but is not always easy. It also has a cost: in SOGE the coordinating secretariat for the dispersed model was undermined by insufficient funding and was also not perceived as neutral by the other partners, leading to communication difficulties. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 65 Coordination difficulties can arise even where there is some in-house fund management, where it is split between different departments (e.g., SOGE).97 4.10.2 FORWARD-LOOKING QUESTIONS FLQ 10.1: WHAT GOVERNANCE STRUCTURES/MODELS AND DECISION￾MAKING PROCESSES ARE MOST EFFECTIVE FOR GC PROGRAMING GOING FORWARD? FLQ 10.2: HOW CAN PARTNERSHIPS BE FORMED AND MANAGED TO MAXIMIZE COMPARATIVE ADVANTAGES GOING FORWARD? The analysis shows that there is no single model for success. Rather, effectiveness is determined by how the model is designed and implemented. The multi￾donor model is a strong one where there is sufficient alignment of vision, priorities and expectations; strong relationships between the individuals involved; and a sound process and willingness to resolve differences when they inevitably arise (as is the case in CHIC, which has three Government donors). Where these factors are absent then difficulties arise, as demonstrated earlier in this section. From this analysis, we make several recommendations for how to govern and manage GCs well. Recommendation: Build a vision and objectives for the GC that is shared by all parties involved in governing and managing it to ensure that the GC is not undermined by conflicting priorities and disagreements. Articulate the vision in clear objectives for the GC, and a theory of change setting out how it is expected to be achieved. Involve external experts in the process. (Strategic recommendation for USAID GC Managers) 97 As a further point, the evaluation team’s first-hand experience of managing funds is that an external grant manager is better able than an in-house team to provide flexible levels of inputs to manage peaks and troughs in the fund management cycle (e.g., around launching new calls, reporting cycles etc.). The governance and management structure should be informed by a well-articulated theory of change or intervention model, taking into account: • Clearly defined objectives and level of ambition. • Stages of innovation to be supported. • The nature, geographic distribution, and capacities of potential solvers. • The intended extent of ecosystem engagement. • The required level of engagement of country-level Missions and other OUs. • The required level of technological expertise. Recommendation: To ensure effectiveness of a GC governance model and decision￾making processes, design and document them in detail at the outset and ensure that they are adequately resourced. (Programmatic recommendation for USAID GC Managers) • Allow sufficient time to ensure alignment of objectives from founding partners. • Address and resolve any potential areas of conflict. • Determine cost requirements. Recommendation: Define, document, and communicate partners’ roles and responsibilities, reflecting their strengths. Establish and maintain trust between all parties throughout. Identify partners’ comparative strengths at the outset. Ensure that it is clear to everyone involved in managing the GC and to the innovators how parties relate to each other and contribute to the whole. Invest in building relationships. (Programmatic recommendation for USAID and Partner GC managers) 66 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Photo: Global Devlab/USAID USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 67 5 CONCLUSIONS AND RECOMMENDATIONS 5.1 CONCLUSIONS USAID’s Grand Challenge portfolio, a total investment of $463.4M since 2011, has made 767 grants across nine challenges in the agriculture, energy, education, health, humanitarian, and governance sectors. Innovations have been supported in 83 countries in every region of the globe, and achievements are considerable: lives have been saved and improved in conflict settings; communities are using clean energy systems and millions of people have off-grid energy connections; farmers have produced more food, increased their incomes, and reduced water consumption; mothers’ and babies’ lives have been saved; and education has become more inclusive. This evaluation has assessed the effectiveness of the Grand Challenge model in terms of both “what” GCs have achieved and “how” they have done it. Conclusions are presented against the two overarching evaluation questions that encompass these themes. OEQ 1: TO WHAT EXTENT HAS THE IMPLEMENTATION OF THE GRAND CHALLENGE APPROACH LED TO THE EFFECTIVE DELIVERY OF RESULTS AND ACHIEVEMENT OF DEVELOPMENT AND HUMANITARIAN OUTCOMES? Most (five) of the nine GCs have met or exceeded targets. CHIC substantially exceeded its targets for end user access to improved products or services and humanitarian intermediary use of the same. Zika was a high-risk innovation platform that achieved what it set out to do, surfacing a number of leading technologies and improving public health. PAEGC has contributed to the development of community scale technologies and achieved outcomes for clean energy use. SWFF was highly successful, meeting or exceeding all its targets for food production, water consumption, and farmer income. SL@B successfully achieved its objectives of seeding and accelerating innovations in maternal and neonatal health, although outcomes were largely attributable to the four innovations that reached scale. Three GCs could not be rated due to achieving short-run results with no longer term objectives set (EBOLA), not having evidence of overall achievement of objectives despite some significant results (ACR GCD), or not yet reaching targets (SOGE). MAVC disappointed and, although it did achieve some outcomes, was closed early. Some GCs articulated clear objectives and targets, but these were not consistently well defined and in place across the GC portfolio, meaning that detailed results assessment across GCs was not possible. Further, only just under two thirds of grantees responding to the survey agreed that their reporting mechanisms adequately captured their results and impact. There is sufficient evidence to believe that the results of certainly the five strong performing GCs are sustainable, and this is partly confirmed from the survey as 64 percent of the innovations considered that the GC had provided support to ensure that the service or innovation would continue and develop beyond the lifetime of the grant. Some innovations – those that have achieved large-scale results and scaling – seem very likely to establish and progress beyond GC support, from the evidence seen, but GCs have not explicitly assessed the expected overall sustainability of the innovations and their outcomes. OEQ 2: TO WHAT EXTENT HAS THE GRAND CHALLENGE APPROACH BEEN AN APPROPRIATE METHOD FOR ACHIEVING SUSTAINABLE RESULTS AT SCALE AND STRENGTHENING INNOVATION ECOSYSTEMS? USAID’s GCs overall have achieved positive results in varied sectors, many of which are likely to be sustainable, and have supported the scaling of some significant innovations. The GC model is an appropriate method for its results ambitions and, to a lesser extent, scaling and strengthening ecosystems. Key to the model’s effectiveness in practice is context-informed design and implementation and strong, supportive partnerships. 68 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION The model’s other distinctive and supporting features98 are a focus on innovation in products and services (including but not exclusively technological); collaborative approaches facilitating engagement of new actors; supporting entrepreneurship by creating an enabling environment; exploring partnerships and leveraging funds and resources of other actors (including the private sector); and competitive open calls, all of which support results, scaling, and strengthening innovation ecosystems. Looking at the GC approach firstly in relation to achievement of results, it is clear that the model can be effective. Substantial funds have been leveraged and a wide range of actors have been brought in to tackle development challenges, many non-traditional. The GCs have supported innovative solutions with both funds and technical assistance. Given that GCs are an effective model, their achievement of results is dependent not on the model itself but upon how well the model is designed and implemented, particularly in relation to understanding of context, objective setting, learning and adaptation, supporting pathways to scale, and coordinated engagement by donors. Where GC performance fell short, it was largely due to weaknesses in these areas. Outcomes and performance of GCs, however, are not consistently visible. To ensure they are, objectives and results must be aligned, defined and differentiated for women and other disadvantaged groups, and progress should be robustly measured at grant level and aggregated at GC level. With the use of standard indicators, some GC level results could even be aggregated across GCs. A further means of demonstrating the strength of the GC model would be through CEA of results achieved across the whole GC portfolio, but the meta￾evaluation deemed this not feasible due to the absence of consistent cost data which precluded creating a framework for analysis. This means that at present, USAID cannot demonstrate the cost-effectiveness of the GC model compared to other instruments. GCs would therefore benefit from an overarching strategic approach both to monitoring, evaluation, and learning, and to cost effectiveness measurement. Critically, this strategic approach needs to be included in the design of the GC and given major emphasis at the start of implementation. A proposed CEA framework for GCs is provided in Annex 11. Secondly, the GC model has been appropriate for sustainable results at scale only to some extent. All 98 See Section 1.1 on GC Context GCs had scaling as an objective, even if expressed in general terms for many. The GC approach is sound: providing technical assistance and scaling support, setting scaling targets in some cases, and catalyzing funding (where there has been some success). While there are some stand-out examples of successfully scaled GC innovations, including those showcased in the case studies, there are few overall. There are three main reasons for this: • It takes time and funding to take an innovation to scale, and these are limited. • Effective technical support, which is dependent on expertise and understanding of context, is essential but resource intensive and logistically challenging, particularly when spanning multiple geographies and even innovation types. • In practice, GCs focus more on progressing early- to mid-scale innovations along the innovation pathway. GCs are better positioned to support innovation progress to get ready to scale rather than to achieve scale and should be used to funnel innovations through to this point for other instruments (such as public sector bodies, particularly for education and health, and other donor programs or investments) to take forward. Finally, the GC model does support innovators to engage effectively with the innovation ecosystem but has been more limited in attempts to strengthen that ecosystem. All GCs have engaged with actors in the wider ecosystem, and there are numerous instances of this being done successfully, leading to increases in reach and impact for GC-funded innovations (e.g., through investment or partnerships for innovation development and uptake), despite no evidence of a formalized approach. The PAEGC hub provides a good example of how a GC has taken steps to strengthen the ecosystem itself through a targeted investment. This, however, is atypical, resource￾intensive, and highly context and location specific, so it is not suited to GCs with a wide global presence. Engagement with governments to shift policy and practice is another way in which GCs can strengthen the ecosystem. This can improve the enabling environment (e.g., by removing regulatory barriers) and increase innovation uptake (or openness to uptake) by a significant player. Innovations with direct relevance to public services, such as education and health, have had most success. GC engagement with USAID Missions on ecosystem strengthening occurs rarely but could be fruitful, as it has been for comparator programs. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 69 5.2 STRATEGIC RECOMMENDATIONS Strategic recommendations for strengthening USAID’s Grand Challenges are set out below. These are directed to USAID GC policy makers overseeing the full GC portfolio (who are situated in the EPIC team) and to USAID GC Managers working with donor funding partners as shown. Headline recommendations are listed here, cross referenced to the supporting detail set out under the forward-looking questions in Section 4 of the report. TABLE 5.1: USAID META-EVALUATION: STRATEGIC RECOMMENDATIONS USAID POLICY USAID GC MANAGER S1 GCs should undergo complete and rigorous design ahead of launch, involving effective USAID engagement with donors, and partner alignment around the GC objectives, taking into account the success factors identified by the evaluation. Results Section 4.2.1 Why? The GCs that completed a more thorough ex ante design and analysis of the context, defined objectives, and had effective USAID engagement were the best performing GCs. How? The approach will vary according to the nature of the sector, with quick action and solutions needed for humanitarian responses and Ebola outbreaks and deeper analysis of market systems and the political economy for agriculture, energy, water, and governance interventions (or even protracted humanitarian crises). For this latter group of GCs, the culture of learning and adapting the acceleration support to the context should continue during implementation of the GC. S2 Ensure that the partner GC manager has the necessary skill and experience to understand and build local partnerships and provide appropriate support to acceleration through providers who know the context. Results Section 4.2.1 Why? GCs understanding the local conditions and having the appropriate local skills was a key success factor in enabling innovations to progress toward scale. How? GCs should build relationships with local acceleration services and partnerships with the ecosystem and investors to enable the innovations to reach scale. S3 Focus GCs on getting innovations ready to scale rather than on implementing them at scale. Results Section 4.2.1 Why? The GC instrument is best suited to taking innovations from early- and mid-stage toward scale and developing partnerships for longer term implementation at scale with government and private investors. How? The GC should manage its innovations as a portfolio providing tailored support only to those innovations that have the potential to reach scale. S4 Strengthen developmental outcomes overall and for women and other disadvantaged groups specifically by embedding gender equality and social inclusion principles into GC design and objective setting. Results Section 4.2.1 Why? Supported innovations are much more likely to reach and meet the needs of women and disadvantaged groups if the particular challenges they face within the GC focus (e.g., livelihoods) and barriers to accessing innovations, particularly technology, are understood. This was done by some GCs but not all. Tackling these issues will contribute to meeting USAID’s SDG commitments. How? Undertake contextual analysis to understand the barriers and challenges faced by women and other excluded groups, including issues of poverty, rights, and access to services. Apply analysis to all aspects of GC design including objective setting. S5 Ensure that GCs have an acceleration and scaling strategy in place from the outset and focus them on supporting early- to mid-stage innovations. Scaling & acceleration Section 4.3.2 Why? There are multiple pathways to scale and the GCs (SWFF, Zika, SLA@B, PAEGC) that succeeded were those that had a clear understanding of acceleration support and scaling from the outset and managed tailored support to the innovations. How? Require GC design and inception plans to include acceleration services (see P1) and select and fund principally early￾to mid-stage innovations that have the potential to reach scale. 70 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION S6 Create a shared model, methodology, and practical toolkit for ecosystem engagement to be used by all GCs, including case studies of successful examples of where GCs have done this, together with advice on likely costs of engagement and examples of how other GCs have budgeted for these costs. Ecosystems Section 4.4.2 Why? All GCs have links with different ecosystems at the outset through USAID and donor partner networks. These networks are assets which should be leveraged to start building an enabling environment for GC-funded innovations. How? Bring Missions and other USAID OUs into the design phase of GCs, to identify and map the key influencers and connectors who can build awareness of, interest in, and enthusiasm for innovative solutions in specific territories. S7 Include ecosystem engagement in program design from the outset, building connections with other USAID programs operating in the same space and with similar objectives. Use USAID’s convening power to forge early links between a wider network of innovators, experts, funders, and potential investors. Build cross-cutting, portfolio-wide links between GCs, Missions, and other USAID OUs. Ecosystems Section 4.4.2 Why? To create the enabling environment for testing and adoption of innovations and to lay the foundations for long term uptake and sustainability. How? Define and map the ecosystem for GC innovations at the outset and draw on experience across the portfolio to create a simple toolkit for grantees and program managers to engage with different ecosystem actors at different stages along the GC timeline. S8 Share and apply learning from the evaluation about what has worked well to catalyze funding and awareness to both design and implementation of GCs, tailoring chosen approaches to the specific challenge and context. Catalytic effects Section 4.5.2 Why? To increase GC effectiveness and funds for innovation. How? Tailor approaches to the specific challenge and context; establish a clear strategy for how a GC will achieve catalytic effects; establish requirements, such as securing match funding; provide TA on securing investment and becoming investor ready; set targets for catalytic results; support innovators to evidence and showcase achievements; leverage the brand value of donor funding; map the investor ecosystem and involve investors in the GC; and showcase the distinctiveness and strengths of the GC model. S9 Plan engagement of USAID Missions upfront and use it to inform plans for designing and implementing the GC. USAID engagement Section 4.7.1 Why? To avoid missing opportunities to strengthen both GCs and Missions through effective engagement – e.g., to increase relevance to context and in-country support to innovators. How? For GCs with a geographic focus: research the distribution and nature of the problem and identify Missions who could join design; reach out to understand alignment with the CDCS; consider reserving GC funding for Missions to allocate or a buy-in mechanism; use Missions to understand political economy. For global GCs, reach out to Missions where applications are likely to come from and where awards are made; identify opportunities for mutual learning; consider a proposed model for formalized engagement (see Section 4.7). S10 Establish and implement an overarching approach and principles for MEL in GCs, which sets out clear expectations and guidance for each (new) GC to apply and adapt to their particular dimensions (e.g., sector, geographies, timescales, innovation stages supported, scaling ambitions, target beneficiaries by type etc.). Results management Section 4.8.1 Why? The evaluation found that lack of clear objectives and results measurement hampered the ability to assess results for some GCs and that there was no overarching framework to guide GCs to establish MEL at both GC and individual grantee level. How? Set out the broad expectations for every GC to meet (theory of change, MEL framework, evaluation plan, objectives, supporting indicators and MEL systems) that they can apply to their own Challenge. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 71 S11 GC managers (USAID and other donors) should specify their data requirements and ensure that GCs have a dedicated MEL team and an efficient MEL mechanism focused on collection of relevant and useful data against defined indicators and targets. Results management Section 4.8.1 Why? To ensure that GCs put in place the processes and resources that are needed for results to be measured and learned from, and that data is used. How? By aligning innovator and program reporting to donor agency requirements, aggregating grantee data to build GC level results, and ensuring only data needed is collected. S12 Design for and incentivize longer-term reporting to provide valuable post-grant evidence of innovation success. Tie resources to post-award reporting. Results management Section 4.8.1 Why? Only two GCs do this to any extent currently, and little is known about the longer-term outcomes of innovations supported by GCs. Innovators need incentives to keep reporting. How? Develop plans for longer-term tracking. Provide funding and non-financial incentives (such as free publicity) to do this. S13 Move forward on measuring cost effectiveness across the range of different GCs by sector and stage of innovation by implementing the five steps listed. Cost effectiveness Section 4.9.2 Why? There is little systematic analysis of the cost effectiveness of the GCs and a very strong interest within USAID and the GCs to develop some broad principles of CEA for all GCs. How? USAID needs to set up a cross-sectoral working group to agree on a broad set of principles, standard costs, and outcome indicators according to the 6 key steps set out in the CEA Framework (Annex 11). S14 Build a vision and objectives for the GC which is shared by all parties involved in governing and managing it. Governance Section 4.10.2 Why? GCs (e.g., MAVC) can be undermined by conflicting or unclear priorities and by disagreements about basic GC design parameters. How? Develop and establish: clear objectives, possibly with support from external experts; a well-articulated theory of change and intervention model that takes account of the level of ambition, stages of innovation to be supported, nature, geographic distribution, and capacities of potential grantees; and intended extent of ecosystem and Mission engagement. 5.3 PROGRAMMATIC RECOMMENDATIONS Many recommendations in Section 4 of the evaluation are about GC implementation. They are directed to USAID GC Managers (again, working with other funding partners) and to partner or implementing GC managers, usually an external party. As with the strategic recommendations, headline recommendations are listed, and cross referenced to the supporting detail set out under the forward-looking questions in Section 4. TABLE 5.2: USAID META-EVALUATION: PROGRAMMATIC RECOMMENDATIONS PARTNER GC MANAGER USAID GC MANAGER P1 Ensure that GCs have an acceleration and scaling strategy in place from the outset and focus them on supporting early to mid-stage innovations. Scaling & acceleration Section 4.3.2 Why? Acceleration support that was designed at the outset and where the GC provided tailored acceleration services to the innovations was the most effective model adopted by the GCs (SWFF, Zika, SL@B, PAEGC). How? There are 5 key elements to acceleration support, listed below: P1.1 Ensure acceleration and scaling support is in the design of the program, and that the GC Manager has an understanding and awareness of the principles of acceleration to manage a portfolio of innovations. 72 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION P1.2 Provide tailored 1:1 acceleration support to innovator combined with a mentorship approach and the ability to facilitate networking and connections, including for public sector uptake (e.g., for health and education innovations). P1.3 Monitor and measure the performance of acceleration services whenever they are used to enable learning within and across GCs, and to ensure that these services are demand-driven and relevant. P1.4 Iterate the approach to acceleration and scaling over time to ensure that services adapt from the experience and the changing context. P1.5 Ensure that the acceleration support provider, likely to be external, has the necessary local knowledge and networks and experience of the local context. P2 Dedicate time and attention at the outset to agree on a common definition of what constitutes the relevant ecosystems for each GC and an approach to engaging with wider system actors. Ecosystems Section 4.4.2 Why? It is important that all funding partners and program managers have a common understanding of what constitutes the ecosystem surrounding different innovations, so that investment in ecosystem engagement is optimized and not dissipated through mismatched understanding or expectation. How? Work together to define and map the ecosystem, drawing on the existing learning and experience from other GCs or programs with similar objectives and sectoral focus. P3 Apply the ecosystem strengthening methodology and toolkit (which Strategic Recommendation 7 recommends is developed) to the specific context of each GC. Ecosystems Section 4.4.2 Why? There is no need for GCs to keep reinventing the process of engaging with different ecosystem actors. They should rather use a standard approach based on ten years of experience across the portfolio. How? Program managers should use the toolkit as a starting point but customize their engagement plans based on local context and market and regulatory requirements. P4 Leverage existing USAID relationships in sectors and geographies relevant to individual GCs. Ecosystems Section 4.4.2 Why? Missions and other USAID OUs have built networks in specific territories where GC grantees will be testing their solutions. These networks should be treated as key assets for building openness to new solutions. How? Program managers should reach out to Mission or OU staff to build alliances at program level, and then work with colleagues to co-create workable engagement plans for different actors in different local contexts. P5 Apply learning from the evaluation about what has worked well to catalyze funding and awareness to both design and implementation of GCs, tailoring chosen approaches to the specific challenge and context. Please see S8 above. Catalytic effects Section 4.5.2 P6 Take steps to deter ineligible or weak applications and funnel applications effectively through a staged application process after initial eligibility screening has taken place. Procurement Section 4.6.2 Why? This would reduce volumes and increase speed and efficiency without compromising quality. How? Have very clear eligibility and screening criteria; automated screening; request concept notes, and short-list for full proposals; and review and adapt after each funding round. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 73 P7 Define target grantees carefully in relation to fund objectives; tailor application processes to make them accessible to these organizations. Procurement Section 4.6.2 Why? To ensure the GC reaches the most suitable innovators, and because non-traditional actors are different and are not all equally suitable. How? Research the innovator landscape; set eligibility and selection criteria and tailor communications and processes to match; improve based on feedback. P8 Increase the quality and quantity of applications from local innovators by conducting research to understand and support the local market. Procurement Section 4.6.2 Why? To identify and reach potentially strong innovators. How? Scoping studies; involvement of USAID Missions; engage potential applicants; support their applications with advice through local champions. P9 Undertake local and regional analysis of context and political economy to inform local outreach and selection criteria. Procurement Section 4.6.2 Why? To understand the enablers and inhibitors of innovation and scaling, including systemic issues. How? Research and analyze the issues, power dynamics and entry points. Use the analysis to inform targeting, selection, and support of local innovators, particularly on how they can impact the wider system. P10 GCs should engage with USAID Missions to add value, recognizing that they are independent agencies which differ markedly in priorities, resources, programing, and ability to engage. USAID engagement Section 4.7.1 Why? To recognize and best fit with their individual needs. How? Take a shared value approach, guided by the CDCS and demonstrating how the GC could support it. Establish focal points, and maintain regular communication which fits with day-to-day priorities, schedule and capacity. P11 Engage Missions for ecosystem level change and longer￾term MEL. USAID engagement Section 4.7.1 Why? Missions are well positioned to support ecosystems; longer-term effects of GCs are not currently known, and systems are needed. How? Maximize Mission staff contacts and networks locally including with governments; involve Missions in promoting the GC to local innovators and supporting adoption, replication, or scale up through strategic introductions; engage Missions in tracking innovation progress and maintaining connection with grantees. P12 Ensure that MEL is aligned to the GC objectives and innovation stage, and streamlined to minimize the burden on grantees while maintaining clear and regular reporting requirements. Results management Section 4.8.1 Why? Evaluation findings showed that a third of innovators did not consider MEL to be aligned to objectives, that grantees experienced reporting burdens, and that the right data could be powerful for innovation strengthening. How? Link indicators to objectives (e.g., scaling objectives could be evidenced by indicators about the pathway to scale); ensure that data collection and reporting is proportionate and useful; minimize indicators and ensure utility; measure what you want to see. P13 Give early consideration to evaluation plans, defining evaluation needs and the methodology best suited to meeting them. Results management Section 4.8.1 Why? Six out of nine GCs commissioned evaluations, all of which provided valuable data and insights. How? Define evaluation questions and methods appropriate to objectives of the GC; budget and resource evaluation appropriately; plan procurement timescales. 74 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION P14 Ensure that GC objectives and MEL frameworks make gender and other characteristics of exclusion (e.g., age, disability) visible so that the impact of GCs on equality and inclusion (which is strongly linked to impact on poverty and developmental outcomes) can be seen and understood. Results management Section 4.8.1 Why? Inclusion is strongly linked to developmental outcomes. Without making GESI visible, both positive results and uneven benefits for different groups cannot be known. How? Set targets and measure results in disaggregated rather than general terms (e.g., men, women, boys and girls rather than ‘people’); define terms used (e.g., ‘marginalized’) and ensure data is collected accordingly; gauge grantee awareness and capacity on GESI and why it matters; provide training and tools to support. P15 GCs and projects should undertake structured learning and use data and learning for adaptation and improvement. Results management Section 4.8.1 Why? So that they can constantly improve: grantees found this was an important way of developing their innovation. How? Develop a culture of learning, introduce systems and processes to support learning, establish learning objectives, learn from failures as well as successes, and apply the learning. P16 Ensure that ecosystem strengthening goals are incorporated within GC theories of change and the overarching MEL framework. Results management Section 4.8.1 Why? To incentivize steps to change ecosystems and make achievements visible. How? Include change pathways and indicators like stakeholder engagement, collaboration, research, and convening; include them as evaluation questions; use advocacy assessment tools; include ecosystem strengthening proposals as a selection criterion. P17 Ensure effectiveness of a GC governance model and decision making processes, design them in detail at the outset, and ensure that they are adequately resourced. Governance Section 4.10.2 Why? To ensure that structures are in place to support good quality decision making throughout the design stage. How? Set out processes, structures, membership, roles, and terms of referenc,e as well as relationships between different boards and committees; address and resolve any areas of conflict; determine cost requirements. P18 Define, document, and communicate partners’ roles and responsibilities, reflecting their strengths. Establish and maintain trust between all parties throughout. Governance Section 4.10.2 Why? To avoid overlaps, gaps and confusion in decision-making, and conflicts in priorities which could undermine the GC, and to make the most of the strengths of different parties. How? Identify partner roles and strengths. Ensure that it is clear to everyone involved in managing and implementing how the parties relate to each other. Invest in building relationships. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 75 6 DISSEMINATION PLANS The meta-evaluation has been focused on actionable evidence, principally for those involved in GC policy and implementation decisions. Its dissemination is supported by a set of knowledge products (KPs) and a Learning and Dissemination Workshop. In addition to the core USAID GC team, these are directed at a wide audience that may include any or all of the following groups: • USAID staff directly involved in the different Grand Challenges. • Other USAID staff working on other innovation programs. • Senior officials and staff in bureaus across the agency working on development problems in the same sector. • Donors currently or previously partnered with USAID on specific GCs. • Other donors who have expressed interest in the GCs and may become funding partners on future GCs. • Critical stakeholders and supporters, e.g., congressional supporters of innovation at USAID and selected members of other development organizations who are practitioners of open innovation. The dissemination plan for the KPs and final participants at the Learning Workshop are at the discretion of the USAID team leading the meta-evaluation, to be agreed closer to the time of the workshop. 6.1 KNOWLEDGE PRODUCTS In order to facilitate uptake and in line with the utilization-focused approach of the evaluation, a range of standalone knowledge products resulting from the evaluation’s various activities will be produced and disseminated among interested parties. These smaller, more targeted deliverables will be of particular interest to specific target audiences and increase the evaluation’s relevance in the internal USAID ecosystem as they address more specific topics, and are intended to function as key reference resources. The purpose of the KPs is to present critical information and learnings from the meta-evaluation with user and action-oriented content in an accessible format. The knowledge products will highlight key findings and takeaways so that they are a) informative to audiences looking for focused analysis on the GCs, and b) directly useful to program managers looking for design and implementation advice. Two of the proposed knowledge products – an overview of the GC portfolio and a summary of the focus, findings, and recommendations of the evaluation – will also provide an orientation roadmap to readers of the full evaluation report. As well as being utilization-focused and actionable, the knowledge products will be designed to be visually appealing and accessible through the use of infographics and visual representations of data. The knowledge products to be disseminated are elaborated in Table 6.1. TABLE 6.1: EVALUATION KNOWLEDGE PRODUCTS TOPIC PURPOSE AUDIENCE The Grand Challenges Portfolio Visual representation of the Grand Challenges portfolio with data presented on the portfolio in aggregate Wide within USAID and beyond, including donor partners Delivering MEL in the Context of the GCs Challenges faced by GCs with regards to MEL, and practical tips for developing a fit￾for-purpose MEL framework for GCs (e.g., measuring scaling, disaggregating data, common indicators, use of beneficiary feedback, use and timing of evaluations) USAID GC Policy USAID and partner GC Managers Taking Innovations to Scale with the GCs Practical advice on scaling innovation for current and future GCs Wide, as per learning and dissemination workshop Developing a CEA Framework for GCs Practical tips for developing a CEA framework to be used by all GCs USAID GC Policy USAID and partner GC Managers The USAID GC Meta-Evaluation Overview of the meta-evaluation and recommendations for increasing the effectiveness of existing GC programs and the design of future GCs – the executive summary of this report. Wide within USAID and beyond including donor partners 76 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 6.2 LEARNING AND DISSEMINATION WORKSHOP The evaluation team will also host a remote learning workshop at the end of the evaluation so as to increase the profile of the evaluation, and to disseminate the top-line findings, conclusions, and recommendations to a wider audience than might be reached simply through dissemination of the report. The workshop is intended to be attended by representatives of the different audience and stakeholder groups as outlined. The workshop will ensure that the meta-evaluation is not simply a “paper exercise”, but has a life and a value beyond the evaluation process itself as a useful and actionable knowledge asset for USAID staffers and partners working on GCs or other open innovation programs. The workshop will be hosted via an online video conferencing platform and will make use of appropriate remote working tools to aid engagement and interactivity. Evaluation team members will present findings on select topics and stimulate group discussions in plenary and in breakout rooms. A memo capturing the discussions at the workshop will be circulated following the event. Photo: Global Devlab/USAID USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 77 ANNEX 1 PORTFOLIO MAPPING 99 Due to partial information, the heatmap does not include the details of CHIC Round 2 grantees, some SL@B Round 2 grantees, nor any SL@B Round 8 grantees. A1.1 GC PORTFOLIO HEATMAP The following heatmap indicates the number of GC-funded projects operating in countries around the world. Some projects have operations in multiple countries. Global and regional projects have not been included. The 10 countries with 21 or more operations have been labelled with breakdowns by GC. EBOLA has not been labelled since neither of its country-specific projects have operations in any of those 10 countries. Kenya hosts by far and away the highest concentration of GC-funded projects (86). The country with the next highest concentration (India) hosts almost a third fewer innovations (52). Broadly speaking, there appear to be three regional clusters: East Africa, South Asia, and Southeast Asia (namely Indonesia and The Philippines). It is worth noting, however, that the distribution of projects around the world is of course affected by the sometimes highly localized thematic areas of some of the GCs such as Zika (Brazil and Latin America). The heatmaps do not include projects with global scopes. This information has instead been provided in Annex Table 1. OVERALL PORTFOLIO99 Nigeria: 32 No. of Projects Legend ACR GCD CHIC MAVC PAEGC SL@B SOGE SWFF Zika 51-100 1-10 0 11-20 21-50 India: 52 8 3 3 31 7 Ethiopia: 21 3 2 10 3 Philippines: 29 4 23 1 1 Ghana: 26 5 7 1 9 2 2 1 6 3 1 7 13 1 Kenya: 86 4 6 8 6 28 26 6 2 Tanzania: 33 1 7 4 10 8 2 1 Bangladesh: 25 1 5 2 2 12 3 South Africa: 24 2 14 5 3 2 2 3 16 11 4 7 Uganda: 45 3 A1.2 PORTFOLIO ANALYSIS ANNEX TABLE 1: NO. OF COUNTRIES IN WHICH EACH GC OPERATES GC NO. OF COUNTRIES NO. OF GLOBAL PROJECTS All Children Reading 38 19 Combating Zika and Future Threats 16 5 Creating Hope in Conflict 27 2 Fighting Ebola 2 12 Making All Voices Count 20 4 78 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION GC NO. OF COUNTRIES NO. OF GLOBAL PROJECTS Powering Agriculture 23 0 Saving Lives at Birth100 40 27 Scaling Off-Grid Energy 23 25 Securing Water for Food 40 0 TOTAL101 83 94 ANNEX TABLE 2: APPLICATIONS AND AWARDS GC NO. OF ROUNDS TOTAL NO. OF APPLICATIONS TOTAL NO. OF AWARDS AWARDS AS PERCENTAGE OF APPLICATIONS All Children Reading 3 1,195 92 7.6 percent Combating Zika and Future Threats 1 850 26 3.1 percent Creating Hope in Conflict 3 1,845 50 2.7 percent Fighting Ebola 1 1,500 14 0.9 percent Making All Voices Count 7 2,849 178 6.2 percent Powering Agriculture 2 1,344 24 1.8 percent Saving Lives at Birth 8 4,444 147 3.3 percent Scaling Off-Grid Energy102 5 345 196 - Securing Water for Food 4 1,594 40 2.5 percent TOTAL 34 15,966 767 4.8 PERCENT 100 Due to partial information regarding Round 2, these figures may be slight underestimates. 101 The total number of countries is the total number of unique countries. 102 Providers were not required to collect the number of applications to SOGE. Data on the number of applications were thus collected only for two rounds of follow-on funding (the Solar E-Waste and Household Solar Working Group Challenges) which continued after SOGE’s end date and do not relate to the GC proper. The figures for SOGE also do not include the number of awards for Round 3 due to partial data. This means that the number of applications and awards for SOGE detailed here are likely to be significant underestimates. The application success rate for SOGE has also not been included for these reasons. Source: GCD Metrics Check, accessed 02/05/2020. ANNEX TABLE 3: DIVERSITY OF AWARDS GRAND CHALLENGE LMIC APPLICATIONS PERCENTAGE OF TOTAL APPLICATIONS FIRST TIME APPLICATIONS PERCENTAGE OF TOTAL APPLICATIONS LMIC AWARDS PERCENTAGE OF TOTAL AWARDS All Children Reading: A Grand Challenge for Development 543 45 percent - - 35 38 percent Combating Zika and Future Threats - - - 69.5 percent 2 7.7 percent Creating Hope in Conflict - - - - 16 30.8 percent Fighting Ebola - 54 percent - - 2 14.3 percent Making All Voices Count 2,849 100 percent 2,597 91.2 percent - - USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 79 GRAND CHALLENGE LMIC APPLICATIONS PERCENTAGE OF TOTAL APPLICATIONS FIRST TIME APPLICATIONS PERCENTAGE OF TOTAL APPLICATIONS LMIC AWARDS PERCENTAGE OF TOTAL AWARDS Powering Agriculture: An Energy Grand Challenge 803 59.7 percent 339 25.2 percent - - Saving Lives at Birth 2,225 49.4 percent - - 25 17.0 percent Scaling Off￾Grid Energy - - - - - - Securing Water for Food 1,038 66.3 percent 1,198 76.5 percent 22 55 percent 103 The USAID/partner funding breakdown for SL@B is not available. It received a total of $100M in funding. Source: GCD Metrics Check, accessed 02/05/2020. ANNEX FIGURE 1: TOTAL FUNDING – USAID AND DONOR PARTNERS (USD, MILLIONS)103 9% 11.2 30% 15.4 22% 10.0 31% 12.4 35% 12.1 100% 30.0 48% 14.0 100% 8.9 91% 113.3 70% 36.4 78% 35.0 69% 27.5 65% 22.2 52% 15.0 0 20 40 60 80 100 120 140 SOGE PAEGC MAVC CHIC SWFF Zika ACR GCD EBOLA USAID Other Partners 80 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ANNEX 2  EVALUATION MANAGEMENT Triple Line has been engaged by Resonance on behalf of USAID to undertake this meta-evaluation. Resonance manages the contract and is the day￾to-day point of contact for the Triple Line team, engaging on their behalf with USAID and vice versa, and facilitating joint discussions with USAID. Resonance has guided and advised on the evaluation process and has provided quality assurance and sign-off on deliverables before they are submitted to USAID. The project to deliver the evaluation is entitled Catalyst. The evaluation team has undertaken quality assurance on all evaluation outputs across key process, normative, and technical criteria. The evaluation team has participated in weekly progress calls with the Catalyst team and bi-weekly reviews with Catalyst and USAID to provide progress updates and identify risks. An up-to-date evaluation work plan is updated and shared before each meeting. The Triple Line contract has had the following technical and contractual oversight: • Courtney Roberts, Moonshot Global/ Catalyst, lead for technical oversight. Courtney has reviewed, commented, and requested necessary revisions on methodology and all technical deliverables. She has been responsible for recommending approval by Catalyst and USAID. • Stephen Rahaim, Resonance/Catalyst, contract lead and technical oversight contributor. Stephen has reviewed, negotiated, and approved any changes to scope, timeline, or the contract. He has approved all deliverables in coordination with USAID and invoices for payment. He has reviewed and commented on all technical deliverables. • Lorin Kavanaugh-Ulku, USAID Open Innovation Competitions lead, primary point of contact. Lorin or her designee has reviewed, commented, and requested revision to technical deliverables. She has approved all technical deliverables and acted as the primary point of contact between Triple Line and USAID. • Scott Jackson, USAID. Scott has provided technical support and coordination to USAID’s oversight of the project. He has, when designated, approved, commented, and requested revision to technical deliverables. Internal workshops have been hosted to provide training on how to use evaluation tools to ensure consistency and to share findings. All deliverables are peer reviewed within the team and overall quality assurance is conducted by the team leader. Catalyst has frequently added additional technical oversight and quality assurance to the team in a review and coordination capacity. The Triple Line evaluation team consists of: • Clarissa Poulson, Interim Team Leader (from March 2021)/Senior Evaluator • Martin Wright, Team Leader/Senior Evaluator • Sudhanshu Joshi, Senior Evaluator • Katherine May, Senior Evaluator • Jeevan Raj Lohani, Senior Evaluator • David Smith, CEA Specialist • Matthew Kentridge, Innovation and Scaling Strategy Specialist • Dr Shoa Asfaha, Gender and Social Inclusion Specialist • Julian Ratcliffe, Analyst and Data Manager Triple Line has worked with Athena International to produce various report inputs, including MEQ analyses, the case studies, and report and knowledge product visual design. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 81 ANNEX 3  EVALUATION METHODOLOGY A3.1 DATA COLLECTION The evaluation team made use of a range of data sources as evidence on which its findings, conclusions, and recommendations are based. They include: • Portfolio mapping. • Desk review. • Key informant interviews (KIIs). • Case studies. • Grantee survey. • CEA review and framework. • Comparator analysis. • Gender equality and social inclusion (GESI) review. To facilitate data collection, the evaluation team compiled a data collection toolkit (see Annex 6) with input and sign off from USAID that was utilized to secure KIIs, to guide interview discussions, and to guide the creation of the case studies. The data collection toolkit ensured consistency of approach between evaluation team members and ensured that the data collected was both relevant and useful. A3.1.1 PORTFOLIO MAPPING An extensive portfolio mapping exercise was conducted during the inception phase of the evaluation. The evaluation team was granted access to USAID’s GC database spreadsheet which included details of all GC grantees and summary analyses broken down by GC and by funding round. The document was live and updated, with the most recent version updated on 26th August 2020. This database formed the primary data source for the majority of the evaluation team’s portfolio-level analysis. The portfolio mapping exercise was used to inform a prioritization exercise to determine which grantees should be contacted for the grantee survey. Grants which had finished over a year before the evaluation were deprioritized as response rates were likely to be lower than recently closed and ongoing grantee projects. This exercise involved reviewing grantee-level documentation to fill in data gaps in collaboration with USAID. Analysis was also conducted along various other relevant axes including call type/mechanism, applicants, awards, and funding catalyzed. A series of geographical heat maps were created to visually represent the distribution and concentration of grantees around the world (Annex 1). Using open￾source Quantum Geographical Information System (QGIS) software, the number of grantees by country at an overall portfolio level and by GC was plotted onto a world map so that geographic diversity across the GC portfolio could be easily and accessibly determined. A3.1.2 DESK REVIEW As mentioned in Section 2.1, responsibility for data collection, capture, and analysis for each GC was distributed amongst evaluation team members. The first stage of this process was to review an extensive set of program and strategic level documentation sent to the evaluation team by PoCs and USAID. Key documents reviewed included: • High level USAID & partner donor GC documentation. • GC design documents. • Intervention logic and/or theory of change, and framework for measuring results. • Governance documents – MOUs, partner donor agreements, etc. • Program strategies or internal guidance on all aspects of the grant management cycle, which might include: selection process and criteria; due diligence processes; monitoring, evaluation and learning frameworks/strategies; outreach, etc. • Fund level reporting – annual, final, etc. • Portfolio analysis and program level data. • Individual grantee documentation – grantee applications, budgets, reports, MEL frameworks, etc. • Learning papers and fund level evaluation reports. • Comparator program literature. The evaluation team was granted access to 2,397 documents in total. The team proceeded to select key documents relating to design, implementation, and results by GC, and logged key findings in a standardized data capture template (one per GC reviewed) using one row per data source to 82 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION enable clear traceability of evidence sources. In total, the team extracted data and evidence from 502 documents. A full list of the documents the evaluation team had access can be found at Annex 8. A3.1.3 KEY INFORMANT INTERVIEWS Key informant interviews (KIIs) were a critical plank of data collection for this meta-evaluation. During implementation, KIIs were conducted with a total of 64 different people across nine GCs.104 The KIIs augmented available evidence from the document review, gave voice to and collected different perspectives, and generated qualitative insights to triangulate data against other sources of evidence. The team interviewed USAID management, implementing partners, donor partners, GC PoCs, leads of other USAID programs, and external stakeholders, including for the CEA, comparator analysis, and case studies. Some interviewees were interviewed more than once by different team members for different purposes, and some interviews involved more than one interviewee. Given the closure of MAVC in 2017, the evaluation drew on synthesized evidence from interviews conducted with MAVC key informants in 2018 for Triple Line’s meta-evaluation of 10 of Sida’s challenge funds. A full list of KIIs can be found at Annex 7. The KIIs were intentionally semi-structured to allow for emphasis on particular insights to emerge organically through the process of consultation while ensuring consistency across stakeholder types and interviewers. To that end, a set of five interview guides were constructed to steer discussion. They were differentiated by target audience and stakeholder type according to the questions that each stakeholder type would be most likely to be able to answer insightfully and informatively. Oral consent was requested at the start of each KII as well as written consent in the form of a form sent ahead of time to each interviewee detailing the purpose of the evaluation and interview (Annex 6, Section A6.2). The KIIs yielded a wide range of insights into GC performance, design, implementation, lessons learned, and best practice. The detailed notes taken during each KII have collectively formed a core data set on which the evaluation team’s conclusions and recommendations are based. 104 The GCs for which interviews were conducted included EEHSC and excluded MAVC. It was partially on the back of the KIIs conducted for EEHSC that it was decided to exclude it from analysis due to its comparatively lower amount of funding, small number of grantees for which USAID was responsible, and limited data availability. A3.1.4 CASE STUDIES Case studies were planned with the objective of digging deeper into some of the thematic priorities that emerged during the meta-evaluation. Initiated towards the end of the process, the purpose was to document key learnings from selected awards that could be used to inform both the evaluation and other challenges in the future. Three key thematic priorities were identified: scaling up and sustaining the innovation, partnering with USAID Missions and in-country governments, and using MEL effectively for acceleration and scaling. The evaluators asked USAID Grand Challenge Managers to shortlist awards that would offer good learning in these three thematic areas. Eight awards were identified from five GCs.All were approached but only five responded. The awards were studied in detailed through desk review of available reports/documents. The review was followed by an interview with the award manager/s. The qualitative data compiled from both sources was collated and analyzed to develop the case studies. A3.1.5 GRANTEE SURVEY The evaluation team conducted an anonymous online survey with current and recently completed grantees. This served as a data source and evidence base to feed into the findings and conclusions. As highlighted in the evaluation matrix, we have assessed the online grantee survey as providing useful perceptions and value judgements for MEQs 1, 2, 4, 7, 8, 9, and 10. Further details of the survey can be found in Annex 6, Section A6.4, and a summary of survey results can be found in Annex 15. The evaluation team selected a sample of projects that were either still ongoing or had only recently ended for inclusion in the online grantee survey. Since data on grant end dates was inconsistently available, the evaluation team focused on projects that had started in 2017 or later. The projects considered corresponded to about 11 percent of the overall GC portfolio. Responses were received from 54 grantee organizations, representing about eleven percent of the total number of grants awarded under the GC portfolio. Responses were received from all GCs to whom it was sent. (It was not sent to MAVC grantees as this fund closed in 2017.) Only one response was received from each of ACR GCD and EEHSC, but USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 83 there were at least five respondents from each other GC. Being anonymous, the grantee survey allowed grantees the space to provide confidential insights and for the evaluation team to fill qualitative and quantitative data gaps. In order to maximize response rates, completion time was limited to around 8 minutes. The tool was designed to ensure that only data of immediate relevance and utility to specific evaluation questions was collected. A3.1.6  CEA REVIEW AND FRAMEWORK In relation to MEQ 3, a review of the feasibility of conducting a portfolio-level CEA in the future was originally planned to be disseminated as a standalone product. It was determined early on in the evaluation that there was insufficient data to conduct a meaningful CEA as data on activity costs and beneficiary-level outcomes had not been collected systematically across the GC portfolio. A workshop was held with the USAID team and Points of Contact on 22nd December 2020 to identify key priority areas for developing a set of standards and principles for a cost effectiveness framework that could be applied to GCs. A brief review of CEA methodologies and practices by USAID and other donors was undertaken to help guide the development of the forward-looking CEA framework. Both the CEA Review and CEA Framework will be disseminated as standalone documents (see Annex 10 and Annex 11), and some of the key findings have been integrated into Section 4. The overall approach undertaken was to set out guidance for USAID towards developing a standard methodology for measuring the cost effectiveness for a GC at two levels: the Challenge level and the Grantee level. At Challenge level the CEA looks at the management and support costs to the portfolio of innovations. At Grantee level the CEA measures the cost effectiveness of the innovations and, where possible, the unit cost of the outcomes achieved per beneficiary. The approach proceeded in four steps: • Completed literature review of the reports from the 9 GCs and cost effectiveness papers on innovation funds. • Conducted KIIs with USAID and other researchers on current and proposed approaches to capturing standard outcome indicators for the GC portfolio including health, education, agriculture and livelihoods, energy, and governance. • Conducted KIIs with GCs on performance data currently collected, evaluations planned, and potential new data to support CEA. • Conducted a workshop with USAID and PoCs to identify priorities for a CEA Framework. • Prepared two standalone documents: The CEA Framework, supported by the CEA Review (see Annexes 10 and 11). A3.1.7 COMPARATOR ANALYSIS The comparator analysis was used to address the following questions related to the rationale for selecting the GC mechanism for programing rather than other development intervention mechanisms: • What is the value of putting funds into Grand Challenges rather than other program types? • Do Grand Challenges generate a greater range of innovative solutions than other programs? • What additional value do Grand Challenges bring compared with other programs, and are there constraints associated with Grand Challenges which other programs manage to avoid? The evaluation team identified four programs whose analysis would address these questions by focusing on the specific differences between GCs and more traditional programs that operate in the same sectors or have comparable delivery objectives. The list of comparators was derived from discussion at the Prioritization Workshop held at the end of July and subsequent discussions with USAID and Catalyst. The list comprises the following programs: • The Innovation Investment Alliance. • Partnering for Innovation. • Local Works. • SHOPS Plus. 84 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ANNEX TABLE 4: USAID COMPARATOR PROGRAMS SECTOR PROGRAM OBJECTIVES PARTNERSHIP MODEL RATIONALE FOR SELECTION Agriculture, Environment, Health, WASH Innovation Investment Alliance Invest in proven, transformative social enterprises to scale their impact. Share insights on how to scale social impact. USAID Global Development Lab; Skoll Foundation, Mercy Corps Program focused on innovation, but using different partnership model from GCs. Agriculture Partnering for Innovation Raise income of smallholder farmers and reduce hunger. Increase agricultural productivity. Create partnerships to build successful agribusinesses to serve smallholders. Investment assistance: expert guidance, technical support. USAID, global and local agribusiness, impact investors, farmers, universities, tech companies, financial institutions, foundations, NGOs, governments Program focused on innovation, but using different partnership model from GCs. Various Local Works Program Develop flexible solutions to advancing locally-owned development within USAID. Use systems approaches for sustainable outcomes with local actors. Test approach to local leadership/ownership. Adapt existing processes for greater local ownership of development processes. Partnership with B&MGF and GCC Programs that dispense grants but use traditional procurement mechanisms. Lower focus on innovation.105 Health SHOPS Plus Harness the potential of the private sector. Catalyze public-private engagement to improve outcomes in a range of health areas. Brokering partnerships between government, civil society and corporates Program set up using different procurement mechanisms & partnership models – to be compared with GCs across key dimensions (e.g., leverage, time to launch, impact, etc.). 105 We are grateful to Catalyst for pointing out that within the Local Works program ‘there are some solutions that are funded that take on the form of more traditional implementation projects vs investments in specific social enterprises (or product innovations) with a solution.’ When conducting the comparator analysis around Local Works we will consider projects which cover both approaches. The comparator analysis was conducted through a mixture of KIIs and document review. Relevant background materials were sourced from program websites (general information as well as documents accessible through publicly available links); documents available through key platforms such as grants.gov; beta.SAM. gov (formerly fedbizopps.gov); dec.usaid.gov; and documents provided by specific PoCs for each of the comparator programs. Documents that aligned with the GC desk review were prioritized. • Research or briefs written to inform the design and procurement mechanism of different programs. • Solicitation documents including Requests for Proposal (RFPs). • Project Appraisal Documents (PADs). USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 85 • Contracts with delivery partners (where relevant). • Impact reports. • MEL reports and learning documents. • Workshop reports. • Partnership agreements (where relevant). • Evaluation reports (if available). The evaluation team also conducted interviews with key PoCs for each of the selected comparator programs. Between the KIIs and document requests, the evaluation team engaged with comparator PoCs over three rounds of data sourcing. A3.1.8 GESI REVIEW The GESI review was used to address the following questions about how GESI had been integrated into GCs at each stage of the intervention, so that evidence could be integrated into MEQ findings where appropriate. The review was structured by the three themes of the GC intervention model. DESIGN • How was GESI considered during the design stage? For example, was any gender or social analysis incorporated into any context analysis undertaken to develop understanding of the problem being tackled or the potential innovator landscape to address it? • How has GESI been considered in the grantee selection criteria? IMPLEMENTATION • How has GESI been considered in the provision of grant management and technical support? • How has GESI been considered in the monitoring, evaluation, and learning plans developed for the GC? RESULTS • What are the reported results in terms of gender equality and social inclusion in the GCs? • To what extent has innovation contributed to better outcomes for marginalized groups? • The review analyzed evidence on GESI captured for the meta-evaluation through desk review and KIIs to reach its assessment and conclusions. For each intervention area, the review identified key findings in response to the review questions, and generated key insights about the factors which had contributed to them. It used this to make a set of recommendations for how GCs might strengthen their integration of GESI into programing in the future. A3.2 DATA ANALYSIS AND SYNTHESIS The intervention model analytical framework was a tool which was developed to interrogate the design, implementation, and results of each of the GCs. The evaluation team developed this tool to ensure consistency in the review of the GCs against which each of the MEQs will be assessed, and to guide data collection and capture. The analytical framework is set out in Annex 5. The intervention model analytical framework was split into the three focus areas of the intervention model – design, implementation, and results under which sit different “component areas” – such as “Thematic Objectives and Theory of Change” in design. As a first step in the analytical process, the evaluation team prepared descriptive profiles of the design, implementation, and results of each GC. These were shared with relevant PoCs for validation and to ensure factual accuracy. The descriptive profiles enabled comparative analysis of the basic characteristics of each GC and the identification of commonalities and differences. Once the data collection and capture processes was completed, the evaluation team synthesized evidence into an analytical framework for each GC, organized by intervention phases (design, implementation, and results), by both strengths and weaknesses. Using the framework, the evaluators assessed the evidence to code each MEQ at each phase with a Red, Amber, or Green (RAG) rating for each GC. The common framework ensured consistency of approach across team members and a clear link to evidence collected, with each judgement justified with evidence and examples. Annex Table 5 provides a sample extract of a GC analytical framework, to illustrate. 86 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ANNEX TABLE 5: ILLUSTRATIVE REPRESENTATIVE EXAMPLE OF THE GC ANALYTICAL FRAMEWORK GRAND CHALLENGE: (NAME) – MEQ 4: ACCELERATION DESIGN IMPLEMENTATION STRENGTHS WEAKNESSES STRENGTHS WEAKNESSES Acceleration strategy was to provide TA to support innovations, alongside funding itself: recognised that innovators need tailored support to prove concept or to scale. Initially this support not planned for seed grants but was included in the end. Tailored TA provided to all innovators on the basis of a needs assessment: accelerate phase (6months) after initial acceleration week. KII evidence that managing the relationship with two suppliers of TA and streamlining provision was difficult Innovators can connect and learn from each other and from experts across different disciplines. Positive feedback through surveys. Budget does not extend to one-on-one support for Seed grants. OVERALL DESIGN RAG RATING: OVERALL IMPLEMENTATION RAG RATING: Example: GREEN: Good appreciation of the need for innovator support from the outset and it evolved Example: AMBER: Good package of support provided – high satisfaction rates - but limits to tailored support for some The GC analytical frameworks were in turn complemented by the compilation of MEQ and SEQ analytical frameworks. These analytical frameworks compiled all of the synthesized findings and evidence collected and captured from all of the data sources listed in Section 3.1 for each GC and organized them by the MEQs, SEQs, and FLQs on which they came to bear. This ensured that evaluation team members had a full set of data and evidence located in a single document to refer to while drafting the findings in Section 4 of this report. Individual analyses were brought together in an internal insights workshop using the Miro virtual whiteboard platform in order to share findings and preliminary conclusions between evaluation team members and across GCs. Similar to the MEQ and SEQ analytical frameworks, the workshop was structured by MEQ, meaning that the findings and evidence from the review of GC documentation could be cross-referenced against findings from other GC reviews. This enabled the evaluation team to validate and triangulate common themes and findings, to draw connections across GCs, and to identify common areas for improvement. A co-creation workshop conducted in partnership with Catalyst was also convened in order to articulate a first draft of the evaluation’s conclusions and recommendations. The purpose of the workshop was to ensure that the recommendations were appropriately utilization-focused and to receive early feedback on the content of the conclusions and recommendations so as to tailor them to USAID’s needs. A3.3 LIMITATIONS While the evaluation team has gone to great lengths to ensure that all of its findings, conclusions, and recommendations are evidence-based, there have nevertheless been limitations on the extent of the conclusions that they have been able draw. Some of these limitations are the result of unavoidable information gaps. Similarly, interviewees are able to relay only the information they can recall, a fact that betrays both an amount of recall and selection bias and the long period of time over which GC have been implemented. The evaluation team has also encountered significant data gaps during portfolio mapping and quantitative analysis exercises. Catalyzed funding figures, for instance, are only incompletely available across the GCs since catalyzation was not an explicit objective for a number of GCs such as EBOLA and Zika. There have also been gaps in the monitoring of grantee and applicant characteristics across the GC portfolio, leading to an incomplete picture, most significantly, of: • LMIC applicants. • Grantee and applicant organization types. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 87 • First-time grantees and applicants. • Women grantees and applicants. The evaluation team has included appropriate qualifications and caveats where necessary to reflect these limitations, as ensuring that claims are evidence-based is any evaluation’s first virtue. The evaluation team has endeavored to fill in gaps where they exist through other means where possible. For instance, data found through web trawls has been used to determine procurement lags across the GC portfolio (see Section 4.6.1), the end dates for many grantee funding periods were sourced through an extensive review of grantee-level contract documentation, and two rounds of follow￾up emails have been sent to comparator program PoCs to request additional information where requested by USAID. 88 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ANNEX 4 EVALUATION MATRIX Please click here to access the evaluation matrix. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 89 ANNEX 5 ANALYTICAL FRAMEWORK Please click here to access the analytical framework. 90 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ANNEX 6 EVALUATION TOOLS A6.1 KII INTRODUCTORY EMAIL My name is [INSERT NAME OF INTERVIEWER]. I am part of a team which is conducting a meta￾evaluation of ten of USAID’s Grand Challenges for Development (GCs). GCs are multi-year partnership platforms that focus global attention and resources on specific international development problems. They promote innovative approaches and solutions to solving these problems by sourcing new solutions, testing new ideas, and scaling what works. Purpose and benefit of the evaluation This meta-evaluation will help USAID to understand the extent to which the Grand Challenge model has been an appropriate method for achieving sustainable results at scale and strengthening innovation ecosystems. It will also consider the extent to which the implementation of the Grand Challenge approach has led to effective achievement of development outcomes. The findings from this meta-evaluation will generate actionable recommendations that can help USAID and its partners to guide future investment decisions and advance the design and management of GCs. We would like to have a discussion with you in which you share your insights about your engagement with [INSERT NAME OF GC]. We will ask you about topics such as the results and impact of the GC program and how these were measured, cost effectiveness, scaling results achieved and pathways to scale (including scaling support), governance and partnership models, and catalyzing of further funds. This is a semi-structured interview, so while we will be asking broadly the same questions to everyone we interview, we will also ask you follow-up questions on the fly. Evaluation ethics The information you provide during this discussion will be treated with strict confidentiality and what you have told us will not be revealed to others besides the people conducting the evaluation. The information will be anonymous and your anonymity will be guaranteed. Further information on how your data will be used is provided in the consent form. Your views and perspectives are important to us, and we thank you for your time in participating in this interview. A6.2 KII CONSENT FORM For facilitator to draft and interviewee to complete: Relevant GC: Name of person interviewed: Role of person interviewed: Name of organization: City and country: ……………………; Name of interviewer: Date of interview: .….. / ….. / ……….; Duration of interview: Use of data What will we do with the data? The data collected from this interview will be used for the evaluation. Your answers will be treated confidentially, and what you have told us will not be revealed to others besides the people conducting the evaluation. The information will be anonymous, and your anonymity will be guaranteed outside the evaluation team. Your name will be included in any report annex which includes a list of interviewees (unless you request otherwise) but your name will not appear against interview information presented in any report: it will not be possible for any reader to link the information you provide us to you as a person. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 91 Who will your data be shared with? This data will only be used by the evaluation team. USAID has a right to raw data but will be anonymized in the rare event that it is requested. How long will we keep your data? We will keep it in a form that can identify you only for the duration of the evaluation, after which all personal data will be anonymized. We will store the interview notes in a secure corporate system in compliance with USAID and EU data protection regulations. We will retain the data for up to five years after the end of the evaluation. Do I have to do this interview? No, it is a voluntary interview and we will consider your participation and completion of the interview as consent to use your data. You may refuse to answer any question or to withdraw your consent at any time. You may also request that the interviewer a) does not record your name or b) records your name but does not include it in any report (e.g., in a list of interviewees). CONSENT – for interviewee to complete: I fully understand what this interview is about and I agree to take part in this interview. I give permission to record the interview with an audio-recorder and for notes to be taken. _______________________________ _________________ (Interviewee’s signature) Date 92 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION A6.3 KII TOPIC GUIDES A6.3.1  TOPIC GUIDE 1: USAID / PARTNER GC POC DISCUSSIONS This KII is intended for those USAID or partner staff who are/were involved in the design, management and day to day operations of the GC. Please note: During the interview this guide should be used selectively as a flexible tool to verify and fill in data gaps from desk research i.e., it is not expected that each PoC will be asked every question below. Additional probing questions are provided to support further discussion as required. Where necessary, interviews can be split into two separate sessions to ensure convenience and avoid interviewee fatigue. Numbers in brackets (4) or (4.1) refer to MEQs and SEQs respectively addressed by each question. N.B. Interview introductory text and written consent form emailed to participant in advance of interview (see separate document). GENERAL GUIDANCE: • Introduce yourself and anyone else on the evaluation team attending the interview. • Briefly explain the purpose of the interview – as set out in the previously sent introductory email. • Explain that: – they are free to leave the interview process at any time; and that, – when findings are presented, all evaluation participants will be anonymised so that, as much as is possible, specific findings and comments cannot be traced back to individuals. • Ask participant for verbal consent, if not already sent their written consent in advance, and to take notes. • Audio record the interview with permission of the interviewee if possible. GENERAL INFORMATION: • Full name and Job title (and time in this role): • Point of Contact(s) for which GC/s and time in this role (if different): MEQ 1: ACHIEVEMENT OF RESULTS AND FACTORS AFFECTING THESE • Opening question to get into the interview: In your own words, please can you tell me what the objectives of the GC were? (1.1, 1.2) – Prompt/ if further clarity required: The objectives of the GC are stated to be xxx (taken from x doc in desk review). Do you agree that these were the original objectives of the GC as set out at design stage? If not, please clarify the original objectives of the GC. – To what extent was the GC designed and delivered in response to local, national and global development priorities? What mechanisms or considerations were in place to ensure this, if any? • Where no clear intervention logic from documentation: Do you think that a clear intervention logic (or ToC) (with clear targets), was set out at the start of the GC which indicated how the objectives and targets would be achieved (i.e., what kind of resources and investments would be required?) (1.1, 1.2, F 1.3) (Probe/ further questions: if yes: did this intervention logic accurately portray what the GC intended to achieve? Did the intervention logic hold true (i.e., that if x and y happened, and certain assumptions held true, z was achieved)? If no: do you think that the lack of an intervention logic has had any impact (positive or negative) on how the GC has been implemented?) • Where there is a clear intervention logic from documentation: Do you feel that the intervention logic for the challenge set out in XX document accurately portrays what the challenge intended to achieve and how? (Probe/ further questions: if yes, did the intervention logic hold true (i.e., that if x and y happened, and certain assumptions held true, z was achieved)? If no: has this had any impact (positive or negative) on how the GC has been implemented?) In your view, was it clear to you/others designing the GC what kind of resources and investments would be required to achieve these objectives? (1.1, 1.2). Please explain. • Was the intervention logic/ ToC reviewed / adapted over the life of the program? • Do you feel that the selection of a GC was the best mechanism to use to meet this particular development challenge? If so – why? If not – why not? And which instrument/approach do you think would be better? • To what extent do you feel that the setting of a clear program design/objectives has supported or constrained the results that have been achieved? • As you look back at the origins of the GC, how effective was the design? – If you were starting again today, would you do things differently, and if so, what changes would you make? USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 93 – To what extent were considerations around gender and social inclusion incorporated into the design of the program? (e.g., call for proposals, guidelines, applications, selection criteria – ask as appropriate) • What specific outcomes have been achieved at the GC level? (1.1, 1.2) – Has the GC achieved its overall objectives / is it on track to achieve objectives in terms of both outcomes and impact? (1.1, 1.2) – To what extent have the results achieved been differentiated by gender and other social categories? (e.g., tracking # of male/female/ PWD etc reached through the GC) – How and to what extent have the solutions supported by the GC contributed to better outcomes for marginalised groups, if any? (e.g., women, girls, people with disabilities, people from vulnerable groups etc.) • To what extent was sustainability considered at the outset of the GC? (Further probe as necessary: in your view, how sustainable have innovations been? Were they able to mobilise additional funding etc?) • Have there been unintended positive or negative impacts of the fund so far? If known: What evidence is there for this? What were the reasons for these? (1.1, 1.2) • What has been your role in the selection process? (F1.3) – Do you feel that the selection processes/ investment criteria for the GC were appropriate to support the achievement of its objectives? (Further probe as necessary: what were the key criteria?) (F1.3) – (If not evident from desk review: What were the planned stages of innovation to be supported by the GC? e.g. early stage/ start up; proof of concept/ piloting/ prototyping; testing and transitioning to scale; or expanding/ scaling (explain stages as required) Then: Do you feel that the types of innovation supported by your GC (early stage/middle/late stage innovations – tailor as appropriate) were the right types, with respect to achieving the original fund objectives? (1.3, F 1.4) • GCs often aspire to create systemic change in markets, or in policies and behaviors. How do you define systemic change with respect to your GC? • To what extent are grantees (and thus the portfolio of projects supported by grants) contributing to systemic change (either in the market, or in government policies/legislation etc as appropriate)? • In your opinion, how can GCs be best used to address systemic challenges going forwards? (F1.1) • For comparator programs (type 2) analysis: can you suggest any comparator external challenge fund programs with similar objectives to this GC, which might add value to our analysis of effectiveness of the GC? MEQ 2: MEASURING IMPACT, RESULTS AND UPTAKE • If not evident from program docs: Do you have a program level MEL strategy which enables you to aggregate performance at GC/program level? Discuss – including all aspects of monitoring, evaluation and learning. – To what extent was this strategy/framework established at design stage vs. later in GC implementation? (Further questions/probes as relevant: If there were multiple competitions within the same GC, was there an overarching MEL framework? Were there any frameworks that were specific to some or all competitions?) – Does the program level MEL framework that you have developed enable you to monitor and report results differentiated by gender and other social categories? if so, to what extent? – Do you use ‘standard indicators’ at the program level? If so, please give examples of key indicators used. – Did the MEL strategy or framework evolve over time? What drove the changes? – Was there a data collection and management system that supported MEL? • How do grantees monitor and report their achievements at output, outcome and impact levels? – Is this disaggregated by gender and other aspects of social differentiation, including reach of marginalised groups? (e.g., women, girls, people with disabilities, other vulnerable groups etc.) • What support do you (or another provider) provide to grantees to assist with MEL? / To what extent are you involved in MEL processes (if at all)? Do you think you should have more/less involvement in MEL responsibilities? • Where appropriate: To what extent did you assess the performance of each round, adapt and feed 94 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION learning into subsequent rounds? (Further probe: could you provide us with some examples?) • In your opinion, how should success be measured across stages of innovation, particularly early￾stage innovations? (F2.3) • In your opinion, how can GCs best measure longer term outcomes and impacts, particularly those which may go beyond the lifetime of the fund/ ecosystem effects? (F2.4, F2.6) • Do you have any suggestions as to how USAID/ partners can better coordinate data collection/ MEL efforts, so as to better manage GCs moving forwards? (F2.5) MEQ 3: MEASURING COST-EFFECTIVENESS • In general, who is most engaged in conducting the analysis of costs and outcomes/effectiveness: USAID, GC manager, independent evaluator, researcher? Who initiates the process if/when it is conducted? • To what extent do the program design documents consider the relationship between the overall budget for the GC and the outcomes? What elements of costs are usually considered at the design stage? • Prompts: – Administrative costs of running the challenge? Monitoring and evaluation of GC? Independent/Impact evaluation? Research? – Management/Admin costs by the grantee? Distribution of costs between beneficiaries and support/capacity building of the grantees? • Are these cost elements considered in more detail at some later stage of the engagement? Contracting? Inception? First year? Mid-term? End? • How are different sources of funds defined and allocated: USAID; other donor; grantee? – Is there a common understanding of the distinction between leveraging funds from other donors and catalyzing funds/cost sharing from the grantee? • On the output side, is there a common understanding of who are the direct and indirect beneficiaries of the GC? • Is any quant/qual research undertaken to measure the ecosystem effects of the GC? • What reporting and by whom is conducted on the performance of the GC in achieving its outcomes? How is this linked to a theory of change/logframe/performance matrix? MEQ 4: SCALING INNOVATIONS • With respect to XX GC, how is scale defined, if at all? Was this definition of scale created at the outset/design of the program, or is it something that has become clearer over time? (4.1, F 4.1) • Were pathways to scale clearly defined in the design of the GC, and (where relevant) in the design of specific funding rounds? (Further probe: to what extent they were, or were not, as necessary) (4.1, F4.1) • To what extent was ‘support for scaling’ incorporated in the original program objectives/ design logic, if at all? Was the GC set up well for this (in terms of governance/management/ partnership arrangements)? • In your view, was the type of grant management/ TA support provided (either internal or external) appropriate to support the scaling required to achieve the GC objectives? In your view, how can you best support grantees to achieve scale and systemic impact? • What mechanisms do you use to collect evidence on innovations reaching scale? – How were scaling results monitored and reported on? – What evidence is there that innovations supported through the GC have / will reach scale? Could you share some examples with us? • What are some of the key learnings from this? In your opinion, are there any common characteristics that you can highlight for innovations that tend to successfully reach scale? (4.2) • For case study selection: can you suggest any particular innovators/projects that achieved considerable success (or challenges) around scaling results, whom we may wish to follow up with in more depth? MEQ 8: ACCELERATION • What does ‘acceleration’ mean in the case of this specific GC? (i.e., is it just to do with scaling solutions, or is it also about organizational/enterprise development that allows for an easier pathway to further donor or private investment? i.e., goes beyond scaling to benefits in the wider ecosystem). • Was there a clear strategy for acceleration at design stage? Was there sufficient budget for this? • In your view, have the acceleration strategies aligned with the GC’s scaling objectives? USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 95 • Do you feel that the type of TA acceleration support provided (i.e., in-house or external) was appropriate for achieving the program’s acceleration/scaling objectives? (8.1) Similarly, do you feel that the intensity of acceleration support provided was pitched at the ‘appropriate level’? • If you could re-run the GC again, what might you do differently (if anything) with regards to acceleration support/strategies? (F8.1) MEQ 5: INVESTING IN ECOSYSTEMS • How is the ‘ecosystem’ defined for your GC? (If no clear definition: share our working definition for reference – see footnote106 below.) • To what extent was the need to engage in the innovation ecosystem considered in the design of the GC? Has this influenced how well the program has done? • When determining the scaling objectives of the program, do you feel that there was sufficient/ appropriate consideration of the need for ecosystem strengthening support? (Probe: If yes, what were the key considerations in systems strengthening? If not, what was missed?) • To what extent did the program invest in ecosystems strengthening? E.g. How did the GC support ecosystem change/development of enabling environments (links to scale)? What has been the result of this? • How could this be improved going forwards? (F5.1) • Where relevant: To what extent was the GC able to measure the effectiveness of ecosystem support provided (if any)? What were the mechanisms to do so? • As relevant: based on your experiences with XX GC, which types of innovator and stages of innovation may best benefit/benefit most from ecosystem strengthening support e.g., brokering partnerships/linking innovators to investors/ linking to govt/supply chains, influencing policy/ regulation? (5.1) MEQ 6: ENGAGING USAID AND PARTNER MISSIONS AND OUS • To what extent was the engagement of USAID and partner missions or other OUs considered at the program design stage? 106 Whilst we recognize there is no single common definition of ‘ecosystems’ within USAID or across the GCs, for the purpose of this evaluation we have taken the ‘ecosystem’ to be what is happening within the wider context within which the USAID-supported innovator operates. Depending on the nature of the specific GC, this might include the actions of other donors, private sector and market actors, government, regulators, civil society and other actors in the wider operating context. • To what extent did the program effectively engage with missions or other OUs? E.g., engagement in design of calls, launch, selection processes. How was this reflected in management/governance arrangements? (Further probe: How was the GC affected by the nature of engagement with USAID and partner missions?) • What lessons can be learnt from the engagement with Missions and other OUs? Is there scope for GCs to be better integrated with USAID and GC partner Mission programing? (F6.1) MEQ 7: GOVERNANCE, PARTNERSHIP MODELS & OPERATIONAL MODELS • What type of partnership development processes did you go through for this GC? With who, what nature etc? (Prompts: Funding partners; management partners e.g., separate partners for acceleration/ TA/fund management support; may also include partnerships with innovators). • If not clear from desk review: please briefly explain how the GC was governed and managed? (Prompts: USAID’s role? Partnerships with/roles of other donors? How funding partners were engaged in governance of the GC? Governing bodies/committees across donors? Management partners e.g., separate partners for acceleration, fund management etc.?) • Where relevant/ more than 1 partner involved: how did you work to ensure the best use of each partner’s comparative advantages to ensure the best outcomes for the program? • To what extent did the program design, governance and partnerships models align with the program intervention logic? • Do you think that the program governance, partnership and management models supported the effective design of calls, launch, award selection? Please elaborate. • Do you feel that the way in which the GC has been governed and managed has contributed to (or hindered) the overall achievement of intended objectives? What could have been done differently? (7.1, 7.2) – Where relevant: how involved has USAID/ xx other partner been in the ongoing management of the program? – Has the type of (e.g., external vs in house) and intensity of grant management support 96 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION provided been appropriate to support the achievements of the GC? (7.3) • Do you have any recommendations around how best to form and manage partnerships for GCs in the future? (F7.1, 7.2) MEQ 9: PROCUREMENT & REDUCING BARRIERS TO FUNDING FOR NON￾TRADITIONAL PARTNERS • If unclear from desk review: what type of organizations/innovators were eligible for funding from the program? • To what extent did you consider the diversity of potential innovators at design stage? • Was the GC accessible to ‘non-traditional partners’ i.e., those who had not previously accessed USAID funding in the last 5 years? Was it accessible to local partners? • Do you feel that the GC was accessible to the right/appropriate type of organization/innovator, required to achieve the objectives of the GC? i.e., were you successful in recruiting the types of innovators that you sought to reach? • To what extent did the allocations for grant funding consider the range of size of grants offered in relation to the types of organizations supported (e.g., large enough to attract interest, or proportionate to capacity of the organization type/size)? • Check what is available via desk research and use to anchor this question where possible. How did you reach these organizations? i.e., what outreach/ marketing strategies were used at launch? Selection processes? Engagement of USAID Missions/partners? • To what extent were aspects of GESI considered? e.g., encouraging applications from women￾owned small businesses/women-led or locally led organizations.? • Based on lessons learnt to date, what could be done differently to reach non-traditional partners? MEQ 10: CATALYTIC EFFECTS • In your view, have the individual innovators been able to catalyze additional funding for their innovation, as a direct or indirect result of taking part in the GC, compared to if they had taken part in another grant funding mechanism? (NB, Direct result = related to acceleration or TA support provided directly from the GC). (10.1) (Further prompts: To what extent? Was the additional funding generated from USAID or another funding source? What types of innovators have been able to do this? What additional support does it take from the GC, if any? What factors separate innovations with catalytic effects from innovations without it?) • At the program level, how did you attract funds from other sources, leverage additional donor funds and work to raise awareness of the GC? (10.2) Do you have any thoughts on how you might have done this differently (F10.1)? • How well do you think the GC model supports the raising of awareness about the development challenges addressed by the GC, and in which audiences? • How well do you think the GC model supports the raising of awareness about evolving solutions to the challenges? • In your view, would this program have been possible / achieved the same level of catalytic effects using a traditional TA program in place of a challenge fund? 6.3.2 TOPIC GUIDE 2: USAID GC SENIOR LEADERSHIP This guide is intended for USAID GC Senior Leadership/ Advisors, including those who may have specific insights on cost effectiveness analysis for GCs. Please note: During the interview this guide should be used selectively as a flexible tool to verify and fill in data gaps from desk research. i.e., it is not expected that each participant will be asked every question below. Numbers in brackets (4) or (4.1) refer to MEQs and SEQs respectively addressed by each question. N.B. Interview introductory text and written consent form emailed to participant in advance of interview (see separate document). N.B. Interview introductory text and written consent form emailed to participant in advance of interview (see separate document). GENERAL GUIDANCE: • Introduce yourself and anyone else on the evaluation team attending the interview. • Briefly explain the purpose of the interview – as set out in the pre-sent introductory email. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 97 • Explain that: – they are free to leave the interview process at any time; and that, – when findings are presented, all evaluation participants will be anonymized so that, as much as is possible, specific findings and comments cannot be traced back to individuals. • Ask participant for verbal consent, if not already sent their written consent in advance, and to take notes. • Audio record the interview with permission of the interviewee if possible. GENERAL INFORMATION: • Full name and Job title (and time in this role): • Role/involvement in which GC/s and time in this role (if different): MEQ 1: ACHIEVEMENT OF RESULTS AND FACTORS AFFECTING THESE: HIGH LEVEL QUESTIONS ACROSS THE PORTFOLIO OF GCS (I.E., NOT PROGRAM SPECIFIC QUESTIONS) • Looking across the portfolio of GCs… To what extent do you feel that the setting of clear program design objectives supports or constraints what results are eventually achieved? • In your view, looking across the portfolio, to what extent is sufficient attention given to setting up a program’s ToC or intervention/design logic, during the design stages? Are these sufficiently reviewed and reflected upon during implementation to inform ongoing changes to management practices? • To what extent are considerations around gender and social inclusion incorporated into the design of programs (e.g., call for proposals, guidelines for applicants and selection criteria)? Do you have any suggestions around how this might be strengthened in future programing? • Looking across the portfolio of GCs, what types of outcomes have been achieved? Which GCs do you think have demonstrated differentiated results by gender and other social categories? Do you feel the solutions supported by the GCs contributed to better outcomes for marginalized groups? (e.g., women, girls, people with disabilities, other vulnerable groups etc.) • What types of scaling results have you seen/ stick out? • In your view, to what extent is sustainability and / or scale considered during the GC design stage? (Ask to give GC specific answers if this makes more sense) • Can you see any patterns across the portfolio in terms of the sectors or contexts that the GCs work in, and which of these a GC model may be better positioned to achieve success in compared to traditional models of development? • As someone who sits in the USAID GC leadership/advisory team, looking across the portfolio of GCs, what has your role been in the design of individual GCs, the design of individual funding rounds, and the ongoing grant management processes (e.g., launch, selection processes, MEL etc.)? Do you feel this was an ‘appropriate’ level of involvement? • In your view, to what extent are grantees (and programs) contributing towards systemic change (either in the market, or in government policies/ legislation, as appropriate)? • In your view, are there any examples of specific GCs where, on reflection of achievements to date, you feel another type of development approach/ funding instrument could have been used or yielded better results? • For comparator programs (type 2) analysis: can you suggest any comparator external challenge fund programs with similar objectives to the GCs we are reviewing, which might add value to our analysis of effectiveness of the GCs? MEQ 3: MEASURING COST EFFECTIVENESS – QUESTIONS SPECIFICALLY FOR USAID ADVISORS AND THOSE USAID PERSONNEL RESPONSIBLE FOR LEADING ON CEA • In general, who should take responsibility for conducting the analysis of costs and outcomes/ effectiveness: USAID, GC manager, independent evaluator, researcher, someone else? Looking ahead, what needs to change in the future? • What elements of costs and benefits are generally considered in the program design documents? What elements of costs are usually considered at the design stage? • Do you think this should change in the future? If so, how should this change with regard to considering the relationship between the overall budget for the GC and the measurement of outcomes? 98 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION • When should this analysis be conducted/ established: Contracting? Inception? First year? Mid-term? End? On-going or ex post? • Should there be standard cost definitions and budget formats? How feasible is it to do this? • How are different sources of funds defined and allocated: USAID; other donors; grantee? – Is there a common understanding of the distinction between leveraging funds from other donors and catalyzing funds/cost sharing from the grantee? • On the output side is there a common understanding of who are the direct and indirect beneficiaries of the GC? • Should any quant/qual research be undertaken to measure the ecosystem effects of the challenge? • Do you favor developing USAID’s list of standard outcome indicators and for GCs to adhere to this? • Would you favor some standardized reporting on measuring the performance of the GC in achieving its outcomes, and linking cost effectiveness to a theory of change/logframe/ performance matrix? MEQ 4, 8 & 5: SCALING INNOVATIONS, ACCELERATION & ECOSYSTEM INVESTMENT • In your view, across the GCs, what are the most appropriate types of grant management/TA support (e.g., either internal or external) that best support the scaling or acceleration required to achieve GC objectives? In your view, how can the fund manager best support grantees to achieve scale and systemic impact? • In your opinion, looking across the GCs, are there any common characteristics that you can highlight for innovations that tend to successfully reach scale? (4.2) • Looking forwards, do you have any suggestions with regards to improving acceleration & scaling support? (F8.1) • In your opinion, when determining the scaling objectives of a program, do you feel that sufficient consideration is given to the need for ecosystem support strengthening? (MEQ 5) MEQ 6: ENGAGING USAID AND PARTNER MISSIONS AND OUS • What lessons can be learnt from the engagement with USAID and partner Missions and other OUs across the portfolio of GCs? Is there scope for GCs to be better integrated with USAID and GC partner Mission programing? (F6.1) MEQ 7: GOVERNANCE, PARTNERSHIP & OPERATIONAL MODELS • What type of partnership development processes have been used on the various GCs you have been involved in? With who, what nature etc.? (Prompts: Partnerships with/roles of other donors? Funding partners & their engagement in governance and /or management of GC? Role of management partners e.g., acceleration/TA/fund management partners?) • In your view, looking across the GCs, what types of partnership & operational models work best and why? • Do you have any recommendations around how best to form and manage partnerships for GCs in the future? (F7.1, 7.2) MEQ 9: PROCUREMENT & REDUCING BARRIERS TO FUNDING FOR NON￾TRADITIONAL PARTNERS • Based on lessons learnt to date, how might you improve the accessibility of GCs for non￾traditional partners i.e., those who have not accessed USAID funding in the last 5 years? MEQ 10: CATALYTIC EFFECTS • In your view, have the individual innovators been able to generate additional funding for their innovations, as a direct or indirect result of taking part in the GC, compared to if they had taken part in another grant funding mechanism? (NB, Direct result = related to acceleration or TA support provided directly from the GC). (10.1) (Further prompts: To what extent? Was the additional funding generated from USAID or another funding source? What types of innovators have been able to do this? What additional support does it take from the GC, if any? What factors separate innovations with catalytic effects from innovations without it?) • How well do you think the GC model supports the raising of awareness about the development challenges addressed by the GC, and in which audiences? • How well do you think the GC model supports the raising of awareness about evolving solutions to the challenges? USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 99 A6.3.3  TOPIC GUIDE 3: EXTERNAL STAKEHOLDERS (EVALUATION PROVIDERS AND DONOR PARTNERS) • This topic guide is for stakeholders external to USAID but with a role/interest in a specific GC, primarily (but not necessarily limited to) evaluation providers and donor partners.107 Where questions are specific to only one of these two stakeholder groups, this is indicated. Questions should be adapted to GC-specific external stakeholders as appropriate. • Please note: During the interview this guide should be used selectively as a flexible tool to verify and fill in data gaps from desk research i.e., it is not expected that each participant will be asked every question below. Numbers in brackets (4) or (4.1) refer to MEQs and SEQs respectively addressed by each question. N.B. Interview introductory text and written consent form emailed to participant in advance of interview (see separate document). • N.B. Interview introductory text and written consent form emailed to participant in advance of interview (see separate document). GENERAL GUIDANCE: • Introduce yourself and anyone else on the evaluation team attending the interview. • Briefly explain the purpose of the interview – as set out in the pre-sent introductory email. • Explain that: – they are free to leave the interview process at any time; and that, – when findings are presented, all evaluation participants will be anonymized so that, as much as is possible, specific findings and comments cannot be traced back to individuals. • Ask participant for verbal consent, if not already sent their written consent in advance, and to take notes. • Audio record the interview with permission of the interviewee if possible. GENERAL INFORMATION: • Full name and Job title (and time in this role): • Engagement (level and type) in which GC/s and time in this role (if different) i.e., how were you involved with this particular GC? 107 It is recognized that donor partners’ roles vary significantly on the GCs, with some playing only a light touch role as a ‘funding partner’ and others playing a more significant management role. As such, questions will be tailored as appropriate for each GC. MEQ 1: ACHIEVEMENT OF RESULTS AND FACTORS AFFECTING THESE • Opening question to get into the interview: Based on your understanding of the GC and in your own words, please can you outline the original objectives of the program as set out at design stage? (1.1,1.2) – Prompt/ if further clarity required: The objectives of the GC are stated to be xxx (taken from x doc in desk review). Do you agree that these were the original objectives of the GC as set out at design stage? If not, please clarify the original objectives of the GC. • To what extent was the GC designed and delivered so that it responded, and was relevant, to local and national development priorities? For donors only: how involved and able to influence were you in the design of the program? To your knowledge, to what extent were relevant USAID or partner Missions based in-country engaged in the design of the program? • What do you see as the intervention logic or ToC of the program? E.g., if we do x, it will lead to y, with these underlying assumptions? • For evaluators: in your opinion, do you think that a clear intervention logic or ToC (with clear targets), was set out at the start of the GC which indicated how objectives/targets would be achieved (i.e., what kind of resources/investments would be required)? Did you review the ToC during your evaluation? Did it hold true (i.e., that if z and y happened, and certain assumptions held true, z was achieved)? Was the ToC adapted over the life of the program? (1.1, 1.2, F1.3) • Why was a GC approach considered the best mechanism to meet this particular development challenge? Did it meet expectations? - why? why not? Do you think that a different instrument/ approach would be better? • To what extent do you feel that the setting of a clear program design/objectives has supported or constrained the results that have been achieved? • As you look back at the origins of the GC, how effective was the design? – If you were running the GC yourself, would you do things differently, and if so, what changes would you make? – Likely for evaluators only: To what extent were considerations around gender and social 100 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION inclusion incorporated into the design of the GC (e.g., call for proposals, guidelines for applicants and selection criteria)? • In your view, what specific outcomes would you say have been achieved at the GC level? (1.1, 1.2) – Has the GC achieved its overall objectives / is it on track to achieve objectives in terms of both outcomes and impact? (1.1, 1.2) – To what extent have the results or outcomes achieved been differentiated by gender and other social categories? (e.g., tracking # of male/ female/PWD etc. reached through the GC). – Do you think the solutions supported by the GC contributed to better outcomes for marginalized groups? If so, to what extent? (e.g., women, girls, people with disabilities, other vulnerable groups etc.). • As far as you know, to what extent was sustainability considered at the outset of the GC? (Further probe as necessary: i.e., how sustainable have innovations been? Were they able to mobilize additional funding etc.? • For evaluators only: Have there been unintended positive or negative impacts of the fund so far? What evidence is there for this? If known, what were the reasons for these? (1.1, 1.2) • For donor partners only: What has been your role in the selection process? (F1.3) – For both: Do you feel that the selection processes/investment criteria for the GC were appropriate to support the achievement of its objectives? (Further probe as necessary: what were the key criteria?) – Do you feel that the types of innovation supported by the GC (early stage/middle/ late-stage innovations – tailor as appropriate) were the right types, with respect to trying to achieve the original GC objectives? (1.3, F 1.4) • GCs often aspire to create systemic change in markets, or in policies and behaviors. – To what extent do you feel that grantees (and thus the portfolio of projects supported by the grants) are contributing to systemic change (either in the market, or in government policies/legislation etc. as appropriate)? – Do you have any views or suggestions on how GCs can be best used to address systemic challenges going forwards? (F1.1) MEQ 2: MEASURING IMPACT, RESULTS AND UPTAKE • Are you aware of a program level MEL strategy which enables aggregation of performance at GC/ program level? Discuss – including all aspects of monitoring, evaluation and learning as appropriate. – As far as you know, to what extent was this strategy/framework established at design stage vs. later in GC implementation? (Further questions/probes as relevant: If there were multiple competitions within the same GC, was there an overarching MEL framework? Were there any frameworks that were specific to some or all competitions?) – Does the program level MEL framework enable you to monitor/understand results differentiated by gender and other social categories? If so, to what extent? (e.g., women, girls, people with disabilities, other vulnerable groups etc.). – Does the GC use ‘standard indicators’ at the program level? If so, can you highlight any examples of key indicators used. – Did the MEL strategy or framework evolve over time? What drove the changes? – Was there a data collection and management system that supported MEL? • For donor partners: To what extent were you involved in MEL processes (if at all)? • For evaluation providers only: How did grantees monitor and report their achievements at output, outcome and impact levels? – Is this disaggregated by gender and other aspects of social differentiation, including reach of marginalized groups? (e.g., women, girls, people with disabilities, other vulnerable groups etc.). • For evaluation providers: What type of MEL support did grantees receive? • For evaluation providers: To what extent did you feel that the GC manager assessed the performance of each round, adapted and fed learning into subsequent rounds? (Further probe: could you provide us with some examples?) • For evaluation providers: In your opinion, how should success be measured across stages of innovation, particularly early-stage innovations? (F2.3) USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 101 • In your opinion, how can GCs best measure longer term outcomes and impacts, particularly those which may go beyond the lifetime of the GC/ ecosystem effects? (F2.4, F2.6) • Do you have any suggestions as to how USAID/ partners can better coordinate data collection/ MEL efforts, so as to better manage GCs moving forwards? (F2.5) MEQ 3: MEASURING COST EFFECTIVENESS • Donors/Evaluation partners. In general, who should take responsibility for conducting the analysis of costs and outcomes/effectiveness: USAID, GC manager, independent evaluator, researcher? • Evaluation partner: To what extent have you been engaged in measurement of cost effectiveness? If not, was this done by others? If yes, what was your brief? • Donors: How does the analysis of cost effectiveness compare with other donor partners that you are engaged with? • Donors/Evaluation partners: Looking ahead, what needs to change in the future? • What elements of costs and benefits should be considered at each stage of the project cycle? Should an ex-ante cost effectiveness analysis be undertaken? At what other points in the project cycle is this important? On contract? Inception? First year? Mid-term? End? On-going or ex post? • How should this change in the future with regard to considering the relationship between the overall budget for the GC and the measurement of outcomes? • Should there be standard cost definitions and budget formats? How feasible is it to do this? • How are different sources of funds defined and allocated: USAID; other donors; grantee? – Is there a common understanding of the distinction between leveraging funds from other donors and catalyzing funds/cost sharing from the grantee? • On the output side, should there be a common definition of who are the direct and indirect beneficiaries of the GC? • Should any quant/qual research be undertaken to measure the ecosystem effects of the GC? • Do you favor developing USAID’s list of standard outcome indicators and for GCs to adhere to this? • Would you favor some standardized reporting on measuring the performance of the GCs in achieving its outcomes, and linking cost effectiveness to a theory of change/logframe/ performance matrix? MEQ 4: SCALING INNOVATIONS • For donor/evaluator as/if appropriate: Based on your understanding of the program, how was scale defined in XX GC, if at all? Was this definition of scale created at the outset/design of the program, or is it something that has become clearer over time? (4.1, F 4.1) • For donor/evaluator as/if appropriate: Were pathways to scale clearly defined in the design of the GC, and (where relevant) in the design of specific funding rounds? (Further probe: to what extent they were, or were not, as necessary) (4.1, F4.1) • To what extent was ‘support for scaling’ incorporated in the original program objectives/ design logic, if at all? (NB. Scaling support might include networking & partnership engagement, development opportunities – connections with investors, govt, private sector etc.). Was the program set up well for this (in terms of governance/ management/partnership arrangements)? • For evaluation provider only: Was the type of grant management/TA support provided (either internal or external) appropriate to support the scaling required to achieve the GC objectives? – In your view, how can grantees be best supported to achieve scale and systemic impact? • What evidence is there that innovations supported through the GC have / will reach scale? • For evaluation provider only: How were scaling results monitored and reported on? • What are some of the key learnings in terms of innovations reaching scale? In your opinion, do you think there are any common characteristics amongst those innovations that tend to successfully reach scale? (4.2) • Evaluators only: For case study selection: can you suggest any particular innovators/projects that achieved considerable success (or challenges) around scaling results, whom we may wish to follow up with in more depth? MEQ 8: ACCELERATION • In your view, what does ‘acceleration’ mean in the case of this specific GC? (i.e., is it just to do with scaling solutions, or is it also about organizational/ 102 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION enterprise development that allows for an easier pathway to further donor or private investment? i.e., goes beyond scaling to benefits in the wider ecosystem). • Was there a clear strategy for acceleration at the design stage? Was there sufficient budget for this? • In your view, have the acceleration strategies aligned with the GC’s scaling objectives? • Do you feel that the type of TA acceleration support provided (i.e., in-house or external) was appropriate for achieving the program’s acceleration objectives? (8.1) Similarly, do you feel that the intensity of acceleration support provided was pitched at the ‘appropriate level’? • If you could re-run the GC again/ or were providing advice to a new fund manager, what might you suggest is done differently with regards to acceleration support/strategies? (F8.1) MEQ 5: INVESTING IN ECOSYSTEMS • How is/was the ‘innovation ecosystem’ defined for the GC that you are involved with? (If no clear definition: share our working definition for reference – see footnote108 below.) • Likely for evaluators only: In your view, to what extent was the need to engage in the innovation ecosystem considered in the design of the GC? Do you think this has influenced how well the program has done? • Likely for evaluators only: When determining the scaling objectives of the program, do you feel that there was sufficient/appropriate consideration of the need for ecosystem strengthening support? (Probe: If yes, what were the key considerations in systems strengthening? If not, what was missed?) • Likely for evaluators only: How effectively did the program invest in ecosystems strengthening? E.g., did the program provide support for ecosystem change/development of the enabling environment, and how successful do you think this was? What was the result of this? Do you have any suggestions as to how this could be improved going forwards/in a future similar fund? (F5.1) • As relevant: based on your experiences with XX program, do you have any thoughts on which types of innovators and stages of innovation may best benefit/ benefit most from ecosystem strengthening support e.g., brokering 108 While we recognize there is no single common definition of ‘ecosystems’ within USAID or across the GCs, for the purpose of this evaluation we have taken the ‘ecosystem’ to be what is happening within the wider context within which the USAID-supported innovator operates. Depending on the nature of the specific GC, this might include the actions of other donors, private sector and market actors, government, regulators, civil society, and other actors in the wider operating context. partnerships/linking innovators to investors/ linking to govt/supply chains, influencing policy/ regulation? (5.1) MEQ 7: GOVERNANCE, PARTNERSHIP MODELS & OPERATIONAL MODELS • For donors only: What type of partnership development processes did you go through with USAID for this GC? With who, what nature etc.? (Prompts: Partnerships with/roles of other donors? Funding partners & their engagement in governance and /or management of GC? Role of management partners e.g., acceleration/TA/fund management partners?) • If not clear from desk review: please briefly explain how the GC was governed and managed? (Prompts: USAID’s role? Partnerships with/roles of other donors? How funding partners were engaged in governance of the GC? Governing bodies/committees across donors? Management partners e.g., separate partners for acceleration, fund management etc.?) For donors only (if not already covered): and please explain your precise role in this e.g., in ongoing management of the program. • Where relevant/ more than 1 partner involved: do you think that you (or the other donors – tailor as appropriate) and USAID were able to work together to make the best use of each other’s comparative advantages, to ensure the best outcomes for the program? • Where relevant: Do you think that the budget allocations for governance/management were appropriate for the successful delivery of the program? Please elaborate. • Where relevant: Do you think that the program governance, partnership and management models supported the effective design of calls, launch, award selection? Please elaborate. • Do you feel that the way in which the GC has been governed and managed has contributed to (or hindered) the overall achievement of intended objectives? What could have been done differently? (7.1, 7.2) – Has the type (e.g., external vs in house) and intensity of grant management support provided been appropriate to support the achievements of the GC? (7.3) USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 103 • Do you have any recommendations around how best to form and manage partnerships for GCs in the future? (F7.1, 7.2) MEQ 9: PROCUREMENT & REDUCING BARRIERS TO FUNDING FOR NON￾TRADITIONAL PARTNERS • For evaluators only: From your knowledge, to what extent did the program consider the diversity of potential innovators at design stage? Roughly what proportion would you say fell into the ‘non-traditional partner’ bucket (if known)? If particularly high/low – why do you think this was? • Was the GC accessible to ‘non-traditional partners’ i.e., those who had not previously accessed USAID funding in the last 5 years? Was it accessible to local partners? Roughly what proportion would you say fell into the ‘non-traditional partner’ bucket (if known)? If particularly high/low – why do you think this was? • Do you feel that the GC was accessible to the right/appropriate type of organization/innovator, required to achieve the objectives of the fund? i.e., was it successful in recruiting the types of innovators that were sought? • For evaluators only: From your knowledge, to what extent did the allocations for grant funding consider the size of grants offered in relation to the types of organizations supported (e.g., large enough to attract interest, or proportionate to capacity of the organization type/size)? • For donors: What involvement did you have (if any) in marketing strategies, outreach at launch, or selection processes? If some involvement: do you feel that these processes were appropriate for reaching all types of innovators? • As relevant: From your knowledge, does/ did the program consider aspects of GESI e.g., encouraging applications from women-owned small businesses/women-led or locally led organizations? • In your view, how might a program be made more accessible to non-traditional partners? MEQ 10: CATALYTIC EFFECTS • Adapt question as appropriate for donors: (if not already covered): How did you come to be involved? How much additional funding have you provided? (10.2) • For donors: how have you worked to raise awareness of the GC? Do you have any thoughts on how you might have done this better? (10.2, F 10.1)? • For evaluators: How successful were the program’s efforts to raise awareness, attract funds from other sources, including leveraging additional donor funds? (10.2) • For evaluators: In your view, have the individual innovators been able to catalyze additional funding for their innovation, as a direct or indirect result of taking part in the GC, compared to if they had taken part in another grant funding mechanism? (NB, Direct result = related to acceleration or TA support provided directly from the GC). (10.1) (Further prompts: To what extent? Was the additional funding generated from USAID or another funding source? What types of innovators have been able to do this? What additional support does it take from the GC, if any? What factors separate innovations with catalytic effects from innovations without it?) • How well do you think the GC model supports the raising of awareness about the development challenges addressed by the GC, and in which audiences? • How well do you think the GC model supports the raising of awareness about evolving solutions to the challenges? • In your view, would this program have been possible / achieved the same level of catalytic effects using a traditional TA program in place of a challenge fund? (10.1) A6.3.4  TOPIC GUIDE 4: TECHNICAL ASSISTANCE PROVIDERS It is assumed that the majority of TA providers will provide TA in relation to scaling and acceleration support. As a result, question topics in this guide are focused on these specific areas. This should be tailored by the evaluation lead as appropriate for the GC in question. Please note: During the interview this guide should be used selectively as a flexible tool to verify and fill in data gaps from desk research i.e., it is not expected that each TA provider will be asked every question below. Numbers in brackets (4) or (4.1) refer to MEQs and SEQs respectively addressed by each question. N.B. Interview introductory text and written consent form emailed to participant in advance of interview (see separate document). 104 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION N.B. Interview introductory text and written consent form emailed to participant in advance of interview (see separate document). GENERAL GUIDANCE: • Introduce yourself and anyone else on the evaluation team attending the interview. • Briefly explain the purpose of the interview – as set out in the pre-sent introductory email. • Explain that: – they are free to leave the interview process at any time; and that, – when findings are presented, all evaluation participants will be anonymized so that, as much as is possible, specific findings and comments cannot be traced back to individuals. • Ask participant for verbal consent, if not already sent their written consent in advance, and to take notes. • Audio record the interview with permission of the interviewee if possible. GENERAL INFORMATION: • Full name and Job title (and time in this role): • Engagement (level and type) in which GC/s and time in this role (if different) i.e., how were you involved with this particular GC? MEQ 4: SCALING INNOVATIONS • With respect to XX GC, how is scale defined, if at all? (4.1) • If not yet covered/clear or requires more detail: What has been the exact nature of you/your organization’s involvement in working support the scaling of innovators’ work? i.e., what level of TA/capacity building support do you provide to innovators during the lifetime of a grant? Please describe in more detail the type of support provided. • If known/appropriate: As far as you’re aware, did the design of the GC include a clear definition for impact at scale, what ‘successful impact at scale’ would look like, and what pathways to scale might look like? (Further probe: to what extent they were, or were not, as necessary). (4.1, F4.1, MEQ 1) • If known/appropriate: As far as you’re aware, did the design of the GC incorporate ‘support for scaling’ into its original objectives/design logic? (4.1, F4.1, MEQ 1) • If known/appropriate: Based on your engagement to date, do you feel that the GC was set up well for incorporating ‘scaling support’ e.g., in terms of governance or management arrangements? • How do you/ USAID/other fund managers decide on the level of TA to provide? Is it standard for all grantees or tailored on a case-by-case basis? (F4.1) • In your view, was the type of TA support provided (e.g., in house/external; light touch or significant), appropriate to support the scaling required to achieve the GC objectives? (NB. Scaling support might include networking & partnership engagement, development opportunities – connections with investors, govt, private sector etc.). • If known: In your view, were the launch and selection processes appropriate for the scaling requirements of the GC? • What mechanisms do you use to collect evidence on innovations reaching scale? – How were scaling results monitored and reported on? (Further prompts: What were the challenges in doing so? What are the key learnings?) – What evidence is there that innovations supported through the fund have / will reach scale? Could you share some examples with us? • In your view, how can grantees be best supported to achieve scale and systemic impact? • What are some of the key learnings in terms of innovations reaching scale? In your opinion, are there any common characteristics that you can highlight for innovations that tend to successfully reach scale – or indeed the opposite – for failing to scale up? (4.2) (Further prompts: What sorts of interventions? What types of support mechanisms are effective?) • For case study selection: can you suggest any particular innovators/projects that achieved considerable success (or challenges) around scaling results, whom we may wish to follow up with in more depth? • If able to provide this information: what level of resources are committed to (or do you spend) providing this scaling support? MEQ 8: ACCELERATION • What does ‘acceleration’ mean in the case of this specific GC? (i.e., is it just to do with scaling solutions, or is it also about organizational/enterprise development that allows for an easier pathway to USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 105 further donor or private investment? i.e., goes beyond scaling to benefits in the wider ecosystem). • If not already covered above: Were you involved in this? i.e., acceleration support. If so, how and in what way. • If known/appropriate: Was there a clear strategy for acceleration at design stage? • In your view, have the acceleration strategies aligned with the GC’s scaling objectives? • If not already covered above: Do you feel that the type and intensity of TA acceleration support that you have provided has been appropriate for achieving the GC’s acceleration/scaling objectives? (8.1) • If you were to start again, what might you do differently (if anything) with regards to acceleration support/strategies? (F8.1) MEQ 5: INVESTING IN ECOSYSTEMS • How is the ‘innovation ecosystem’ defined for the GC that you are involved with? (If no clear definition: share our working definition for reference – see footnote109 below.) • How have you been involved in supporting ‘ecosystem change/development,’ if at all? • If appropriate: What support did you provide for ecosystem change/development of enabling environments? What has been the result of this? • Do you have any suggestions as to how this might be improved going forwards? (F5.1) • How is/was the effectiveness of this support measured? • If known/appropriate: To what extent was the need to engage in the innovation ecosystem considered in the design of the GC? Do you think this has influenced how well the GC has done? • When you think about the scaling objectives of the GC, do you feel that there was sufficient/ appropriate consideration of the need for ecosystem strengthening support? (Probe: If yes, what were the key considerations in systems strengthening? If not, what was missed?) • In your opinion, how can GCs best measure longer term outcomes and impacts, particularly those which may go beyond the lifetime of the fund/ ecosystem effects? (F2.4, F2.6) 109 While we recognize there is no single common definition of ‘ecosystems’ within USAID or across the GCs, for the purpose of this evaluation we have taken the ‘ecosystem’ to be what is happening within the wider context within which the USAID-supported innovator operates. Depending on the nature of the specific GC, this might include the actions of other donors, private sector and market actors, government, regulators, civil society, and other actors in the wider operating context. • As relevant: Based on your experiences with XX GC, which types of innovator and stages of innovation may best benefit/ benefit most from ecosystem strengthening support e.g., brokering partnerships/ linking innovators to investors/ linking to govt/supply chains, influencing policy/regulation? (5.1) MEQ 10: CATALYTIC EFFECTS – ONLY WHERE RELEVANT TO INCLUDE • Based on your experiences to date, have the individual innovators been able to catalyze additional funding for their innovation, as a direct or indirect result of taking part in the GC, compared to if they had taken part in another grant funding mechanism? (NB, Direct result = related to acceleration or TA support provided directly from the GC). (10.1) (Further prompts: To what extent? Was the additional funding generated from USAID or another funding source? What types of innovators have been able to do this? What additional support does it take from the GC, if any? What factors separate innovations with catalytic effects from innovations without it?) • How well do you think the GC model supports the raising of awareness about the development challenges addressed by the GC, and in which audiences? • How well do you think the GC model supports the raising of awareness about evolving solutions to the challenges? • In your view, would this program have been possible / achieved the same level of catalytic effects using a traditional TA program in place of a challenge fund? A6.3.5  TOPIC GUIDE 5: GC COMPARATORS This guide is intended for USAID Advisors and implementers overseeing or managing programs selected as comparators for the Grand Challenges. The guide addresses questions related to the rationale for selecting the GC mechanism for programing rather than other more ‘traditional’ mechanisms. Please note: During the interview this guide should be used selectively as a flexible tool to verify and fill in data gaps from desk research. i.e., it is not expected that each participant will be asked every question below. N.B. Interview introductory text and written consent 106 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION form emailed to participant in advance of interview (see separate document). Wherever possible, the interview will build on information sourced through desk research, but it will be necessary to check or verify where there are information gaps, or conflicting accounts. N.B. Interview introductory text and written consent form emailed to participant in advance of interview (see separate document). GENERAL GUIDANCE: • Introduce yourself and anyone else on the evaluation team attending the interview. • Briefly explain the purpose of the interview – as set out in the pre-sent introductory email. • Explain that: – they are free to leave the interview process at any time; and that, – when findings are presented, all evaluation participants will be anonymized so that, as much as is possible, specific findings and comments cannot be traced back to individuals. • Ask participant for verbal consent, if not already sent their written consent in advance, and to take notes. • Audio record the interview with permission of the interviewee if possible. GENERAL INFORMATION: • Full name and Job title (and time in this role): • Role/involvement in the programs under discussion (if different): Questions to be posed primarily to USAID Advisors on the origins, design and results of the program: QUESTION 1: WHAT ARE THE ORIGINS OF THIS PROGRAM? (VERIFY UNDERSTANDING FROM DESK RESEARCH) • When was it set up and where is it located within USAID? • What is / was the duration of the program and in what countries was it implemented? • What development problem(s) was it designed to address? • What are the specific objectives of the program? • What are the key success indicators of the program? – At output level. – At outcome level. • Has the program met its objectives? What results have been achieved? QUESTION 2: HOW WAS THE PROGRAM DESIGNED? WHAT WAS THE PROCESS FOLLOWED? • Why did you choose this particular procurement mechanism (i.e., grant based, or contracted third￾party service provider, as appropriate)? QUESTION 3: DID YOU CONSIDER ANY OTHER DELIVERY MODALITIES (E.G. CHALLENGE, COMPETITIVE SELECTION, CO-FINANCING ETC.) WHEN DESIGNING THE PROGRAM? • If so, what were they, and why were they rejected? • Did you consider using a Grand Challenge for this program? If so, why did you ultimately decide not to use this approach? • Under what circumstances would a Grand Challenge have been an appropriate format to achieve the objectives you set for yourselves? • If you did not consider any other delivery modalities, why did you feel that the current design was the only possible format to achieve your objectives? Questions to be posed primarily to implementation service providers on the evolution of, and learnings from, the program: QUESTION 4: WHAT DO YOU SEE AS THE PRIMARY STRENGTHS OF THE PROGRAM? (PROBE FOR SPEED TO LAUNCH AND IMPLEMENTATION, REACH, COST EFFECTIVENESS, SIMPLICITY, LEARNING, IMPACT) • What evidence exists to support this point of view? • How widely, and in what format have learnings from the program been shared? USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 107 QUESTION 5: HAVE ANY WEAKNESSES IN THE PROGRAM DESIGN BECOME APPARENT OVER TIME? • If so, what are they? • What would you do differently if you were to design the program today? • Would you choose a different delivery modality in hindsight? • In retrospect do you think a Grand Challenge format might have been a more effective delivery approach? Why? Why not? QUESTION 6: HAS THE PROGRAM CHANGED OVER TIME? HAVE YOU ALTERED THE DESIGN IN ANY WAY, EITHER AS CIRCUMSTANCES HAVE CHANGED, OR AS YOU HAVE LEARNED MORE DURING THE IMPLEMENTATION PROCESS? • If yes, in why, and in what way, has the design changed? QUESTION 7: HOW WOULD YOU DESCRIBE THE PARTNERSHIP MODEL SELECTED FOR THIS PROGRAM? • Why did you choose this particular partnership model? • How does the partnership model operate in practice? QUESTION 8: HAS THE PROGRAM BEEN SUCCESSFUL IN IDENTIFYING OR GENERATING INNOVATIVE SOLUTIONS TO DEVELOPMENT PROBLEMS? • If yes, what kind of innovations have been identified? (Probe for early stage / test / scaling) – Can you give some examples of these innovations, their effectiveness and their impact? • If no, was this because innovation was not a key objective of the program, or was there a different reason? QUESTION 9: WHAT PLANS ARE IN PLACE FOR ENSURING LONG-TERM IMPACT ONCE THE PROGRAM COMES TO THE END OF ITS DESIGNATED LIFESPAN AND WRAPS UP? HOW WILL THIS BE MONITORED AND BY WHOM? 108 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION A6.4 SURVEY QUESTIONNAIRE ALIAS QUESTION ENUMERATION VALUES INPUT FIELD LOGIC MANDATORY (Y/N) MEQ/ NOTES PAGE 1: INTRODUCTION This survey is being conducted on behalf of the United States Agency for International Development (USAID), one of the donors who provided the funding for your project through a Grand Challenge for Development (GC). This is a survey of your experience of receiving a grant under this GC, your relationship with the USAID or the Fund Management partner who administers the funds and the donor(s) providing the funding, and your project’s achievements. The findings will be used to inform future USAID programing. The survey will take approximately 10 minutes to complete. Please be frank: understanding the challenges and difficulties, as well as the successes, helps us to learn. If you are at the early stages of your grant you may find some of the questions more difficult to answer. Please answer as best you can or go on to the next question. Before we begin: A statement on data, privacy and consent What will we do with the data? The findings from the survey will be used to develop the GC approach. Your answers will be treated confidentially, and will not be used to identify you individually, unless you provide specific consent for us to do so. We will store your responses in a secure online survey system and will only include your contact details if you choose to provide them in the form below. How will your data be used? Data will be used primarily by the evaluation team to draw aggregate findings and learnings about the GC that you were/are involved with. We will ensure your confidentiality by sharing only the aggregate data and removing data which could identify you, unless you give specific consent for us to do otherwise. Findings from the data collection exercise will be shared with USAID, other funding partners/donors, and wider stakeholders interested in how GCs are implemented. How long will we keep your data? Once we have received your responses, all personal data will be anonymized and saved on a secure online survey system. At the end of the evaluation, data will be deleted from our servers within 20 business days. Do I have to respond to this? No, this is a voluntary activity. If you respond, we will assume that you give your consent to the data being used in the way described in this box. Thank you for taking the time to complete this survey. If you have any questions, please contact xxxx who will be able to provide support. PAGE 2: AWARDEE PROFILE TYPE_ORG How would you describe your organization? For profit organization, including for profit social enterprise Non-profit organization: International NGO Non-profit organization: civil society organization Non-profit organization: social enterprise Non-profit organization: academic or research institution Other (please specify): Radio button and free text for “Other” All YES MEQ1; MEQ 9; + filtering results USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 109 ALIAS QUESTION ENUMERATION VALUES INPUT FIELD LOGIC MANDATORY (Y/N) MEQ/ NOTES LOCAL _ORG Is your organization international or national/ locally based? (NB. If your organization has more than one office, please respond in relation to the office that was the direct recipient of this USAID grant). International; National or local; I don’t know/unsure Radio button All YES MEQ1; MEQ 9; + filtering results ORG_SIZE What was your organization’s most recent annual income/operating budget? Under $100k $101-500k $501k-$1m $1m-$5m Over $5m I don’t know/unsure Radio button All NO MEQ 1 MEQ 9; +filtering results GC_NAME Please indicate the name of the GC which has provided you with your most recent USAID GC funding: NB. Please respond survey questions with respect to this GC. If you have received funds from USAID under more than one GC, and would like to respond to the survey about a secondary GC as well, please feel free to conduct the survey twice, but there is no obligation to do so). All Children Reading Combating Zika and Future Threats Creating Hope in Conflict Ensuring Effective Health Supply Chains Fighting Ebola Making All Voices Count Powering Agriculture Saving Lives at Birth Scaling Off Grid Energy I don’t know/unsure Radio button; only one All YES Filtering results 110 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ALIAS QUESTION ENUMERATION VALUES INPUT FIELD LOGIC MANDATORY (Y/N) MEQ/ NOTES GC_ALL If you have received funding from USAID through more than one GC, please name the other GCs that you have received funding through. All Children Reading Combating Zika and Future Threats Creating Hope in Conflict Ensuring Effective Health Supply Chains Fighting Ebola Making All Voices Count Powering Agriculture Saving Lives at Birth Scaling Off Grid Energy I don’t know/unsure Not relevant to me – I only received funding through one GC Check box All NO Filtering results PAGE 3: GRANT/ PROJECT IMPORTANT: IF YOU HAVE RECEIVED FUNDING FROM MORE THAN ONE GC, PLEASE ANSWER THE REMAINING SURVEY QUESTIONS WITH RESPECT TO THE MOST RECENT GC THAT YOU HAVE RECEIVED FUNDING THROUGH, IDENTIFIED IN Q # ABOVE GRANT_ NUMBER How many grants have you received under this GC? NB. If you have received more than one grant under this GC, please answer the survey questions in relation to the most significant such grant for your organization. _ _ _ _ _ _ _ I don’t know/unsure Number field with “I don’t know” option All YES MEQ1 + filtering results GRANT_DATE What date did your project start? (If the exact date is not known to you, please provide an approximate date). _ _ _ _ _ _ _ I don’t know/unsure Number field with “I don’t know” option All YES MEQ1 + filtering results GRANT_LENGTH What is/was the length of the grant received (in months)? _ _ _ _ _ _ _[months] I don’t know/unsure Number field with “I don’t know” option All YES MEQ1 + filtering results USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 111 ALIAS QUESTION ENUMERATION VALUES INPUT FIELD LOGIC MANDATORY (Y/N) MEQ/ NOTES STAGE_INO_ START What stage(s) of innovation is/was your project working at when you were awarded your USAID grant? (Please select all that may be relevant) Early stage/ start up (e.g., ideation, research & development) Proof of concept/ piloting/ prototyping Testing and transitioning to scale Expanding/ Scaling More than one: please specify I do not know/Unsure Check box All YES MEQ1, 4 STAGE_INO_ NOW What stage(s) of innovation is your project working at now (or was, by the end of your engagement with the GC)? (Please select all that may be relevant) Early stage/ start up (e.g., ideation, research & development) Proof of concept/ piloting/ prototyping Testing and transitioning to scale Expanding/ Scaling More than one: please specify I do not know/Unsure Check box All YES MEQ1, 4 CALL_TYPE Through what type of call or competition did you get the award? Challenge Prize Other – please specify I do not know/unsure Radio button + free text for other All YES MEQ1 ORG_MISSION To what extent do you agree with the following statements: “Addressing social impact is a central part of our project.” “Improving the lives of women and girls and/or other disadvantaged groups is a central part of this project” “Ensuring better outcomes for marginalized groups is a central part of our project” Strongly Agree Agree Neutral Disagree Strongly Disagree N/A I do not know Radio button / matrix All YES MEQ1 112 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ALIAS QUESTION ENUMERATION VALUES INPUT FIELD LOGIC MANDATORY (Y/N) MEQ/ NOTES PAGE 4: APPLICATION PROCESS OUTREACH How did you hear about the GC? (Please select any that apply) Launch Event Web Search Other Media Referral/Network Direct Approach from Donor/ Fund Manager Other (please specify): Check box and free text for “Other” All YES MEQ 9 NON_TRAD Prior to applying for this grant, had your organization received any funding from USAID in the past 5 years? Yes No I don’t know/unsure Radio button All YES MEQ 9 GC_OBJECTIVES Was it clear to you what international development problem the GC was aiming to tackle, and how your project/ innovation might work towards solving this problem? Yes No I don’t know/unsure Radio button All YES MEQ 1 APPLI_PROCESS To what extent do you agree with the following statements about the application process for the grant you received: The call for proposals for the grant was written in language we could easily understand. Information available to us about the application process was easy to understand. We could contact USAID/ the Fund Management partner easily and ask for help if we needed it. The time required to complete the application process from start to finish was reasonable. It was clear to us how our project would be reviewed and the criteria upon which selection would be made. Strongly Agree Agree Neutral Disagree Strongly Disagree N/A I do not know Radio button matrix All YES MEQ1, 7 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 113 ALIAS QUESTION ENUMERATION VALUES INPUT FIELD LOGIC MANDATORY (Y/N) MEQ/ NOTES PAGE 5: GRANT SET-UP PROCESS (THESE QUESTIONS REFER TO THE PERIOD BETWEEN AWARD OF THE GRANT AND THE START OF YOUR PROJECT) SETUP_TRNG What orientation or training support did you receive from USAID or its partners at the outset about how the grant would be managed and reported on? Please tick all topics that apply. The GC’s objectives for social impact. The GC’s objectives for improving the lives of women and girls and/or other disadvantaged groups. Budgeting and financial management. Financial reporting (how & when to do this). Annual reporting (how & when to do this). Final project reporting (how & when to do this). Collecting and presenting evidence for results. Collecting and presenting for results by gender (male, female), age, and other aspects of social differentiation. We were offered training but did not take it up. We did not receive any training or support from USAID or its partners. Other (please specify): Check boxes All YES MEQ1, 7 114 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ALIAS QUESTION ENUMERATION VALUES INPUT FIELD LOGIC MANDATORY (Y/N) MEQ/ NOTES USE_TRNG To what extent do you agree with the following statement: “We found the training and other support we received at the start of the grant useful”. Strongly Agree Agree Neutral Disagree Strongly Disagree N/A I do not know Radio button All YES MEQ1, 7 GOV_CHECK Did USAID or its implementing partners check your organization’s financial management and governance systems? Yes No I don’t know Radio button All YES MEQ 1, 7 USE_CHECK To what extent do you agree with the following statements about the financial management checks you received: The financial checks were reasonable considering the size of the grant. The financial checking process was useful for the development of our organization / business. Strongly Agree Agree Neutral Disagree Strongly Disagree N/A I do not know Radio button matrix If answered “Yes” to GOV_ CHECK YES MEQ1, 7 COM_SETUP Please add any other comments you have on the application or grant set up processes for your grant. Free text All YES MEQ1, 7 PAGE 6: GRANT MANAGEMENT PROCESS MGMT To what extent do you agree with the following statements about the grant management processes for your project? “We find that project funds are released on time.” “The ongoing grant management support provided by USAID or its partners is useful/relevant for us and our project . Strongly Agree Agree Neutral Disagree Strongly Disagree N/A I do not know Radio button matrix All YES MEQ1, 7 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 115 ALIAS QUESTION ENUMERATION VALUES INPUT FIELD LOGIC MANDATORY (Y/N) MEQ/ NOTES TA_MGMT_ RATING How would you rate the support you have received from USAID or their partners, including any training, during implementation of your grant? (Please rate on a scale of 1 to 10 where 1 = not at all useful and 10= extremely useful, or select the N/A button if you did not receive this type of support.). Support on reporting as required by the GC. Support on monitoring and evaluation e.g., indicators, monitoring results and/or impact, evaluations. Support in project budgeting and financial management. Support in mainstreaming gender and /or social inclusion issues into our project. 1 2 3 4 5 6 7 8 9 10 Scale of 1 to 10 + N/A box All YES MEQ1 MEQ2 MEQ 7 REP_INDIC Were you asked to report on specific or standard program indicators linked to your progress in any of the following areas? Please tick all areas that apply. Impact and /or scale (e.g., sales, number of beneficiaries, job creation) Inclusiveness (e.g., age, gender…) Organizational capacity Profitability Funding catalyzed Other: please specify Check boxes & free text option All YES MEQ1 MEQ2 MEQ 4 MEQ 10 116 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ALIAS QUESTION ENUMERATION VALUES INPUT FIELD LOGIC MANDATORY (Y/N) MEQ/ NOTES REP_SYSTEM To what extent do you agree with the following statements about the grant management processes for your project? “The tools (templates, reporting formats) we receive for reporting results are easy to use.” “The reporting mechanisms in place adequately captured our results and impact.” Strongly Agree Agree Neutral Disagree Strongly Disagree N/A I do not know Radio button All YES MEQ1 MEQ2 MEQ 7 COM_MGMT Please add any other comments you have on grant management and reporting processes. Free text All YES MEQ1 MEQ2 MEQ 7 PAGE 7: SCALING AND ACCELERATION TA_OVERALL Did you receive specific technical assistance in relation to scaling or accelerating your innovation from USAID or partners, or any other provider? Yes No I don’t know/ unsure N/A Radio button All YES MEQ 4 MEQ 8 MEQ 7 TA_SUFF The level of support and technical assistance provided in relation to scaling or acceleration is (or was): Not enough for us Enough for us Too much for us Radio button If answered “Yes” to TA_ OVERALL YES MEQ4 MEQ8 MEQ 7 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 117 ALIAS QUESTION ENUMERATION VALUES INPUT FIELD LOGIC MANDATORY (Y/N) MEQ/ NOTES TA_RATING How would you rate the specific acceleration or scaling support received? (Please rate on a scale of 1 to 10 where 1 = not at all useful and 10= extremely useful, or select the N/A button if you did not receive this type of support.). Support in developing and/or refining the business or intervention model (e.g., market study, customer identification, strategy). Support on business systems and organizational development. Support in ongoing/long-term fundraising (e.g., identifying opportunities and applying for funding, networking, referrals). Support in identifying partners for scale (e.g., distribution channels). Learning and sharing opportunities on pathways to scale, including from other grantees. Other (please specify): 1 2 3 4 5 6 7 8 9 10 Scale of 1 to 10 + N/A box + free text for other If answered “Yes” to TA_ OVERALL YES MEQ4 MEQ8 MEQ 7 COM_TA_SCALE Do you have any other comments regarding the scaling or acceleration support you received, including any recommendations or areas for improvement? Free text If answered “Yes” to TA_ OVERALL YES MEQ4 MEQ8 MEQ 7 118 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ALIAS QUESTION ENUMERATION VALUES INPUT FIELD LOGIC MANDATORY (Y/N) MEQ/ NOTES DIVERSIFI CATION To what extent do you agree with the following statement: “The support we have received through the GC has catalyzed the development of other solutions by our organization or other players.” Please explain this further. Strongly Agree Agree Neutral Disagree Strongly Disagree N/A I do not know Radio button + free text option If answered “Yes” to TA_ OVERALL YES MEQ 4 MEQ10 MEQ 7 PAGE 8: ACHIEVEMENTS, IMPACT AND SUSTAINABILITY PROJ_BENEF Who would you describe as the ultimate beneficiaries or end users of your innovation or solution? Please select any that apply. Women (24 years or under) Women (above 24 years) Men (24 years or under) Men (above 24 years) People with disability People with low income (extreme poor or poor) Emerging middle class or non-poor Academic or health institution staff in the country where the project is implemented Agricultural workers in the country where the project is implemented Businesses in the country where the project is implemented (please specify type of business): Other (please specify): Check boxes + free text All YES MEQ 1 MEQ2 AVMT_KEYOBJ To what extent did you achieve (or are on track to achieve) your key objectives or targets? We achieved (or are on track to achieve) our key objectives or targets We did not achieve (or are not on track to achieve) one or more key objectives or targets I do not know/Too early to tell Radio Button All YES MEQ1 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 119 ALIAS QUESTION ENUMERATION VALUES INPUT FIELD LOGIC MANDATORY (Y/N) MEQ/ NOTES AVMT_SCALE To what extent did you achieve (or are on track to achieve) your scaling objectives? We achieved (or are on track to achieve) our scaling objectives We did not achieve (or are not on track to achieve) one our scaling objectives I do not know/Too early to tell Not Applicable Radio Button All YES MEQ4 COM_AVMT Please provide a short description of your innovation, your achievements, and how you achieved them, and any results specifically around scaling that you wish to highlight. Free text All NO MEQ1 PROJ_EVID What methods, if any, do (or did) you use to collect evidence about the achievements of the project? Please select any that apply. Baseline/end line surveys Impact modelling/ quantification of benefits Qualitative assessments Beneficiary / user feedback mechanisms Product performance tests Anecdotal evidence N/A – we don’t collect evidence on our project achievements Other – please specify Check box +free text for other All YES MEQ 2 120 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ALIAS QUESTION ENUMERATION VALUES INPUT FIELD LOGIC MANDATORY (Y/N) MEQ/ NOTES SUSTAIN ABILITY To what extent do you agree with the following statements about the sustainability of your project? “USAID/fund management partners started to engage with us to consider sustainability from the beginning of the project” “We are required to report on progress towards sustainability” “USAID/Fund management partners has supported us to ensure that the impact of our project is sustainable” “USAID/Fund Management partners has supported us to ensure that our project, service or innovation can continue and develop beyond the lifetime of this grant” “Engagement in the GC has enabled us to access or generate further funding for our innovation” “Engagement in the GC has enabled us to develop other solutions” “Engagement in the GC has contributed to increased awareness about the innovation in the context in which we work.” Strongly Agree Agree Neutral Disagree Strongly Disagree N/A I do not know Radio button matrix All YES MEQ 1 MEQ4 MEQ10 MEQ 7 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 121 ALIAS QUESTION ENUMERATION VALUES INPUT FIELD LOGIC MANDATORY (Y/N) MEQ/ NOTES CATFUND_FCT How did the GC help you raise further funds for your innovation? Increased organizational capacity Increased understanding of donors’ requirements and grant application processes Referrals from USAID/partners and networking or investment opportunities Use of evidence (e.g., impact data, evaluations, case studies) and communication materials generated by the GC in fundraising Other (please specify): N/A I don’t know Check boxes If answered “Strongly Agree” or “Agree” to SUSTAIN￾ABILITY “Engagement in the GC has enabled us to access or generate further fund￾ing for our innovation” YES MEQ10 PAGE 9: CONCLUSION CCL_STRGT Please tell us what aspects of your experience with the GC were particularly successful. Free text All NO CCL_WKNS Please tell us what aspects of your experience with the GC could have been improved. Free text All NO We are particularly interested to find out more about innovations that have reached scale, and how this has been achieved. If you are willing to participate in a follow-up discussion on this, please complete the fields below. This information will be used for the sole purpose of identifying respondents willing to have a further discussion about some of the points raised in the survey. We will not use it to attribute findings to any of the answers to this survey. NAME_ORG What is the name of your organization? Free text All NO NAME_PROJECT What is the name of your project (if different)? Free text All NO NAME What is your name? Free text All NO EMAIL What is your email address? Free text All NO End of questionnaire: Thank you page: Thank you for taking part in this survey! Your views are crucial for us to understand how you have benefited from your engagement in the GC and the results that have been achieved. The findings will be used to inform future programing. 122 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION A6.5 CASE STUDIES: GUIDE AND PROTOCOL Objective: The purpose of the case studies is to investigate in greater depth the results of GCs, factors affecting performance, and scaling results – including pathways to scale and catalytic effects. The case studies are intended to supplement other evaluation data collection methods and sources, and will provide evaluation users with further insights into what scaling results can reasonably or realistically be achieved from a GC intending to “work at scale”. Each case study will focus on a type of GC or a type of innovation. Note: As set out in the evaluation SOW, the eventual number and specific focus of the case studies will be informed by initial data collection and analysis. As such, this document is not intended as a detailed guide or protocol, as this will be determined during the data collection phase. Following discussion with Catalyst and USAID, however, we propose to develop a simple framework with (potentially) four categories: 1) challenge type (including, sectoral focus, stated objectives for scaling, and catalytic effects); 2) innovation type (for example those introducing new technologies or those introducing service delivery models); 3) innovation stage; and 4) operating context (for example, influence of geographical context and enabling environment). The evaluation team will use this framework or ‘lens’, during our initial data collection and analysis exercise to log our initial findings around what innovations or GCs fall into which of these categories. In discussion with USAID and Catalyst, this will inform our eventual selection of cases for the case studies. A6.5.1  DATA COLLECTION AND TOOLS DESK REVIEW A detailed review of grantee documentation will be undertaken against areas of enquiry and data captured in a matrix. Sources include project reports and documents and public information such as grantee websites and media articles. (Note: The case study on SL@B’s Gradian project was purely reliant on secondary data collection due to non-availability of grantee personnel for interview). REMOTE KEY INFORMANT INTERVIEWS The overall KII protocol/guidance applies. KII topic guides should be adapted and, for grantees, the informed by the grantee survey and results of earlier analysis, to delve in greater depth into specific experiences of scaling innovations, results achieved and means to achievement. Sample Key Informant Interview Topic Guide: Grantee General information: Full name and Job title (and time in this role): MEQ 1: Achievement of results and factors affecting these I understand that your innovation does [insert] and that your grant was awarded to prototype / test / scale (as relevant) and is now at an end / halfway through / just started (as relevant). Is that correct / please briefly explain further. • Can you tell me what the objectives of your project were? What has the project achieved towards its objectives? Any unintended outcomes? • How does your project contribute to the objectives of the GCD as a whole? • Has your project contributed to any wider, systemic change (either in the market, or in government policies/legislation etc. as appropriate)? if so, can you explain? • What were the key learnings from your experience of implementing the project and driving it to results? What went well, what went less well and why? MEQ 2: Measuring impact, results, and uptake • Please can you briefly outline your arrangements for monitoring and evaluating your project - What were your main indicators of achievement and how did you track progress towards this? [Probe for whether they define achievement in terms of both outputs and outcomes]. • Were any specific requirements or guidance on monitoring and evaluation from the GCD manager? [if time: Did you receive any support from GCD authorities to assist with MEL? If Yes, what sort of supports you received? How useful were they?] • To what extent did you use monitoring evidence to learn so that you could adapt your approach? (Further probe: could you provide us with some examples?) USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 123 MEQ 4: Scaling Innovations • To what extent was scaling up a priority at the beginning of the project? What were your specific aims for scaling? • Did you receive any support for scaling up in any stages of the GCD? • Were you able to scale up the innovation? How did any scaling up compare to your original expectations? • What were the key learnings with regards to scaling up? What helps? What does not help? How did you overcome the challenges if you did? MEQ 5: Investing in ecosystems Innovation takes place within an ‘ecosystem’ or wider context which is shaped by Governments and regulators, private sector and market actors, other donors, civil society and so on. Ecosystem strengthening can include brokering partnerships, linking innovators to investor, or influencing policy or regulations. • Do you think that the GCD helped, or helped you, to strengthen the ecosystem for your innovation? If yes, how did they do this? If no, what support would you have liked? MEQ 6: Engaging with partner Missions • Was there any engagement with country missions in your project implementation? Can you briefly outline what the engagement was and if (how) it supported your project? MEQ 8: Acceleration • Was there a clear strategy for acceleration at the design stage? (if not clear, acceleration = progressing the innovation through its development more quickly) • Did you get any acceleration support? If so, what was this? How useful was this? MEQ 9: Procurement & reducing barriers to funding for non-traditional partners (Applicable if the firm is a non-traditional partner) • How easy or difficult was it for you to apply for the grant? • Please explain your answer: what were the barriers? What did the GCD do to make the fund accessible to you? • What other steps would have helped you with the application process? MEQ 10: Catalytic effects • Have you been able to catalyze (leverage) additional funding for your innovation? • How has being part of the GCD helped you to do this? MODERATED FOCUS GROUP DISCUSSIONS Note: The original methodology, before the COVID-19 pandemic, included optional moderated focus group discussions with small groups of grantees as a supplementary datasource. In the event, the limited numbers involved in the case studies meant that this was not appropriate, and an online version was not undertaken. Some interviews however were conducted with more than one individual person from a grantee, which stimulated discussion. 124 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ANNEX 7  KEY INFORMANTS INTERVIEWED DATE INTERVIEWEE INTERVIEWER ORGANIZATION GC / TOPIC 11/02/2020 Kerry Leigh Sudhanshu Joshi Australian Department of Foreign Affairs and Trade All Children Reading 11/19/2020 Daniel Plaut Jeevan Raj Lohani, Sudhanshu Joshi Results for Development All Children Reading 11/19/2020 Luke Heinkel Jeevan Raj Lohani, Sudhanshu Joshi Results for Development All Children Reading 11/18/2020 Anthony Bloome Jeevan Raj Lohani, Sudhanshu Joshi USAID All Children Reading 11/20/2020 Leo Hosh Jeevan Raj Lohani, Sudhanshu Joshi World Vision All Children Reading 07/02/2020 Michelle Oetman Martin Wright, Jeevan Raj Lohani, Sudhanshu Joshi World Vision All Children Reading 10/19/2020 Shelly Malecki Sudhanshu Joshi World Vision All Children Reading 07/02/2020 Shelly Malecki Martin Wright, Jeevan Raj Lohani, Sudhanshu Joshi World Vision All Children Reading 03/04/2021 Matt Utterback Jeevan Raj Lohani E-Kitabu Case Studies 03/04/2021 Will Clurman Jeevan Raj Lohani E-Kitabu Case Studies 03/05/2021 Nedjma Koval Jeevan Raj Lohani Little Thinking Minds & Integrated International Case Studies 03/11/2021 Eduardo Quevedo Jeevan Raj Lohani World Mosquito Case Studies 03/11/2021 Richard Mora Jeevan Raj Lohani World Mosquito Case Studies 03/11/2021 Simon Kutcher Jeevan Raj Lohani World Mosquito Case Studies 10/16/2020 Crystal Byrd David Smith USAID CEA 10/21/2020 Elena Walls David Smith USAID CEA 10/21/2020 Ku McMahan David Smith USAID CEA 10/16/2020 Scott Jackson David Smith USAID CEA 11/16/2020 Nicholas Sukitsch Katharine May Boston Consulting Group Combating Zika and Future Threats 11/16/2020 Trish Stroman Katharine May Boston Consulting Group Combating Zika and Future Threats 11/23/2020 Erin Barringer Katharine May Dalberg Combating Zika and Future Threats 11/17/2020 Ann Marie Brouilette Katharine May USAID Combating Zika and Future Threats 11/17/2020 Eric Baranack Katharine May USAID Combating Zika and Future Threats 11/13/2020 Marissa Leffler Katharine May USAID Combating Zika and Future Threats 11/12/2020 April Warren Matthew Kentridge abt Associates Comparator Analysis 11/12/2020 Sarah Bradley Matthew Kentridge abt Associates Comparator Analysis 11/12/2020 Susan Mitchell Matthew Kentridge abt Associates Comparator Analysis USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 125 DATE INTERVIEWEE INTERVIEWER ORGANIZATION GC / TOPIC 12/09/2020 Alex Dehgan Martin Wright, Matthew Kentridge Conservation X Comparator Analysis 12/11/2020 Dave Ferguson Martin Wright, Matthew Kentridge Ferguson Strategic Technology Services Comparator Analysis 11/12/2020 Laura Harwig Matthew Kentridge Fintrac Comparator Analysis 12/17/2020 Wendy Taylor Martin Wright, Matthew Kentridge Jphiego Comparator Analysis 11/09/2020 Amanda West Matthew Kentridge Mercy Corps Comparator Analysis 11/09/2020 Chris Walker Matthew Kentridge Mercy Corps Comparator Analysis 11/11/2020 Stephen Rahaim Matthew Kentridge Resonance Comparator Analysis 11/05/2020 Joyce Friedenberg Matthew Kentridge USAID Comparator Analysis 11/16/2020 Katie Garcia Matthew Kentridge USAID Comparator Analysis 11/04/2020 Marea Pappas Matthew Kentridge USAID Comparator Analysis 11/16/2020 Mark Huisenga Matthew Kentridge USAID Comparator Analysis 11/16/2020 Rana El Hattab Matthew Kentridge USAID Comparator Analysis 10/26/2020 Aramins Gerlofsma Clarissa Poulson Dutch Ministry of Foreign Affairs Creating Hope in Conflict 11/11/2020 Tarah Friend Clarissa Poulson FCDO Creating Hope in Conflict 11/13/2020 Chris Houston Clarissa Poulson Grand Challenges Canada Creating Hope in Conflict 07/02/2020 Patrick Coburn Martin Wright Grand Challenges Canada Creating Hope in Conflict 11/13/2020 Patrick Coburn Clarissa Poulson Grand Challenges Canada Creating Hope in Conflict 11/13/2020 Zainah Alsamman Clarissa Poulson Grand Challenges Canada Creating Hope in Conflict 07/02/2020 Devorah West Martin Wright USAID Creating Hope in Conflict 11/09/2020 Devorah West Clarissa Poulson USAID Creating Hope in Conflict 11/09/2020 Meera Pranav Clarissa Poulson USAID Creating Hope in Conflict 11/09/2020 Nicholas Farmer Clarissa Poulson USAID Creating Hope in Conflict 12/03/2020 Emma-Lee Knape Clarissa Poulson WFP Creating Hope in Conflict 07/02/2020 Kevin Pilz Martin Wright USAID Ensuring Effective Health Supply Chains 10/22/2020 Ankunda Kariisa Jeevan Raj Lohani USAID Fighting Ebola 10/22/2020 Meghan Majorowski Jeevan Raj Lohani USAID Fighting Ebola 11/13/2020 Lucie Pluschke Martin Wright GIZ Powering Agriculture 11/02/2020 Headley Jacobus Martin Wright Tetratech Powering Agriculture 07/01/2020 Augusta Abrahamse Martin Wright USAID Powering Agriculture 11/04/2020 Augusta Abrahamse Martin Wright USAID Powering Agriculture 11/17/2020 Christina Tamer Martin Wright VentureWell Powering Agriculture 11/17/2020 Laura Sampath Martin Wright VentureWell Powering Agriculture 12/10/2020 Pia Lindstrom Martin Wright Sida Powering Agriculture / Securing Water for Food 11/03/2020 Ankunda Kariisa Katharine May USAID Saving Lives and Birth / Combating Zika and Future Threats 11/09/2020 Ankunda Kariisa Katharine May USAID Saving Lives and Birth / Combating Zika and Future Threats 126 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION DATE INTERVIEWEE INTERVIEWER ORGANIZATION GC / TOPIC 11/03/2020 Meghan Majorowski Katharine May USAID Saving Lives and Birth / Combating Zika and Future Threats 11/09/2020 Meghan Majorowski Katharine May USAID Saving Lives and Birth / Combating Zika and Future Threats 11/11/2020 Krishna Udayakumar Katharine May Duke Global Health Innovation Center Saving Lives at Birth 11/10/2020 Shirlie Addies Katharine May FCDO Saving Lives at Birth 11/04/2020 Janne Dingemans Katharine May Grand Challenges Canada Saving Lives at Birth 11/04/2020 Tina-Marie Assi Katharine May Grand Challenges Canada Saving Lives at Birth 11/11/2020 Laura Sampath Katharine May VentureWell Saving Lives at Birth 11/11/2020 Mark Marino Katharine May VentureWell Saving Lives at Birth 11/06/2020 Steven Hunt Maeva Amarger FCDO Scaling Off-Grid Energy 11/18/2020 Katrina Pielli Maeva Amarger Power Africa Scaling Off-Grid Energy 11/24/2020 Catharine Pham Maeva Amarger USAID Scaling Off-Grid Energy 11/12/2020 Maurice Kent Maeva Amarger USAID Scaling Off-Grid Energy 11/13/2020 Molly Dean Maeva Amarger USAID Scaling Off-Grid Energy 11/17/2020 Omer van Renterghem Martin Wright Dutch Ministry of Foreign Affairs Securing Water for Food 11/10/2020 Kevan Hayes Martin Wright Partnerships Incubator Securing Water for Food 11/04/2020 Jatin Yadav Martin Wright PM Consulting Securing Water for Food 11/23/2020 Ku McMahan Martin Wright USAID Securing Water for Food 11/11/2020 Mamy Keita Martin Wright USAID Guinea Mission Securing Water for Food 05/18/2020 Lorin Kavanaugh￾Ulku Martin Wright USAID Strategy 05/19/2020 Scott Jackson Martin Wright USAID Strategy USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 127 Photo: Global Devlab/USAID 128 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ANNEX 8  LIST OF DOCUMENTS CONSULTED Please click here to access the list of documents consulted. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 129 ANNEX 9 COMPARATOR ANALYSIS 110 In addition to grants, All Children Reading: A Grand Challenge for Development (ACR GCD) has also awarded prizes to successful applicants. A9.1 EXECUTIVE SUMMARY A9.1.1 PURPOSE The independent evaluation of USAID’s portfolio of 10 Grand Challenges for Development (GC) includes a review of four other USAID programs selected at the outset of the evaluation for comparative purposes: Innovation Investment Alliance, Partnering for Innovation, Local Works, and SHOPS Plus. These programs, all of which have delivered successful results, were selected because they also focus on identifying innovative solutions to development challenges – either explicitly or implicitly. Like the GCs, these programs tap into the ideas, creativity and experience of non-traditional development partners – entrepreneurs and innovators within social enterprises, local actors and communities, NGOs, and private companies – to find new solutions for intractable development problems. The purpose of this review is to identify similarities and differences in approach between these programs and the GCs in order to expose GC program managers to those factors which have driven the success of the comparator programs, and to find opportunities for dialog, shared insight, and active collaboration in the future. There are existing links between the selected comparator programs and the GCs – e.g., Partnering for Innovation has worked together with Securing Water for Food (SWFF) and Powering Agriculture: An Energy Grand Challenge for Development (PAEGC). This paper, however, argues that all GCs can benefit from the experience of these programs when it comes to creating conditions for long-term sustainability of projects; building flexibility into program design; engaging with, and sharing learning across a wider ecosystem of actors; mobilizing non￾traditional actors; and working directly with Missions and other OUs at USAID. We hope that this paper will encourage greater communication between GC program managers and a wider community of program managers at USAID or partner agencies and donors, all of whom have valuable experience and insight with regard to the design, implementation, and sustainability of programs dedicated to innovative solutions to development challenges. A9.1.2  DESIGN FEATURES OF THE COMPARATOR PROGRAMS There are a number of similarities between the selected comparator programs and the GCs – they all make grants to development partners on a competitive basis,110 focus on bringing in non￾traditional development actors, and seek to play a catalytic role, leveraging additional resources through matched funding or co-investment by funding partners. A key difference, however, relates to where the programs are located along the different stages of innovation: while GCs tend to focus at earlier stages – ideation, testing and trialing, and laying the foundations for scaling – the comparator programs are further along the journey to scale, building partnerships, promoting long-term uptake, and commercialization of solutions. (An exception is Local Works, where a key feature of the program is the need to define the nature of the challenge in the first place, before the search for solutions begins.) The best way, therefore, to characterize these four comparator programs in relation to GCs is to see them as complementary approaches to championing innovation at USAID, with all playing a role along the innovation continuum from identification of challenge to roll-out of scaled solutions. The four comparator programs examined above have been successful in encouraging innovation and this paper explores those design features and implementation practices that have driven the success of these four programs. Across the program, a number of key success factors stand out: • Building in sustainability – The four comparator programs demonstrate that, despite the fact that program lifespans are circumscribed by procurement rules, it is possible to drive sustainability both through measures to ensure that individual initiatives are supported and amplified by relevant stakeholders, and through continuous evolution of the original programs themselves. • Flexible design and adaptive programing – All four comparator programs have built in the necessary flexibility to respond to changing needs and operating conditions, with significant 130 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION interpretative freedom when it comes to scope. Program evolution has been driven by new learnings from the implementation of different initiatives, leading to changes in the way program managers understand and define the development problem they are addressing. • A focus on learning – The programs all take a utilization approach to learning, to enable ‘in￾flight’ corrections during implementation; create practitioners’ guides and other knowledge products that widen the knowledge base inside and outside of USAID and allow grantees and other stakeholders to share insights and experience; and to build on experience in order to create enhanced versions of these programs in the future. • An ecosystem approach – The four programs demonstrate the value of considering broader ecosystem effects when designing development interventions, to drive uptake and adoption of innovative solutions by different stakeholders. The more the complex intersections between system actors are understood and incorporated into program design, the greater the prospect for long-term impact. • Engagement with Missions and central operating units – Three of the four comparator programs (the exception is IIA), were all specifically designed as central programs which work directly with Missions and other OUs. These internal partners benefit from the ability to tap into an easy-to-use central procurement mechanism, while program focus on public-private partnerships have made it easier for them to establish direct connections with private sector partners. Conversely, these centrally located programs benefit from the ability to go both broad (at regional or global level through buy-in by central OUs, and deep (focused impact at local level through Mission uptake). • A focus on non-traditional development actors – A focus on non-traditional players has enabled the different programs to open up spaces traditionally excluded from development activity: commercial supply chains serving smallholder farmers; private sector participation in and contribution to public health initiatives; and direct engagement with local communities to address local challenges. • As USAID takes forward the GC portfolio of programs into the next decade of design, partnership and implementation, it is useful to reflect on design elements from these 111 See https://www.usaid.gov/fallsemester/usaid-101/innovation-development. comparator programs that could be incorporated into future GC thinking on the role of the GC model in overall development programing. A9.2 INTRODUCTION Innovation at USAID is defined as “novel business or organizational models, operational or production processes, or products or services that lead to substantial improvements (not incremental “next steps”) in addressing development challenges. Innovation may incorporate science and technology but is often broader, to include new processes or business models.”111 At the same time as USAID was developing its GC approach, a number of other global programs were launched at USAID, housed both in the Lab and in other Bureaus and OUs, which were also focused on driving innovation in development – whether through new products and services or through new partnerships and approaches to development. Four such parallel programs are explored in more detail below. A9.3 OVERVIEW OF COMPARATOR PROGRAMS • The Innovation Investment Alliance (IIA) – A global funding and learning partnership between USAID and Skoll Foundation to enable not-for-profit social enterprises to take proven innovations to scale. Sample projects include Lista by Fundacion Capital, a $1.9m grant to build financial capabilities for those living in poverty across 7 countries in Latin America; and Evidence Action, in Uganda, a $2m grant to provide safe drinking water dispensers in Uganda. • Partnership for Innovation – A global program aimed at selling innovative products and services to smallholder farmers at the base of the pyramid (BoP). The program targeted both small entrepreneurs and large corporations with sample partnerships including Oiko Credit in Peru to provide finance and business skills to smallholder coffee producers; and ATEC in Cambodia to provide biodigesters to farmers to make their own organic fertilizer. • Local Works – A global program to make development more locally-led and locally specific. Local Works funds Missions to adapt and localize existing programs in line with their objectives, USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 131 enabling them to transfer leadership to local entities, or create new, locally-owned programs. Sample initiatives include working with local actors in Bangladesh to address lack of resources for Rohingya refugees; and linking local actors with national policy in the Dominican Republic to address high rates of poverty and crime along the border with Haiti. • Sustaining Health Outcomes through the Private Sector Plus (SHOPS Plus) – A flagship global program that aims both to build a global evidence base for private health sector engagement and strengthen local health systems 112 For detail on SHOPS Plus initiatives to combat HIV/AIDS and TB specifically, see: https://www.shopsplusproject.org/healthareas/hiv and https://www. shopsplusproject.org/healthareas/tuberculosis. 113 $20m from USAID and $30m from Skoll Foundation. 114 Now the Development, Democracy and Innovation Bureau/Innovation Pillar. 115 $5m per annum core funding; partners must provide matching funds to receive grant finance from USAID. 116 See https://www.usaid.gov/sites/default/files/documents/1865/Local-Works-Guidance-2020.pdf. 117 Bureau for Economic Growth, Education and Environment Office for Local Sustainability. 118 SHOPS Plus has field offices in nine countries, and operations in a further 10. 119 See Mercy Corps, Scaling High Impact Innovation of Social Entrepreneurs, Final Report 2012-2018. The report does not further define ”change in status”. 120 Fintrac, Partnering for Innovation Annual Report 2020. 121 Partnering for Innovation is currently analyzing larger system impacts of the program with results due to be published later in 2021. 122 USAID, Local Works Fact Sheet, June 2020. 123 Impact is measured by the degree to which Missions and other Operating Units continue to promote local development activities beyond the term of their involvement with Local Works. Anecdotal evidence suggests that this has been the case for some program alumni, though a host of exogenous factors through engagement of private health actors and innovations in countries around the world. Areas of focus include family planning and reproductive health, maternal, newborn and child health, TB, and HIV/AIDS, by improving service delivery models, strengthening business sustainability and access to finance, developing partnerships and conducting research.112 Sample “innovation” activities include developing an application called TB Star in Nigeria to aid diagnosis and treatment of TB; and piloting a micro-insurance product for those living with HIV/AIDS in Tanzania. ANNEX TABLE 6: KEY PARAMETERS OF COMPARATOR PROGRAMS PROGRAM SIZE ($) DURATION LOCATION WITHIN USAID IP NO. MISSIONS PARTICIPATING IIA $50m113 2012-2018 U.S. Global Innovation Lab114 Mercy Corps N/A Partnering for Innovation $71m115 2012-2021 Bureau for Resilience and Food Security Fintrac 15 Local Works $50m116 2015-present (E3/LS)117 LINC 32 SHOPS Plus $150m ($12m p.a. core funds) 2016-2021 Global Health Office of Population and Reproductive Health ABT Associates 19118 A9.3.1 PROGRAM METRICS The four programs all demonstrate success across a number of different metrics: ANNEX FIGURE 2: KEY PERFORMANCE METRICS OF COMPARATOR PROGRAMS PROGRAM REACH OUTCOMES IMPACT IIA 8.3m people with access to IIA innovations 5.1m people have utilized the innovations 1.5m people with positive change in status119 Partnering for Innovation 75 partnerships in 24 countries 1.7m smallholder farmers reached 133 technologies commercialized120 with sales of $110m N/A121 Local Works Buy-in from 32 Missions 100 percent of 15 locally￾led initiatives achieved their targets122 N/A123 132 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION PROGRAM REACH OUTCOMES IMPACT SHOPS Plus PPP in 27 countries 48m health products distributed 20,000 healthcare providers trained 20 new health products introduced 83 research studies used to improve programing or policy $13.5m finance to private health sector with SHOPS Plus support Treatment of 2.1m children with diarrhea 2.5m couple years of protection124 Iron folate supplements to 85,000 pregnant women 120,000 bed nets distributed Health messaging to 36m consumers can affect Mission program decisions towards lower-risk options even if they broadly subscribe to the principle of locally-led development. 124 Defined as protection from pregnancy provided by contraceptive methods over a one-year period (Marie Stopes International). 125 All GC’s offer challenge grants, but many use additional tools depending on the problem they intend to solve, such as prizes, hack-a-thons, acceleration services, and more. 126 IPE Triple Line interview with Mercy Corps 9.11.20. These awards were the most difficult to secure. 127 An exception is IIA where 7 out of 8 grantees had previously received grants from Skoll Foundation and 4 had received funds from USAID’s Development Innovation Ventures in the past. A9.3.2  SIMILARITIES AND DIFFERENCES WITH GRAND CHALLENGES Key similarities between the four comparator programs and the GCs include: • Competitive awards – All programs make grants to organizations engaged in development activities, either directly or through Missions or implementing partners. Like GC grants,125 these funds are awarded on a competitive basis, with applicants having to meet rigorous eligibility criteria and due diligence requirements to demonstrate fit with program objectives and likelihood of success. – In the case of IIA, grantees did not apply for funding in an open application process. Instead, deals were sourced and vetted by Mercy Corps in collaboration with Skoll Foundation and USAID. A subset of grantees received transition-to-scale grants between $1m and $3.5m,126 similar in size to GC awards. • Non-traditional development actors – The programs were concerned, explicitly or implicitly, in widening the pool of organizations and actors active in development,127 whether bringing the private sector into spaces from which they had traditionally been excluded; creating access to USAID funding opportunities for “non-usual suspects” or focusing USAID programing around locally-led development. • Catalytic role – All programs were designed to play a catalytic role, leveraging additional funds and resources from partners either directly (working with a co-funder or through matched grant investments) or indirectly (through market￾making activities and uptake by third party investors). • Market focus – The GCs provide a platform for enhanced collaboration around market barriers, including policy and regulatory reforms, capacity building, market research and analysis, and advocacy. This is in line with the objectives of both Partnering for Innovation and Shops Plus which are dedicated to opening the small scale agricultural and public health arenas respectively to private sector solutions. • Focus on innovation – In the case of IIA and Partnering for Innovation, the programs were specifically focused on championing innovation in development. SHOPS Plus identified and supported private sector innovations as a by￾product of broader program objectives, while Local Works was not specifically focused on finding new solutions to development problems but rather brought an innovative approach to identifying and tackling development problems in local settings. In other respects, however, the programs differ to a greater or lesser extent from the GC model. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 133 • Sector focus – IIA and Local Works are not sector specific, and even Partnering for Innovation and SHOPS Plus, operating in the smallholder agriculture and health sectors respectively, are designed to achieve broad objectives, rather than a narrowly defined challenge within a sector. • Partnership models – Only IIA uses a co￾funding model similar to the GCs. In all other cases, all funding comes exclusively from USAID and references to partners are to grantees or participants in specific projects.128 • Mission involvement – The comparator programs worked closely with USAID Missions in different countries. In the case of Local Works, Missions apply for program funds through a competitive process to be used for locally￾driven development initiatives outside their regular budgeted activities. Both Partnering for Innovation and SHOPS Plus use a buy-in mechanism, with a total program funding ceiling, whereby Missions allocate funds to participate in these programs. Buy-in funds from each Mission are used exclusively for projects in countries where the Mission is located. • Innovation stage – The programs have not focused on ideation, per se, (identifying or surfacing the “novel business or organizational models” in the definition above) but rather on creating access to, and dissemination of known solutions. The exception is Local Works, where the obverse is true: the program questions the presupposition that USAID “knows” what the key development challenge is in specific local contexts, let alone the “right” solution.129 • Procurement – The procurement mechanisms for these programs also differ from the GCs which tend to use Fixed Amount Awards (FAAs) to disburse funds to grantees on the achievement of pre-determined milestones. FAAs are made when USAID will not be substantially involved in program implementation, the total amount of the award is not capped, and funding duration is up to three years. FAAs are found to be useful mechanisms in building the managerial capacity of grantees with little experience in managing USAID foreign aid programs.130 128 E.g., in the case of Partnering for Innovation, the ‘partners’ in partnering for innovation are private sector organizations receiving program grants, and categorized into four segments: aggregators, distributors, accelerators, and acquirers. 129 ’When one knows what the problem is, and the people we are trying to help agree it’s a problem – that’s when a challenge makes sense, whereas we are stepping back and saying let’s make sure we understand the need (so much of development is based on the assumption that we already know what the problem is)’ IPE Triple Line interview with Local Works program staff 5.11.2020. 130 https://www.usaid.gov/india/partner-resources/implementing-partners-guide-types-assistance-instruments. 131 In the case of IIA, Mercy Corps received a grant from USAID under a Cooperative Agreement for the full amount of the USAID funding with Mercy Corps, and then made sub-awards to social enterprise grantees. – IIA,131 Partnering for Innovation and SHOPS Plus, by contrast, all use Cooperative Agreements as their chosen procurement mechanism. They are set up for defined periods with an overall funding ceiling, part of which is dedicated to core funding, and the remainder for field support. Co-operative Agreements are used when USAID will be substantially involved in the administration of the agreement and are made for periods up to five years with performance measured against predefined program outcomes as opposed to milestones. – Local Work funds are allocated by Congress for locally-led initiatives and can be used for grants, cooperative agreements or contracts. Funds are available for 5 years (instead of the usual 2-year expiry period) and can be deployed over time as needed, not necessarily at a set amount each year. A9.4 DESIGN DIFFERENCES RELATIVE TO GCS • A key difference between the GCs and the comparator programs lies in their focus at different stages along the innovation pathway. In general, GCs concentrate on sourcing and testing innovations, and providing technical and accelerator services that place innovators on the path to scale. The comparator programs, by contrast, either focus time and attention on defining local challenges before any search for solutions (Local Works) or, alternatively, on taking proven innovations to scale. • In the case of IIA, the closest in design to a traditional development challenge fund, the key difference from the GCs is that there was no predefined specific challenge, and the pool of solutions was known, tested, and proven in advance. Instead, each application for funding was assessed on its own merits in terms of its ability to reach scale. In other words, IIA was focused on a particular stage of the innovation cycle – scaling – rather than earlier stages of ideation, piloting, and testing. 134 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION • Partnering for Innovation, similarly, is focused on existing innovations. In this case, the original program objective was to commercialize existing agricultural technical innovations developed by USAID and other organizations for use by smallholder farmers. Like IIA, Partnering for Innovation was also focused on a stage of the cycle – specifically scaling through the adoption and diffusion of existing innovations – with a specific sectoral focus. • The program has similar objectives to the Securing Water for Food (SWFF) and Powering Agriculture (PAEGC) GCs, particularly when working at the global, rather than country, level, and worked together with these programs in an agricultural cluster. In contrast to the GCs, however, which tend to focus on surfacing and developing new solutions to long-standing problems and laying the groundwork for scale, Partnering for Innovation is focused on driving scale by getting proven innovations into the hands of smallholder farmers in a sustainable way. • A key element of the Partnering for Innovation model is that the program draws on a bank of existing solutions. These solutions are proposed by private sector innovators in response to open calls for applications, with eligibility criteria varying depending on the nature of the call (e.g., one funding round may require that applicants are locally￾registered businesses already operating in country in order to scale in-country access to existing technologies, while others may allow for not-yet￾registered companies to apply as the purpose is to bring a new technology into the country. • The innovative nature of Local Works, established as a vehicle to promote locally-led development, rests on the design and approach of the program. Local Works invests in processes to ensure joint understanding and agreement among all local stakeholders of their key development challenges before turning to solutions. The GCs also work with stakeholders (though on a global, rather than a specific, local level) to frame the problem before launching calls for solutions. In the case of Local Works, by contrast, awards are made to Missions first to establish the nature of key local development problems, and then to start identifying potential locally-led solutions. The focus, therefore, is on articulating the problem in the first place, rather than looking for specific solutions to a well-defined challenge (and the preferred solution, once there is agreement on the problem, may not necessarily be new or innovative in itself). • SHOPS Plus differs from GCs in that the program is not specifically designed to identify innovative solutions in the health sector, but rather to create an enabling environment for private sector engagement in countries where private actors have traditionally been excluded from the health system. In the case of SHOPS Plus, the design innovation lies in seeking partnership￾based, systemic solutions that widen the pool of actors engaged in development. It is also the case, however, that to the extent that private companies have innovative solutions of their own, SHOPS Plus brings them into the wider health system. The best way, therefore, to characterize these four comparator programs in relation to GCs is to see them as complementary approaches to championing innovation at USAID. They operate at different points along the innovation cycle – Local Works is at the very start of the funnel, still at the stage of formulating the problem let alone ideating solutions, while the other programs are at the far end of the innovation process, focused on dissemination and scaling of existing innovations. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 135 ANNEX FIGURE 3: COMPARATOR PROGRAMS AND GRAND CHALLENGES ALONG THE INNOVATION PIPELINE Grand Challenges Solution Ideation Piloting and Trialling Transition to Scale Problem Identification Scaled Solutions Grand Challenges Established Development Practice Stages of Innovation Comparator Programs SHOPS Plus Partnering for Innovation IIA Local Works Local Works Annex Figure 3: Legend • The GCs operate at earlier stages of innovation, identifying and testing new solutions and setting them on the path to scale. • Local Works focuses on identifying local development problems as a key requirement for funding. Once the challenge has been identified and agreed, solutions tend to draw on established development practice rather than seeking innovations. • SHOPS Plus supports private sector engagement in public health either by scaling specific innovations, or by bringing established practices into wider health systems. • Partnering for Innovation takes proven concepts and tests them in new market contexts in order to scale and commercialize these innovations for smallholder farmers. • IIA is focused on scaling proven innovations. Many initiatives have been taken up after the end of the program giving them the potential to become established development solutions. A9.4.1 KEY SUCCESS FACTORS The four comparator programs examined above have been successful in fostering innovation in different ways: • Scaling and/or commercializing existing, but not yet widely adopted or disseminated innovations. • Achieving government acceptance and uptake of these innovations by fostering public–private partnerships (SHOPS Plus), bringing government into multi-stakeholder forums (Local Works) or demonstrating the efficacy of the initiatives to the relevant government agencies (IIA). • Widening traditional approaches to tackling key sectoral challenges by expanding the pool of actors (and therefore of ideas and models) engaged in development. 136 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Across the program, a number of key success factors stand out: • Building in sustainability. • Flexible design and adaptive programing. • A focus on learning. • An ecosystem approach. • Engagement with Missions and central OUs. • A focus on non-traditional development actors. These success factors are not unique to these programs. As the final evaluation report will indicate, individual GCs have demonstrated flexibility and adaptability over successive funding rounds or adopted a learning approach over time. It is informative, however, to review how these factors have contributed to the success of the four comparator programs and to consider how their approaches and processes may be carried over and incorporated into the design of future GCs. BUILDING IN SUSTAINABILITY Long-term impact of innovations is a function of their sustainability over time, beyond the limited duration of specific programs. For GCs, where the primary focus is on proving innovations and laying the foundations for future scale, activities to promote sustainability focus on building connections and widening the network of potential future investors for individual projects. Comparator programs in the main, are focused on the uptake, scaling and commercialization of proven solutions, and therefore build sustainability measures into program design from the outset. IIA and Partnering for Innovation, which are focused on adoption and diffusion of innovations have succeeded in increasing reach and utilization of the innovations supported by the programs; Local Works reports success in creating interest in and uptake of local development practices in a wide range of Missions; while SHOPS Plus reports success both at the level of the direct beneficiary (e.g., children treated for diarrhea), and involvement of private sector actors in health systems in different countries. • Since 2018, IIA grantees have been able to maintain a relationship with the Skoll Foundation, even as USAID direct funding came to end, enabling initiatives to be tracked and supported 132 An example is the Evidence Action initiative which is now piloting a franchising model for their chlorine dispensers in Uganda. 133 Fundacion Capital (Colombia), has seen their LISAT scalable digital tool to improve financial health of people in poverty without access to bank accounts, taken up by the governments of Honduras and Mexico. Imazon (Brazil) which continues to work with the Green Municipalities Project to tackle deforestation in the state of Para, has secured an additional $21m to scale impact at municipal level using satellite imagery. 134 Partnering for Innovation Annual Report 2018-2019, p 27. beyond the lifespan of the original program. For example, Living Goods received funding from the Audacious Project, partly funded by Skoll Foundation, among others. Relationships forged with USAID units during the course of the program (as well as a better understanding of funding compliance requirements) has meant that some grantees have been able to apply for additional funding from other USAID programs: One Acre Fund received $2.88m from the Global Innovation Fund (partly funded by USAID) in 2019; and Vision Spring received $500,000 in 2019 from USAID’s Development Innovation Ventures (DIV). Grantees, such as Proximity Designs, which received a further $100,000 from DFAT through the Frontier Innovations initiative, have been able to access other sources of capital as well.132 Other IIA alumni, including Fundacion Capital and Imazon, have benefited from government uptake and endorsement of their innovations following the end of the program.133 • In the case of Partnering for Innovation, sustainability is a function of market acceptance of the innovations supported by the program. Grantees, too, must provide matching funding for each grant, and are subject to rigorous due diligence to assess their internal capability to sustain their initiatives. As the program comes to an end, all but one of the 75 partners remain in the smallholder agricultural market, 133 technologies have successfully commercialized, of which 50 percent have expanded their commercial footprint since the end of the funding partnership.134 • Local Works, in contrast to the other comparator programs, builds sustainability not through a focus on scaling of solutions, but by promoting local ownership of development challenges beyond the lifespan of any individual initiative. This is further underscored by the model of Mission participation where leadership is given to foreign service nationals (FSNs) on the basis that they are likely to have a longer tenure in office than foreign service officers (FSOs) who are in post for a relatively short period before their next rotation. Beyond the sustainability of individual initiatives, the ongoing value of these programs has been further recognized in the fact that both Partnering for Innovation and SHOP Plus have been extended USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 137 beyond their original term and will continue into the future in a new form, and with new implementation partners, but still in service to the same broad program objectives. • The original 5-year Partnering for Innovation program was extended in 2016 and will also have been in place for 10 years by the time it closes in September 2021. Its focus and objectives will continue into the future through a follow￾up program, to be called Market Systems and Partnerships (MSP), also housed in the Bureau for Resilience and Food Security, and due to run from 2020 to 2023 with a budget of ~$65m. Like Partnering for Innovation, it takes a market systems approach to private sector engagement in agricultural resilience and sustainability. The main (but not exclusive) locus of activity will be in the Feed the Future focus countries, building on the experience of Partnering for Innovation, it will be strongly oriented towards learning and building the evidence base for the role of the private sector in food security.135 • SHOPS Plus, too, is the latest incarnation of USAID’s flagship initiative in the health sector focused on private sector-led development. The initiative has been running since the 1990s, and has had multiple names over time: CMS, PSP-One, SHOPS. The program has been bid five times in 25 years136 and is due to be re-bid as a new program entitled Frontier Health Markets. The new program shares the primary objective of SHOPS Plus to increase the role of the private sector in health systems and retains the same thematic focus. It is larger in scale than SHOPS Plus (with a forecast funding ceiling of $250m). An RFI was issued in May 2020, but at time of writing it appears that the implementing partner has not yet been appointed nor total funding confirmed.137 Programs focused on dissemination and scaling of innovations, such as IIA and Partnering for Innovation, have sustainability built into program design. Success is measured on reach and uptake of innovations, rather than proof of concept or potential for scale, which remain a critical objective for GC grantees and the programs overall. Similarly, a focus on driving commercialization (Partnering for Innovation), building public-private￾partnerships (SHOPS Plus) or devolving ownership and responsibility for developing solutions to 135 https://govtribe.com/opportunity/federal-contract-opportunity/feed-the-future-market-systems-partnerships-msp-7200aa20r00003. MSP will work with DAI as implementing partner, as opposed to Fintrac, the implementing partner for Partnering for Innovation. 136 Each time, the program was won by the same implementing partner, Abt Associates, and run through a consortium of partners. E.g., SHOPS Plus had 11 partners, not including local partners sub-contracted for specific local initiatives. 137 https://govtribe.com/opportunity/federal-grant-opportunity/frontier-health-markets-fhm-rfi20ghfhm01. 138 This was a key benefit to grantees, given the range of different pathways to scale that an organization might follow. local communities (Local Works) all contribute to overall sustainability. The GCs are all committed to sustainable solutions to long-standing development challenges, but their focus is on earlier stages of innovation (see Figure 1 above). This points to opportunities for solutions to be proven through GC projects, and then taken up by other programs (within or outside of USAID), using the infrastructure they have in place to create the conditions for long-term sustainability. The four comparator programs have demonstrated that, despite the fact that program lifespans are circumscribed by procurement rules, it is possible to drive sustainability both through measures to ensure that individual initiatives are supported into the future, and through continuous evolution of the original programs themselves. FLEXIBLE DESIGN AND ADAPTIVE PROGRAMING All four programs have evolved and adapted over their course of their lifespans. While none of the programs has specifically set out in advance to be an exemplar of adaptive programing, all provide examples of adaptive programing in action. • For IIA, the fact that there was a small number of grantees (eight in total) allowed the program to adapt to individual needs and build flexibility into the terms of each sub-grant, disbursing funds on a “best effort” rather than a milestone basis. This mechanism was specifically adopted in recognition of the fact that scaling is a complex process and often unpredictable, so rigid milestones are not appropriate disbursement triggers. Furthermore, the partnership with Skoll Foundation meant that the program was able to take a relatively flexible approach to how the funds were used.138 Grantees had access both to restricted USAID funds and unrestricted Foundation funds, which they used to put in place institutional or infrastructural enablers to take their innovations to scale. • The design of Partnering for Innovation is flexible enough support the needs of very different partners: in the case of entrepreneurs, grant financing helps them take their innovations to the next level (similar to GC funding), whereas in the case of established, large-scale 138 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION agribusinesses, the benefit of the program is that it buys down their risk of trying something new or entering a new market. • Partnering for Innovation has the latitude to work with any technology or service as long as it supports smallholder farmers and is not restricted to any one focus. This allows the program to interpret the meaning of technology broadly in line with specific needs of specific Missions. For example, in Mozambique, Partnering for Innovation was able to help with cyclone relief as it addressed a critical factor affecting the enabling environment required before specific services or technologies for smallholder farmers could be introduced. • Other examples of adaptation based on changing environmental circumstances include pivoting 10 partners from their original terms to a focus on responding to COVID-19, without significant procurement constraints as all grantees were existing partners and had therefore met the due diligence requirements. • Real time learning and adaptation is also built into the design and implementation of the program. In the case of Mozambique, each round surfaced learnings that indicated the need for additional, related rounds. The program recognized that each intervention changes the market, creating new system challenges to be addressed by new layers of investment.139 • In the Philippines, the initial Local Works project focused on providing access to finance for small farmers, was changed when it became clear that the fundamental obstacle was not finance per se, but access to water. The Local Works model was flexible enough to redirect funds to more research, and a wider geographic scope, without requiring a new proposal or a substantial change in the budget.140 • SHOPS Plus, for its part, is inherently flexible in program design, as it is not focused, in advance, on specific solutions to specific problems, but rather is designed to work with different stakeholders to explore barriers and find approaches to overcoming them. In relation to USAID, if it is necessary for the program to shift course (e.g., reprograming initiatives in Madagascar and India 139 E.g., the focus on finding ways to increase the availability of specific agricultural inputs for smallholder farmers raised the upstream need for better logistics to move units to remote areas. It also highlighted both gaps in financing available to farmers at the BoP, and the lack of specific insurance products to protect farmers against climate disaster or market failures which affected their ability to repay loans or pay suppliers. In this way, an initial funding round focused on agricultural inputs translated, over time, into successive rounds focused on creating the enabling environment required for a functioning market for these inputs. 140 Following an initial funding round in 2017, and the listening tours (described above) undertaken in 2018, a project focused on analysis of water network systems analysis was launched in 2019. This was a multi-stakeholder project to try to improve failing water distribution and involved identifying the relevant actors, agreeing the key challenges and devising potential solutions. to pivot to address COVID-related challenges), a simple amendment to the work plan is all that is needed to implement the change. In the field, through its implementing partner, the program maintains continuous engagement with government and private sector actors throughout each project to review progress and make any required changes to program focus and design. • Early generations of the project with the same implementing partner helped USAID understand why the private health sector should be engaged and invested in. More recent generations of the project have focused on applying that knowledge to how to carry out that engagement. This includes their capacity for performing in-depth private sector assessments with a customizable but methodologically consistent framework that has been developed and honed over multiple generations of the project. These assessments identify barriers, opportunities and priorities for improved private health sector engagement and strengthening. The programs have significant interpretative freedom in defining their scope. IIA and Local Works are both sectorally or geographically agnostic and can support initiatives in any sector or country, providing that program selection criteria have been met. Partnering for Innovation and SHOPS Plus, which are focused on smallholder agriculture and public health respectively, nevertheless have considerable latitude within their sectors to pivot in response to evolving priorities or the need to address emerging challenges. The GCs, it is true, are considered ‘umbrella’ programs as they are also able to pivot and develop new work streams or complementary initiatives through new funding rounds. These remain within the guard rails of the challenge itself, while programs like Partnering for Innovation or SHOPS Plus, by contrast, have wide latitude for action within their focus sectors beyond the constraints of a predefined challenge. Local Works has even more flexibility as Missions can switch sectoral focus within a single award, depending on the results of the problem-definition activities. All four comparator programs have built in the necessary flexibility to respond to changing needs and operating conditions. Program evolution has been driven by changes in the way program USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 139 managers understand and define the development problem they are addressing, by new learnings that affect key elements of implementation such as stakeholder engagement, by continuous improvement measures to make programs more efficient and effective, and by external situations that force a change in direction or the terms of reference of different initiatives within each program. A FOCUS ON LEARNING Sharing learnings both within programs and across the wider Agency and development community has been another key strength of the four programs reviewed. In some cases, this learning has played itself out in a direct change of approach, while in others, learning products (articles, practitioner’s guides, market research) have become a further output of the program as a whole, in some cases taking on a life beyond the program itself. • IIA’s Scaling Pathways series (produced in collaboration with the Center for Advanced Social Entrepreneurship (CASE) at Duke University), still continues more than two years after the end of the program.141 The series was demand-driven, using survey responses from entrepreneurs across the DIV and Skoll portfolios to ensure that topic areas were highly relevant to the target audience. Uptake of materials is measured by site downloads – current metrics indicate that the series has been accessed more than 10,000 times, with qualitative feedback received from users of the materials and attendees at events organized by CASE. Most recently, Learning Pathways has created a publication page on Medium, which will provide detailed usage metrics over time. • Internally, IIA has also made changes based on learnings from the implementation process. An original top-down performance monitoring plan (PMP) for each grantee was replaced with a bottom-up, co-created plan to ensure that organizations are tracking metrics that make operational sense to the grantee as well as to the funder. • Partnering for Innovation has also produced a number of publicly available knowledge products on the process of commercializing innovations 141 Scaling Pathways has produced a number of detailed thematic reports rather than simple case studies, with the objective of addressing two key audiences i.e., social entrepreneurs wanting to take innovations to scale; and funders looking to support innovations at scale. 142 E.g., the Scale Up conference at Purdue University and the ANDE annual conference. 143 https://linclocal.org/wp-content/uploads/2019/08/LSP_Listening_ProgramDesign_Final.pdf. 144 A further key learning highlighted by LINC is the need to formalize the networks that emerge from listening and other events. E.g., LW has been encouraged to create a centralized resources space, and to connect partners at events so they remain in touch beyond their initial meeting and continue to work together. 145 The Local Works community of practice is made up of participating Missions at any given time. to reach smallholder farmers. These include a Practitioner’s Guide, market surveys for agribusinesses and smallholder farmers, articles in USAID’s Agrilinks and Marketlinks community sites, and convening or presenting at conferences, industry events, webinars and other forums.142 The program also targeted learning products towards the private sector, to build a knowledge base around targeting the smallholder farmer market with commercial offers. Partnering for Innovation plans to create a series of learning products as part of program close-out activities, and uptake and feedback metrics will be tracked once these are available to audiences. • Learning is a key element of the Local Works model. Missions receiving LW funding are expected to design their initiatives as an iterative process, learning from feedback from local actors over the 5-year funding period. From listening tours in the Philippines and Malawi, the program’s implementing partner, LINC, developed a ‘Listening for Program Design’143 guideline based on the successes and failures of attempts to reach out to new partners and designed for widespread use by other USAID Missions and programs.144 • A key element of Local Works’ quarterly Community of Practice145 meetings are the peer assistance and peer learning sessions, the topics of which are based on surveys of participating Missions to determine what they would like to learn and what they would like to share. In addition, Local Works has produced a range of materials including a toolbox of standard operating procedures and sample solicitation and communication documents designed to enhance the efficiency of successive rounds of the program; a newsletter; and guidance notes for other program managers within or outside USAID looking to replicate approaches used by Local Works. Qualitative feedback on these materials from participating Missions is positive and indicates a high level of access and usage. • SHOPS Plus has produced a body of quantitative and qualitative research on healthcare in the private sector and has created a market analysis tool for use by public health researchers. Other communications media to share information and learning generated by 140 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION SHOPS Plus include the program website and monthly newsletter (with over 100 subscribers across Missions and other OUs), other social media outlets, and participation and sponsorship of different events attended by the wider public health community. Examples include Annual Meetings of the American Public Health Association (APHA), the Global Digital Health Forum events, the International Conference on Family Planning’s virtual forum, and the Global Health Practitioner Conference at which learnings and research are shared with and disseminated among donors, UN officials, academics, private sector and government representatives, and healthcare workers. • Through to 2019, SHOPS Plus had recorded ~240,000 pageviews on the program website and 10,000 downloads of SHOP Plus publications. • SHOPS Plus has also focused on internal learning. In 2018, a USAID commissioned mid-term evaluation of SHOPS Plus identified a number of key findings on topics such as gender and youth focus. SHOPS Plus worked with consortium member Iris Group to develop a gender strategy, while the finding that the youth demographic was generally highly underrepresented in program initiatives, led to a youth engagement program in the SHOPS Plus work plan for India. A common characteristic of the approach to learning among all four programs is that it is utilization focused, with different objectives and directed at both internal and external audiences: • Learning in action – learnings from the program are used internally to course correct (e.g., Local Works in the Philippines); deepen impact (e.g., Partnering for Innovation making awards to a wider ecosystem of enablers in Mozambique to support provision of inputs to smallholder farmers); or using insights and experience in one territory to inform interventions in another (e.g., using a model for tackling TB developed in one country to address a similar development in another context146). • Learning for future programing – using learnings from the different programs, captured in annual reports and program evaluations to inform future incarnations of individual programs. This is the case in the development of the new Market Systems and Partnerships or Frontier Health Markets programs within the Food Security and Global Health Bureaus, respectively. Similarly, 146 SHOPS Plus leveraged experience from a successful initiative to combat TB in India into two states in Nigeria as part of a multi-actor network for reporting, diagnosing, treating and following up TB. outside of USAID, learnings from IIA were taken into Skoll Foundation’s current collaborative funding partnership, Co-Impact. • Learning as public good – here the emphasis is on practitioners’ guides and practical advice to different development actors than on academic research, with examples from all four comparator programs. Program officers participate in forums and communities where learnings are shared with people who sit outside the program itself, but nevertheless face similar challenges and objectives. The approach to learning taken by the four programs under review widens the knowledge base inside and outside of USAID and enhances USAID’s role and status as a thought leader in specific areas and strengthens the individual programs by giving them a platform and a pulpit. More inward-focused learning activities also act as a scaffolding within programs, as grantees share knowledge and ideas, strengthening their own initiatives with the hard-won experience of their peers. Both the GCs and the comparator programs emphasize the importance of learning and producing utilization focused products with practical advice for grantees, program managers, and other development practitioners both within and beyond USAID. These products may be outward looking, shared with the wider development community to raise awareness of both the challenge and the range of solutions sponsored by the program, or inward focused to adapt and refocus the scope of the program itself. Comparator programs have used surveys to determine what aspects of program learning is of highest value to different audiences and have tracked both quantitative metrics and qualitative feedback to determine levels of usage and utility. There is scope for GCs to adopt a similarly targeted approach to creating and tracking their own learning activities to ensure a return on effort and investment. There is also clear scope for increased dialog and cross￾learning between the GCs and the comparator programs, both in terms of program content (exploring synergies between programs with similar objectives working in the same space) and in terms of approach to learning with opportunities for GCs to explore the use of tools, forums and approaches described above. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 141 AN ECOSYSTEM APPROACH The four programs under consideration in this paper have all incorporated ecosystem analysis and engagement to varying degrees. In the case of IIA, scaling pathways have necessarily involved engagement with government and other bodies whose support and endorsement are required for specific innovations to continue scaling after the end of the program. It should be noted, however, that this level of engagement was the responsibility of individual grantees rather than the program funders or implementing partners. Nevertheless, successful efforts to engage with and win support from system players resulted in successful uptake of innovations beyond the life of the program itself (see Section 1: Building in sustainability, above). For Partnering for Innovation, by contrast, the program as a whole engages with different players depending on the focus of individual funding rounds (e.g., in the case of vaccines, the program would work with governments and research institutions; to overcome financial barriers, the program would engage financial institutions, or logistics firms when addressing supply chain bottlenecks). Similarly, the program explicitly adopted a portfolio, or ecosystem, approach to funding grantees, looking not just at the results and impact of individual grantees, but at the impact of all grantees working in tandem (sometimes in competition) to meet program objectives. The degree to which Partnering for Innovation has been able to take a wider ecosystem approach across all initiatives is, however, a function of level of Mission buy-in to the program. In the case of a Mission buying into a single funding round, engagement will necessarily be limited, whereas in Mozambique, where the Mission has participated in multiple funding rounds directed at different players along the smallholder supply chain, the program has been able to take a whole system approach to driving innovation to farmers at the BoP. An ecosystem approach is fundamental to the mission and design of the other two programs that are focused on finding systemic solutions to development needs and ensuring that a full complement of stakeholders is involved in agreeing and co-designing responses to these needs. 147 E.g., in Myanmar, USAID/Burma used the funding from Local Works to address the heroin epidemic in Kachin State. They spent three months scoping the challenge listening to local actors, and then convened a workshop of 111 people from civil society, faith-based organizations, government, charities and academia. These stakeholders later coordinated their efforts to create a system-wide approach, with initiatives such as women’s rehabilitation centers, employment schemes, awareness raising and expanded access to health care. 148 E.g., in Nigeria, initiatives focused on integrating private facilities into national Health Management Information Systems, working with local governments to ensure local area data was being integrated into national dashboards. Similarly, an initiative aimed at combating TB was a public-private partnership, with the government providing the drugs and the private facilities providing the services – the whole overseen by clinical associations. • Local Works program officers travel to different communities on ‘listening tours’ to hold dialogues with local actors on their experiences and priorities. Another tool for building relationships across ecosystems is the use of ‘Whole System in the Room’ Workshops, where USAID convenes all actors affected by a common issue, to jointly lead the development of solutions.147 • SHOPS Plus, too, engages with players across the public health ecosystem when designing and implementing projects in specific countries. SHOPS Plus works with civil society organizations and media companies to spread healthcare messaging, with national governments to influence policy and support the contracting of NGOs or other non-state healthcare providers, and with local governance institutions to support public￾private partnerships.148 The four programs have demonstrated the value of considering broader ecosystem effects when designing development interventions in different contexts and countries. The sustainability of innovative solutions through uptake and adoption by users and funders is dependent on widespread understanding of the benefits of these solutions which, in turn, is dependent on understanding systemic impacts. Which parts of the system benefit and which may feel compromised? What enablers and infrastructure needs to be in place for the innovation to have effect, and how might the innovation act as an enabler itself, for other parts of the system? The more these complex intersections are understood and incorporated into program design, the greater the prospect for long-term impact. Ecosystem development puts in place a wider infrastructure of enablers needed for innovations to have effect beyond the life of any individual program. In this sense, focus on ecosystems and on long-term sustainability go hand-in-hand. For Partnering for Innovation, SHOPS Plus and Local Works, building this infrastructure is an explicit element of program design, whereas for the GCs, this was not a specific focus at the outset. The power of ecosystem engagement, however, is well understood and is being incorporated into new versions of the GCs. For example, Water and Energy for Food (WE4F), is building on insights from its predecessor challenges 142 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION to establish regional hubs and the kind of big structural institutions that will endure well after the GC itself has wrapped up. ENGAGEMENT WITH MISSIONS AND CENTRAL OPERATING UNITS Three of the four comparator programs (the exception is IIA), were all specifically designed as central programs that work directly with Missions. Local Works provides direct funding to Missions, while Partnering for Innovation and SHOPS Plus provide financing facilities that USAID Missions, as well as central OUs, can buy into as often as they want within the constraints of the timeframe of the program, or annual spending caps. These models allow individual Missions to adapt their scope and focus in successive funding rounds based on learnings from earlier rounds. To date, 15 Missions have bought into Partnering for Innovation, with 32 accessing Local Works. SHOPS Plus has set up field offices in nine countries and conducted projects in a further 10. Centrally, the Latin American and Caribbean (LAC) and the RFS Bureaus have accessed Partnering for Innovation, while SHOPS Plus has a number of central clients including the Office for Humanitarian Assistance (OHA), the Office for HIV AIDS, and the Maternal and Child Health and Family Planning unit within the Bureau for Global Health. • Partnering for Innovation program officers spend time at the outset with Missions to understand their needs and agree how the program can mitigate their challenges. Some Missions, such as Mozambique, have bought in on multiple occasions, while others have engaged just once. A number of Missions chose not to buy into the program either because the objectives of Partnering for Innovation were not aligned with their own development priorities, or because the program was perceived to be duplicative of existing initiatives. • Overall, however, engagement with Missions has been effective as Partnering for Innovation is able to show that the program is complementary to in-country initiatives and could help scale the impact of that work. Mission staff are also involved in the design and solicitation process for different funding rounds, and Chiefs of Party (COPS) for in-country programs sit on the 149 A portion of funding was set aside for unsolicited proposals, but these would still have to be directed at specific, locally-led initiatives. 150 Mission funding goes to TA in the form of MEL, communications, policy, context-specific research and funding for listening tours to understand the nature of local needs. technical evaluation committees for assessing applicants. • Local Works was specifically designed to work with Missions wanting to trial or take further locally-led development initiatives.149 The program provides funding, flexibility and technical assistance,150 and works with a minimum of 3 Missions per annum, connecting them through a community of practice supported by a newsletter and a quarterly call. Missions also benefit from the program umbrella Annual Program Statement (APS), putting in addendums when soliciting for local applicants for funding rather than having to create an APS of their own. • SHOPS Plus is able to run 8-10 projects simultaneously (current projects span the globe from the Caribbean to Africa and East Asia). SHOPS Plus provides a field support mechanism for Missions buying into the program, and supports Missions to increase their capacity and understanding of the requirements of private sector engagement and market coordination. While not every Mission or OUs chooses to engage with these programs, those that do benefit from an easy-to-use mechanism. Most importantly, given that Missions resources have limited bandwidth, the ability to tap into a central procurement mechanism meant that Missions did not have to invest the time and resources required to set up programs of their own. Program focus on public-private partnerships makes it easier for Missions and other OUs to establish direct connections with private sector partners. Conversely, these centrally located programs benefit from the ability to go both broad (at regional or global level through buy-in by central OUs, and deep (focused impact at local level through Mission uptake). In contrast to GCs, which solicit Mission engagement and interest, but do not directly build Mission involvement in program design, direct awards to Missions through Local Works or the funding ceiling for Missions to buy into Partnering for Innovation and SHOPS Plus reinforced the link between these central programs and Missions and ensured that there was alignment of objectives and priorities. A FOCUS ON NON-TRADITIONAL DEVELOPMENT ACTORS Like the GCs, these programs tap into the ideas, creativity, and experience of non-traditional USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 143 development partners — entrepreneurs and innovators within social enterprises, local actors and communities, NGOs and private companies — to find new solutions for intractable development problems. This focus on non-traditional players has enabled the different programs to open up spaces traditionally excluded from development activity: commercial supply chains serving smallholder farmers; private sector participation in and contribution to public health initiatives; and direct engagement with local communities to address local challenges. All grantees of IIA had pre-existing relationships with Skoll Foundation, and several with USAID as well, but, as social enterprises, were nevertheless non-traditional partners for USAID. The fact that there were pre-existing relationships with these organizations, however, was a key driver of program success as this simplified procurement and due diligence processes. It was also possible to leverage insights from DIV and other programs to determine whether proven innovations were ready for scale and thereby create a pipeline of potential awardees in line with program objectives. All except one of the grantees of IIA had pre￾existing relationships with Skoll Foundation, and several with USAID as well but, as social enterprises, were nevertheless non-traditional partners for USAID. The fact that there were pre-existing relationships with these organizations, however, was a key driver of program success as this simplified procurement and due diligence processes. It was also possible to leverage insights from DIV, from Skoll Foundation’s pre-existing relationships with grantees, and other programs to determine whether proven innovations were ready for scale and thereby create a pipeline of potential awardees in line with program objectives. The other three comparator programs were tasked with bringing non-traditional players into the development community as an explicit component of program purpose and objectives. Tapping into a wider pool of players was expected to bring greater understanding of the challenges facing different communities, and access to a wider set of solutions to long-standing development problems. • Partnering for Innovation targeted actors not traditionally funded by USAID. Whenever a new solicitation round was launched, Fintrac, the implementation partner, spent time on the ground in target countries raising awareness of the program. Program officers visited trade fairs, placed advertisements in trade magazines to garner interest amongst potential partners with no previous relationship with USAID and who were not already receiving funding through other in-country programs. • Local Works was established as a Congressional program to facilitate participation by smaller players who traditionally struggle to compete with large organizations who typically win contracts and enter into cooperative agreements with USAID. Key eligibility criteria included provisions that participating organizations must have received less than $5m in funding from USAID in the past 5 years, and that they had close links with the local communities benefiting from the funding. • The actors targeted by SHOPS Plus were not organizations that would otherwise be reached by USAID, as in many LMICs – in Sub-Saharan Africa and West Africa in particular – private sector organizations tend not to be recipients of donor funding. e.g., SHOPS Plus supported a private pharmacy association in Tanzania to build capacity as a procuring agent and benefiting members who would not typically be reached by USAID assistance. A focus on non-traditional players echoes the open innovation approach taken by the GCs, giving smaller players a greater say and stake in how development programs are designed and implemented. This approach has enabled the different programs to open up spaces traditionally excluded from development activity: commercial supply chains serving the BoP; private sector participation in and contribution to public health initiatives; and direct engagement with local communities to address local challenges. A9.5 CONCLUSION: QUESTIONS RAISED FOR GCS BY THE EXPERIENCE OF THE COMPARATOR PROGRAMS IIA, Partnering for Innovation, Local Works and SHOPS Plus — whose purpose, approach and activities may be seen as complementary to GCs — have all been successful when measured against their program objectives. They have achieved reach, adoption and scaling of innovative solutions and 144 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION approaches, and laid the basis for long-term impact. In the process, these four comparator programs have provided examples of: • Scaling – The variety of pathways that may be followed to take innovations to scale, and which pathways are appropriate under which circumstances. • Serving the smallholder farmers – How innovations can be used to build markets not traditionally served. • Adaptability and an ecosystem approach – The value both of program adaptability to new information and unforeseen circumstances, and of taking an ecosystem approach to lay the foundations for enduring change. • A utility-based approach to learning – How to use learnings to improve program performance ‘in-flight’, create linkages between grantees, and share insights with the wider development community. • Mission and OU engagement – The benefits of creating programs based on buy-in by Missions and other OUs to achieve both depth and breadth of uptake of development innovations. • Non-traditional players – The value of challenging entrenched attitudes and perspectives around who should be involved in development activities and how to engage them. As USAID takes forward the Grand Challenges portfolio of programs into the next decade of design, partnership and implementation, it is useful to reflect on design elements from these comparator programs that could be incorporated into future GC design. Specifically, we propose that the GCs consider the following questions for further discussion: • Ecosystem Focus: Innovator Engagement – Many innovations supported by GCs are at an early stage of development, undergoing a process of test and trial before being put forward for scaling. Can, or should, the grantees engage with a wider ecosystem at this early stage to generate knowledge of, and interest in these innovations to create opportunities for uptake and endorsement once efficacy has been demonstrated? – A GC could build both a demonstrable understanding of, and engagement plan with a wider ecosystem into the eligibility criteria for making awards and, if not already doing so, 151 An example of such infrastructure is the hub set up by PAEGC to support grantees with their energy solutions by providing them with information and assistance in negotiating the tax implications of importing solar pumps. monitor levels of engagement as a key metric of success for each innovation. • Ecosystem Focus – Grand Challenge Engagement – How might GC program design be adapted to build a more supportive and enabling infrastructure around the portfolio of innovations? – The GCs could allocate resources and funds specifically for ecosystem engagement through different vehicles such as awareness raising, creating infrastructure to provide shared resources and information required for long-term viability of innovative solutions,151 or influencing the regulatory environment governing the operating environment for grantees. • Mission engagement – How can GCs encourage Missions to participate in the design of challenges and become actively engaged in supporting grantees implementing projects in their territories? – GCs could ring-fence part of their funding to allow Missions to buy-in for different funding rounds. Access to these funds would follow an application process similar to the one used by Local Works, whereby Missions demonstrate that they face the specific challenge which is the focus of the GC and would be keen to participate through a GC project in their own territory. Funding of these country-specific projects could be through the same grant mechanism used for all awards, or through a buy-in mechanism in order to guarantee that Missions have a direct stake in the program. • Finding synergies with other innovation programs and models – How can GCs incorporate learnings from Local Works (e.g., the listening tours and ‘system-in-a-room’ workshops) to map locally-led development approaches onto the scoping and implementation of challenges in specific countries? Alternatively, is there scope for linking the Local Works approach with the GC mechanism to surface innovative solutions to local challenges, where no clear solution has yet been identified by local stakeholders? – Working together, Local Works, or a Local Works-type process, and EPIC could explore the use of a new or existing GC to call for local solutions to a locally-defined challenge. This would enable the GC to go deep within a specific country to engage with a local USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 145 ecosystem to build an enabling environment for innovators, and would support a cohort of non-tradition, local entrepreneurs to test their context-specific solutions with their own local communities and build sustainability through local involvement and ownership of the solution This model would also enable the GC to engage and partner with the local Mission to support the implementation of particular innovations in the local context, and to demonstrate a complementary relationship between Mission priorities and programing, GC objectives, and a commitment to locally￾led development. The value of comparator analysis – reviewing how different programs across USAID have approached and tackled innovation in development – is to share ideas and experience and find opportunities for mutual learning and improvement. The findings in this paper, and the questions put forward for consideration by GCs and the wider innovation community of practice at USAID, are intended to contribute to this dialogue, sparking new ideas and identifying opportunities for synergies across the Agency. Photo: Global Devlab/USAID 146 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Photo: Global Devlab/USAID USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 147 ANNEX 10  COST-EFFECTIVENESS ANALYSIS REVIEW 152 USAID (2020) Cost Analysis Guidance for USAID-Funded Education Activities 153 Usmani, S and McMahan, K (2020) The Economic Benefits of Supporting Private Social Enterprise at the Nexus of Water and Agriculture: A Social Rate of Return Analysis of the SWFF Grand Challenge. 154 Duke Global Innovation Health Center (2020) Evaluating Saving Lives at Birth- Evaluation Report Rounds 1-8 (2011-2020). 155 USAID (2015) USAID Guidelines Cost-Benefit Analysis. A10.1 PURPOSE The cost-effectiveness MEQ of the USAID GC meta-evaluation (Section 4.9) poses a question which contains two elements: How feasible is it to measure the cost-effectiveness of previous and future GCs and compare the cost effectiveness of GCs to traditional program models? Following the initial consultation and document review conducted by the team, it became clear that whilst there is a strong interest in this area, there is insufficient project management data for most of the GCs to enable a systematic measurement of cost effectiveness. This was further clarified at the Prioritization Workshop on July 27, 2020, which identified that while a number of evaluations and research activities have been undertaken to assess the outcomes and impact of some of the Challenges, there is generally a lack of data to compare these results on the output side with the costs on the input side. The simple answer to the feasibility question is that a comparative assessment of the cost effectiveness of the challenges compared to traditional methods would require a lot of primary data collection as well as a reconstruction of annual project management data. Measuring cost effectiveness is very difficult if the GC implementers have not been requested to present the costing of activities in their reporting to USAID. The overall approach agreed at the USAID Prioritization Workshop is therefore to be forward looking in setting out a practical methodology for calculating cost effectiveness for the respective GCs and provide guidance on how unit costs can be defined, derived, and potentially compared with other programs. This note aims to build on the considerable work that USAID and the Challenges have already completed notably in the Education sector152 and Agriculture/Water sector.153 Some of the GCs have undertaken some cost effectiveness analysis notably SWFF and Saving Lives at Birth.154 But the GCs are currently lacking an institutionalized process of systematic activity-based costing connected to theories of change/impact pathways. This paper outlines a brief summary of the issues in measuring cost effectiveness from reviewing the GCs and best practice. The separate CEA Framework document analyzes how cost effectiveness could be established for different sectors and stages of challenge from early-stage innovations to challenges at scale where there is a greater emphasis on relating costs to outcomes. The intention is that the revised framework could also be applied to the current GCs. A workshop was held on 22 December 2020 with USAID to establish key priorities for the CEA Framework document that has been produced as a separate document. A10.2  DEFINITIONS AND SCOPE A10.2.1  KEY TERMS: CBA, SROR, AND CEA Cost Benefit Analysis (CBA): A CBA measures the stream of cost and benefits over the lifetime of a project. Importantly it discounts future costs and benefits and can therefore be used to compare projects of different lengths and sectors. CBA requires the monetizing of all benefits including items such as a life saved, or a life improved, into a monetary value. Social Rate of Return (SROR): The SROR approach is similar to CBA but adjusts the value of the benefits to take account of additional non￾monetary social costs and benefits to society. It can involve using “shadow prices” to, for example, give greater weight to the incomes and benefits of a poor or marginalized target group. But shadow pricing can also be applied to CBA and therefore there is very little difference between the two approaches and details of how and when to use CBA are clearly set out in the USAID155 guidelines. 148 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION If the main objective is to either measure the overall economic contribution of the GC, or to compare different GCs with different time streams in terms of costs and benefits, then a CBA or the Social Rate of Return (SROR) approach as undertaken by SWFF, is the route to take. The CBA or SROR approach expresses all costs and benefits in dollar terms, which allows a comparison of the relative costs and benefits of X program to Y program even if the two programs are totally different. When the benefit stream cannot be monetized a CEA approach can be adopted. Cost Effectiveness Analysis (CEA): A CEA provides a comparative measure of the overall cost of achieving a development outcome and is typically expressed in terms of the cost of the outcome per beneficiary and reached. As expressed in the USAID guidelines: “CEA seeks to determine the lowest unit￾cost alternative for achieving a desired outcome”.156 The guidelines add that “CEA should be used where it is difficult to place a dollar value (i.e., monetize) the project benefits- as is often the case with nutrition, health and global climate change projects.” CEA enables 156 USAID (2015) Op cit. 157 Tangoren C (2020) Infinitely cost-effective? Common misconceptions in cost-effective analyses. JPAL Blog posted 15 September 2020 (www. povertyactionlab.org). 158 While in most cases CBA uses market prices and actual costs. Some CBA use ‘shadow’ prices to adjust for the social benefits of outcomes. This is termed Social Cost Benefit Analysis and is then similar to a SROR. comparison of the costs and outcomes of different projects or challenges, and for example compare the unit cost of saving one life through improved health care to the unit cost of a life improved as a result of improved education, as highlighted by Tangoren (JPAL)157 who elaborate: “This allows us to compare interventions without imposing our own value judgments about the importance of different outcomes. CEAs are unable to capture all benefits of a particular program. Instead, they compare interventions on a single common outcome”. The aim of this meta evaluation assignment is not to be able to compare the economic value of, for example, Health vs Education Challenges, but to gain a better understanding of how USAID could work towards improving an understanding of the overall value for money of the GC investments. A CEA methodology that is applied consistently across the Challenges is required. Where it is possible and the outcomes can be measured in $ terms, a full CBA or SROR should be considered. The differences between CBA, CEA, and SROR are summarized in Annex Table 7 below. ANNEX TABLE 7: COST BENEFIT ANALYSIS COST EFFECTIVENESS ANALYSIS AND SOCIAL RATE OF RETURN SOCIAL RATE OF RETURN COST BENEFIT ANALYSIS COST EFFECTIVENESS ANALYSIS Activities Costed Yes Yes Yes Outcomes Measured Yes Yes Yes Outcomes Monetized Yes Yes No Activities costed over the length of the project and discounted to Net Present Value Yes Yes No Outcomes monetized over the length of the project and discounted to Net Present Value Yes Yes No Monetized outcomes adjusted for wider benefits to society Yes No158 No USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 149 A10.2.2  WHAT COST EFFECTIVENESS ANALYSIS CAN BE USED FOR Annex Table 8 describes USAID’s Guidance on Cost Analysis for Education159 (see Annex Table 9 in the Appendix for further details). More generally, the introduction of cost analysis can lead to a broader culture of considering cost effectiveness across all aspects of the design, implementation of evaluation of programs. A number of donors have guidelines for undertaking cost effectiveness and provide some useful guidance. A summary of the approach adopted by the UK Government is set out in Annex 1 and shows a useful graphical summary on conducting cost effectiveness. As stated in the USAID Guidance, CEA requires the combining of two types of information: the cost efficiency of the intervention (e.g., the cost of training a teacher) with the effectiveness of an intervention (improved learning outcome per teacher trained). This can then be combined as a ratio (the cost per increase in learning outcomes). While a value does not necessarily need to be placed on the benefit or outcome, there is still a need to find some way of identifying the end beneficiary and attributing the impact pathway to the GC. There are therefore two key requirements here: 1. A clear theory of change and results framework that maps challenge activities to intermediate outcomes as well as other assumptions and influences from the ecosystem. 159 USAID (2020) Cost Analysis Guidance for USAID-Funded Education Activities. 160 Grand Challenge Evaluation (2020) Prioritization Workshop. 161 Brown, E and Tanner J (2019) World Bank IEG Paper 9041 Integrating Value for Money and Impact Evaluations. 2. The collection of precise activity costs data by the implementers of the Challenge. As discussed in the prioritization workshop,160 It is very difficult to conduct an ex-post analysis of cost effectiveness if the activities to produce the outputs have not been costed. Ideally, the theory of change should be supported by an impact evaluation to measure the influence of the challenge. But many observers have commented that there is insufficient consideration given to cost effectiveness in the design of impact evaluations. Elizabeth Brown and Jeffrey Tanner (2019)161 note that fewer than one in five impact evaluations conduct any consideration of cost effectiveness or value for money. The issue appears to start not with a lack of interest by the donors, but the failure “to consistently demand cost analysis be integrated into the funded evaluation”. Brown and Tanner call for the “establishing of standards in what constitutes rigor, resolving methodological issues and improving linkages between policymakers and researchers”. Even if a full impact evaluation is not commissioned, CEA requires a clear results framework which shows a credible link from the GC to the outcomes. A key requirement moving forward therefore is that cost considerations are established at the outset of the project cycle in the design phase and that cost analysis is a key part of the research question posed for evaluators. In order for these results frameworks to be consistent across Challenges there is a need for some standardization on indicators, definitions of costs, methods of analysis, and reporting by the Grand Challenges. ANNEX TABLE 8: COST EFFECTIVENESS ANALYSIS WHAT QUESTIONS CAN IT ANSWER? WHAT CAN COST ANALYSIS HELP ACHIEVE? WHAT DATA WILL ANALYST NEED? What did this intervention cost per outcome delivered? How does that compare to other interventions that produce this outcome? Compare costs of outcomes across different interventions. Identify the intervention that achieves the greatest outcome within a given expenditure per beneficiary. Expenditure and contributions reports disaggregated by cost categories and ingredients; a method for allocating shared costs across cost categories. Credible estimates of the impact. Credible estimates of the cost and effects of comparable interventions. Source: USAID (2020) Cost Analysis Guidance for USAID-Funded Education 150 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Interestingly, Brown and Tanner highlight the need to “develop closer ties to policy makers in order to understand their demand for information”. Lowering research costs was also identified as another priority “which requires investment in promoting standardized methods and promoting operationally relevant VfM analysis in impact evaluation”. A10.3  CURRENT STATUS OF COST-EFFECTIVENESS IN THE GRAND CHALLENGES A10.3.1  KEY OBSERVATIONS There are two clear observations on the cost effectiveness assessment practices in 10 Grand Challenges under review. Firstly, there was no clear cost effectiveness mandate set for the Grand Challenges. As a result, there is limited data/ information available to assess the cost effectiveness of the GCs and the interventions/grants funded by the Challenges. Although some of the GCs reported costs, they were not required to relate different elements or activities of the cost with the outcomes. Considerable work has been done by USAID notably in the Education,162 Agriculture, and Water sectors163. Some of the GCs completed CEA, notably Securing Water for Food (SWFF) and Saving Lives at Birth (SL@B), and others, including Powering Agriculture (PAEGC), have undertaken some case studies on cost effectiveness, notably the costs and benefits of clean energy technologies in the milk, vegetable, and rice value chains. Experience from these GCs has shown that applying CEA generates a better understanding of the wider objectives of the Grand Challenge and in identifying innovations that will have an impact at scale. For example, at SL@B: “CEA process itself helped the innovators to more deeply understand their program and think about how the results would support the scale-up pathway”.164 Additionally, All Children Reading (ACR GCD) had a results framework (developed for the third round) that identified program-specific output indicators that are measurable, such as total number of people/stakeholders trained, as well as outcome and impact indicators that required 162 USAID (2020) Cost Analysis Guidance for USAID-Funded Education Activities. 163 Usmani, S and McMahan, K (2020) The Economic Benefits of Supporting Private Social Enterprise at the Nexus of Water and Agriculture: A Social Rate of Return Analysis of the SWFF Grand Challenge. 164 Duke University (2020) Insights and Lessons Learned from the E-SL@B team on Conducting CEA. 165 Department of Foreign Affairs and Trade. AusAID was merged with DFAT in 2013. 166 Foreign and Commonwealth Development Office. The former Department for International Development (DFID) was merged with the Foreign and Commonwealth Office in 2020. rigorous measurement at the grantee level. Grantee￾level evaluations were conducted to evaluate whether the grantees met specified targets. While this provided a basis to evaluate cost effectiveness of a selection of ACR GCD grants, in the absence of a consistent cost effectiveness design, these estimates are not likely to be comparable between grantees. Annex 12 sets out more detail of the current status of cost effectiveness of the GCs. Secondly, there was no uniform guidance at the level CEA is applicable in the GCs. Although the general approach to design a Grand Challenge has been to ensure that the Challenge remains cost effective, there was no clarity on whether the priority will be to measure cost effectiveness of the individual grants or for the overall Grand Challenge. It is also noted that there is a wide range between the GCs, both in the emphasis and budgets for monitoring and evaluation and data collection. There were mixed views from the GC Managers about the relevance of cost effectiveness for the overall Grand Challenge and the extent to which it should be undertaken. Some regarded the cost effectiveness of an individual grant or innovation made more sense since the prime objective of a Grand Challenge is to identify impactful and cost￾effective approaches and innovations that can be scaled and replicated. Some of the usual cost effectiveness outcome indicators being used by USAID for programs do not apply to the GCs. For example, the standard USAID indicator for the number of direct and indirect beneficiaries may not be applicable for early-stage Challenges, since the focus of an individual grant at the innovation and testing stage is more concerned with evaluating the effectiveness of the scheme than scaling it up to reach a wide number of beneficiaries. This calls for a flexible approach to CEA where stronger emphasis is given to measurement of the overall Challenge for early-stage GCs and more emphasis on the CEA of grants for later stage GCs. It is noted that funding partners, notably Australia’s DFAT165 and UK’s FCDO,166 have influenced the thinking around CEA. For example, DFAT, funding partner to ACR GCD, maintains a separate value for money analysis to circulate internally, and places emphasis on the CEA of grantees to USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 151 identify whether the innovations/interventions can be replicated elsewhere. The practice of cost effectiveness was, however, largely anecdotal, comparing the total investment with overall achievement of the GC and not to specific outcome indicators. A10.3.2 KEY OPPORTUNITIES There are two clear opportunities to build on the CEA activities that have been started in some of the nine GCs under review. First, a GC should aim to ensure an approach which is broadly comparable with the unit cost parameters of similar interventions in the sector. Among the GCs that have applied some CEA analysis, there is broad representation across a number of sectors. This will enable considerable learning for future GCs and for benchmarking outcomes: Agriculture/ Rural Development (PAEGC/SWFF); Humanitarian Assistance and Governance (Creating Hope in Conflict); Health (SL@B); and Education (ACR GCD). It is recommended to adopt cost-effectiveness benchmarking techniques. This involves identifying suitable cost effectiveness parameters (outcome indicators) for the Grand Challenge and benchmarking them with the standards from similar projects/Challenges and/or exploring the available CEA estimates for the selected outcome indicators in literature.167,168 The standardized cost effectiveness parameters for innovations/interventions169 obtained from literature or cost effectiveness registries for benchmarking could assist in determining: 1. Percentage of total GC Budget for GC Management versus Grants. 2. Proportion of direct (i.e., to the grantee) to indirect costs within the GC. Second, there is a need to separate out grantee level cost effectiveness assessment from the GC level assessment, right from the design stage. There was broad agreement among the GC Managers that CEA is an essential part of the management of the GCs. One key observation from a number of GCs is that a CEA Framework should determine a number of key parameters including: the future size of the Grand 167 https://www.cgdev.org/publication/committing-cost effectiveness-USAID’s-new-effort-benchmark-greater-impact. 168 Tufts University, for an example, maintains a registry of cost effectiveness estimates available for various commonly used outcome indicators. https://cevr. tuftsmedicalcenter.org/databases/cea-registry. 169 https://www.who.int/bulletin/volumes/93/2/14-138206/en/. 170 This was also confirmed at the Grand Challenge Evaluation workshop on Cost Effectiveness held in December 2020. 171 https://www.povertyactionlab.org/resource/conducting-cost-effectiveness-analysis-cea. Challenge, size of the grants, and a more optimal balance of resources between activities both for the GC manager and for the Grantees. The decision to give more emphasis to CEA at the grantee or GC level could depend on the size of the Grand Challenge, the objectives related to scaling and replication, and the size of administrative or other indirect support costs. These two layers of cost effectiveness assessment could have some overlap in terms of outcome indicators and methodological approaches, but it is important to establish the emphasis at the design phase. What is required is a CEA framework and methodology that is applied across the full length of the Grand Challenge project cycle. This has implications for the allocation of responsibilities and accountability between various parties. A10.4  PROGRAM MANAGEMENT ISSUES OF CEA The key message from the literature170 is that a cost effectiveness culture needs to be established from the outset of the challenge with the activities conducted by the GC identified and defined. Most important is to define the key cost analysis research questions that need to be answered by the GC or research team. The preparation of the data and the primary analysis of the costs therefore needs to be undertaken by the GC implementer. One of the key benefits of introducing CEA to the GCs is that it leads to a greater attention and focus on controlling costs and holding implementers to account for efficient delivery. A10.4.1  ACTIVITY COST PLANNING There are a number of guidelines on how to deal with contentious issues such as costing in-kind contributions, management costs, and cost collection methods. These costings will be partly dependent on the donor’s public accounting protocols and practices. JPAL171 provides one such guide which sets out a number of methodological issues on costings including overhead costs, discounting to present value, use of exchange rates, and inflation 152 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION assumptions; all of which are critical in developing a standardized approach for comparing challenges. The USAID guidance for Education (op. cit.) also provides a good start for this in defining the different elements of labor costs, operations, and overheads. It has also set out a very useful 6 step guide for a cost data analysis plan which could be established at beginning of the GC:172 1. Preliminary data checks: Review all expenditure reports and check for data gaps. 2. Review of cost analysis questions: Review existing cost analysis questions. 3. Develop a cost analysis plan: Select cost analysis methods. 4. Prepare data for analysis: Prepare the selected data for analysis using standard procedure. 5. Implement analyses: Build worksheets and analyze data. 6. Report and document: Codify the results in a cost analysis final report. A10.4.2  MEASUREMENT OF OUTCOMES As highlighted above, it is essential that each GC develops a results framework tracking the outcomes of the challenge, supported in an ideal situation by an impact evaluation. The CEA can then analyze the unit cost of achieving the defined outputs and outcomes. The common unit measurement typically used is cost per beneficiary at both output and outcome level. While simple enough as a concept, it is essential that there is some standardization of the definition of a beneficiary: has the GC reached the beneficiary directly or indirectly? Do we count households or individual people? USAID/DIV173 has developed the concept of ‘unique direct beneficiaries reached’ in standardizing the understanding of this definition while enabling some flexibility in establishing the precise scope and boundaries with the challenge implementer. This approach critically relies on continuity within USAID to ensure that there is consistency between GCs in the boundary set in defining direct and indirect beneficiaries. 172 USAID (2020) Cost Analysis Guidance for USAID funded Education Activities. 173 USAID/DIV (August 2020) DIV Beneficiary Measurement. Internal Paper. 174 https://www.globalinnovation.fund/practical-impact-assessment/. 175 USAID Education guidance uses the term dosage instead of depth. A10.4.3  SCOPE AND TIME HORIZON Any CEA needs to have a standard approach in comparing the stream of benefits. Under a CBA approach this stream is aggregated over the project lifetime and then discounted to a Net Present Value. CEA has greater flexibility in being able to analyze the unit cost of beneficiary reached on an annual basis or over the lifetime of the GC. Nevertheless, there is a need for some issues of consistency in selecting the time horizon for comparing interventions. The USAID-funded Global Innovation Fund has produced a simple and practical tool174 described as the Practical Impact Assessment (PIA). This assumption-based tool converts all impacts into a standard unit of measurement (PIA) and calculates the breadth and depth175 of impact into a standard unit and the probability of its achievement. It also fixes a 10-year time horizon for the achievement of results, taking the view that this is a reasonable length of time for the innovation to have an impact at scale. For many GCs an analysis of the unit cost per beneficiary at outcome level needs to allow for a realistic time scale of up to 5 years to accommodate wider scale adoption of the innovation. This will apply, for example, to the challenges in the agricultural and enterprise development sectors. A10.5  ANALYSIS OF THE RESULTS OF CEA There are limitations to any CEA with comparability across challenges and sectors difficult and caution is therefore needed in analyzing results. One key factor is the context of the program which could have a major influence on both operating costs and its perceived effectiveness. A10.5.1  CONTEXT MATTERS As highlighted in the USAID Guidance, while it is necessary to apply consistent metrics, the results may not necessarily be the same across different contexts. But this process will assist in analyzing which cost drivers affect the cost of the activity in different ways (e.g., population density or security costs etc.). So, while the country/regional context will mean differences in costs, the data can be used to help plan and budget more effectively. “However, USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 153 we must keep local context in mind. Deworming would not be a cost-effective intervention in an area where there is a low prevalence of intestinal worms because the impact or effect of the program would likely be lower than in a context with a high prevalence of worms. Similarly, an information campaign about the returns to education may not be as cost-effective as other programs such as subsidies or transfer programs in contexts where students already have accurate information about the value of attending school”.176 A10.5.2  IMPORTANCE OF PORTFOLIO ASSESSMENT It is equally important to consider the overall cost effectiveness of an intervention on the basis of an overall portfolio177 of projects rather than on a single case. It is often the case, particularly with innovation programs that aggregate results are very different from those achieved by selected, individual projects. Kremer, M et al. (2019)178 measured the cost benefit of USAID’s DIV. The paper recognizes the three key difficulties of measurement notably: (i) the time taken for an innovation to achieve measurable outcomes; (ii) difficulties in identifying and measuring user benefits; and (iii) the difficulties of placing a monetary value on these benefits. Kremer uses a CBA approach to measure 41 of DIV’s innovations, going back over an implementation period of 8 years. The paper highlights the very concentrated distribution of benefits that such a portfolio can realize with 10 out of the 41 innovations reaching 90 percent of the beneficiaries in the total portfolio. There is a complete analysis of 4 of the mature innovations that had generated $86 million in discounted social benefits compared to a cost stream of $16 million or a cost:benefit ratio of 1:5. The key findings from the USAID- supported Global Innovation Fund179 and the Australian/DFAT Funded Enterprise Challenge Fund180 was that where the total estimated impact was heavily concentrated towards just a few of the projects. While in both cases the funds showed a positive overall return, many of the projects had little or no impact which 176 USAID (2020) Cost Analysis Guidance for USAID funded Education Activities. 177 Portfolio is considered in the broad sense of the term here as this will have a different interpretation and level of aggregation for DIV than for the GCs. GCs generally have been characterized by a collection of projects that provide solutions to different aspects of one problem while DIV is a collection of projects focused on a range of problems. 178 Kremer, M, Gallant, S, Rostapshova, O, O & Thomas, M (2019) USAID Draft Research Paper Is Development Innovation a Good Investment? Which Innovations Scale? Evidence on Social Investing from USAID’s Development Innovation Ventures. 179 Global Innovation Fund (2019) https://www.globalinnovation.fund/wp-content/uploads/2020/02/GIF-Independent-Evaluation-Web-Final.pdf. 180 Triple Line Consulting (2013) R Woolcock et al Cost Benefit Analysis of the Enterprise Challenge Fund. 181 https://www.povertyactionlab.org/resource/conducting-cost-effectiveness-analysis-cea. 182 In September 2020, The UK merged DFID in the Foreign Office to create the Department of Foreign Cooperation and Development Office (FCDO). 183 See DFID Smart Rules: https://www.gov.uk/government/publications/dfid-smart-rules-better-programme-delivery. could lead to a conclusion of program failure. But like a venture capital fund, the success of the Fund or Challenge – either in impact or cost effectiveness language – needs to be measured at a portfolio level. A10.6 NEXT STEPS A forward-looking cost effectiveness framework for the GCs has now been produced (see Annex 11) which sets out a suggested approach for conducting CEA. It includes the following steps that should be followed: 1. Define/refine the key outcomes, beneficiaries and activities that the challenge is addressing. 2. Define the cost effectiveness measure(s) that is relevant to the challenge in terms of: (i) Unit cost of output/outcome achieved; (ii) Unit cost per beneficiary reached. 3. Identify the current data that is currently collected and the key gaps in data. 4. Define the key activity cost headings for the challenges separated between operating costs and investment costs over a 5-10-year time horizon. 5. Set out a cost effectiveness framework and proposed data collection program. While there should be a common approach to GCs in applying the framework, it will be necessary to take account of the different characteristics of the various sectors covered by the current portfolio, including: health, education, governance, agriculture, water and energy. APPENDIX 1: COST ANALYSIS A detailed cost effectiveness analysis guide is available by JPAL.181 The UK Government’s Department for International Development (DFID)182 has developed a series of guidance notes by sector on conducting value for money (VfM) which has since been institutionalized across all program design and management.183 These UK Government Smart Rules on VfM concentrate 154 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION the thinking around costs and results using the so￾called 4 ‘E’s: • Economy (minimizing the cost of inputs). • Efficiency (achieving the best rate of conversion of inputs into outputs). 184 Source: ICAI (2018) DFID’s Approach to Value for Money in Programme and Portfolio Management. • Effectiveness (achieving the best possible result for the level of investment). • Equity (degree to which the results of the intervention are equitably distributed). ANNEX FIGURE 4: FCDO’S APPROACH TO VALUE FOR MONEY184 Framework Components Input: Staff, raw materials, capital. (e.g., Vaccine and vaccination consum ables) Process: The methods by which inputs are used. (e.g., delivery logistics) Output: Results delivered directly by FCDO or our agents. (e.g., children vaccinated) Outcome: We exercise less direct control over outcomes than outputs. (e.g., children less susceptible to major childhood diseases) Impact: Long-term transformative change. (e.g., poverty reduced) The four Es and cost effectiveness Economy: Are we (or our agents) buying inputs of the appropriate quality at the right price? Efficiency: How well are we (or our agents) converting inputs into outputs? (‘Spending Well’) Effectiveness: How well are the outputs from an intervention achieving the intended effect? (‘Spending Wisely’) Equity: How fairly are the benefits distributed? To what extent will we reach marginalised groups? (‘Spending Wisely’) Cost effectiveness: What is the intervention’s ultimate impact on the poverty reduction, relative to the inputs that we (or out agents) invest in it? INPUT ECONOMY EFFECTIVENESS COST EFFECTIVENESS EQUITY PROCESS OUTPUT OUTCOME IMPACT EFFICIENCY USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 155 ANNEX TABLE 9: USAID COST ANALYSIS KEY QUESTIONS ANALYSIS METHOD WHAT QUESTIONS CAN IT ANSWER? WHAT CAN COST ANALYSIS HELP ACHIEVE? WHAT DATA WILL ANALYSTS NEED? Cost- economy analysis What did it cost to deliver this intervention? How much was spent on different tasks? How much should the government budget for a scale up? Help budget for the new activity. Help transition the intervention or its elements to the government. Expenditure and contributions reports disaggregated by cost categories and ingredients; a method for allocating shared costs across cost categories. Local price database for common inputs. Government cost structures; output data. Cost- efficiency analysis What did this intervention cost per output delivered? How does that compare to other delivery methods for this output? Identify unit costs per output. Compare unit costs across delivery methods and identify which one achieves the most outputs, within a given budget. Expenditure and contribution reports disaggregated by cost categories and ingredients; a method for allocating shared costs across categories. Output counts, using a common indicator for all interventions, disaggregated by delivery methods. Cost￾effectiveness analysis What did this intervention cost per outcome delivered? How does that compare to other interventions that produce this outcome? Compare cost of outcomes across different interventions. Identify the intervention that achieves the most outcome, within a given expenditure per beneficiary. Expenditure and contribution reports disaggregated by cost categories and ingredients; a method for allocating shared costs across categories. Credible estimates of the impact. Credible estimates of the cost and effects of comparable interventions. Source: USAID (2020) Cost Analysis Guidance for USAID-Funded Education Activities 156 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Photo: Global Devlab/USAID USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 157 ANNEX 11  COST EFFECTIVENESS ANALYSIS FRAMEWORK 185 Annex 10 of the USAID GC meta-evaluation report. 186 USAID (2020) Cost Analysis Guidance for USAID-Funded Education Activities. A11.1 INTRODUCTION A11.1.1  BACKGROUND AND CONTEXT The original task of the USAID Grand Challenges for Development (GC) meta evaluation was to assess how feasible it is to measure the cost effectiveness of previous and future GCs and to compare the cost effectiveness of GCs to traditional program models. A Cost Effectiveness Analysis (CEA) Review was undertaken which set out the status of all GCs being evaluated and reviewed of some the cost effectiveness literature from USAID and elsewhere.185 The Review found examples of good practice in measuring the costs and benefits of some GCs, notably in Security Water for Food and Saving Lives at Birth, and strong interest in this area, but also that there was insufficient data for most of the GCs to conduct a systematic CEA. This is partly a consequence of the very different management approaches taken by the GCs, with some managing the grants in￾house and others by an implementing partner. This forward-looking CEA framework sets out a roadmap for USAID to start the process of undertaking a systematic CEA of GCs in the future. A11.1.2 PURPOSE OF FRAMEWORK The purpose of this paper is to lay out a forward￾looking framework which builds on the considerable work and learning that has already been undertaken by USAID in the education, health, and agriculture sectors. A standardized framework is presented that could apply to all GCs and be adapted for different sectors and stages of a Challenge. CEA of early-stage projects will differ from Challenges at scale where there is greater evidence of emerging impact and outcomes that can be related to costs. While the purpose of the framework is forward looking, the intention is that where possible, this framework could also be applied to the current GCs. The purpose of this framework is not to compare the economic value of, for example, Health vs. Education Challenges, but to start a process of improving understanding of the cost effectiveness of the GC instrument compared to other types of intervention. A CEA methodology is required and must be applied across the full length of the Grand Challenge cycle from the design of the Challenge through to the ex￾post analysis of outcomes. This has implications for the allocation of responsibilities for data collection and accountability between various parties. A11.1.3 WHAT IS CEA? CEA provides a comparative measure of the overall cost of achieving a development outcome and is typically expressed in terms of the cost of the outcome per beneficiary reached. As expressed in the USAID education sector guidelines, “CEA seeks to determine the lowest unit-cost alternative for achieving a desired outcome.”186 The guidelines add that “CEA should be used where it is difficult to place a dollar value (i.e., monetize) the project benefits – as is often the case with nutrition, health and global climate change projects.” CEA enables the user to compare the costs and outcomes of different projects or Challenges. For example, CEA could be used to compare the unit cost of saving one life through improved health care to the unit cost of improved educational outcomes. The analysis thus enables a series of questions to be answered as set out in Table 1. 158 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ANNEX TABLE 10: COST EFFECTIVENESS ANALYSIS WHAT QUESTIONS CAN IT ANSWER? WHAT CAN COST ANALYSIS HELP ACHIEVE? WHAT DATA WILL ANALYST NEED? What did this intervention cost per outcome delivered? How does that compare to other interventions that produce this outcome? Compare costs of outcomes across different interventions Identify the intervention that achieves the most outcome within a given expenditure per beneficiary Expenditure and contributions reports disaggregated by cost categories and ingredients; a method for allocating shared costs across cost categories Credible estimates of the impact Credible estimates of the cost and effects of comparable interventions Source: USAID (2020) Cost Analysis Guidance for USAID-Funded Education Activities. ‘Where output and outcome data is not available, rather than CEA, GCs can start with the simpler tool of cost or expenditure analysis.’ A11.2  PROPOSED COST￾EFFECTIVENESS ANALYSIS FRAMEWORK The framework presented below builds on some elements of the CEA best practice already in use by USAID and brings in learning from the GCs under implementation. The framework is based on some key common CEA principles that should apply to all GCs. A decision tree presented in Annex Figure 5 sets out how the CEA should be adapted to different types of GC. A11.2.1 KEY PRINCIPLES The following key principles should apply to conducting CEA across all GCs. • Need for Common Principles: The broad principles of the CEA should be the same across all GCs. A CEA framework should propose a common set of broad principles which can be tailored along dimensions that may need special consideration (e.g., sector and stage of innovation). • Start at the Beginning and Finish at the End: To perform any analysis, cost accounting needs to be conducted throughout the life of the GC from its design phase throughout the lifecycle of the grants. The GC Manager and the grantee need to be instilled with an understanding of the importance and culture of measuring cost effectiveness through all stages of the GC – from initial design through to ex-post evaluation. • Cost Benchmarking: should become a standard part of the design and implementation of GCs. This should start with some indicative unit costs of achieving the outcomes at the start of the Challenge based on previous GCs or similar types of intervention. The inventory of data on comparable unit costs for different sectors will develop over time. • Sectoral Differences: While the indicators and tools of measurement may differ across sectors, all GCs should ensure that systems are put into place to measure costs and outcomes at both the aggregate and unit levels. For some sectors outcomes can be measured against standard outcome indicators (e.g., a common indicator in health is cost of life saved). For other GCs, there may not be validated outcome measures; however, this should not be a barrier for completing a CEA. Consistent measures can be piloted with the intention of working toward the development of standard indicators at both activity and outcome level. • Cost Effectiveness of Early-Stage versus Late-Stage Challenges: Some Challenges may have a high proportion of early-stage projects which may not reach outcomes at scale over the lifetime of the Challenge. For such Challenges, it is still important to measure the cost effectiveness of the management of the Challenge. Challenges with more late-stage projects that are delivering measurable outcomes to beneficiaries will require a more in-depth CEA of the grants’ outcomes. This is set out in the decision tree (Annex Figure 5) below. • Responsibility and Accountability: CEA requires that there are processes in place that are followed throughout the Challenge. It is important to note that although the responsibility for generating the data for the CEA may fall on different parties at each stage, the GC Manager should retain responsibility to ensure that these parties are accountable. Data collection can be an expensive process, particularly in tracking the outcomes of the Challenge, and this process may be contracted to a third-party researcher. The GC USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 159 Manager must be accountable for ensuring that the CEA framework is completed and any data collected from grantees. A11.2.2 CEA DECISION TREE GCs have the flexibility to be very diverse both in terms of sectors, each with different issues in terms of measuring outcomes, and the stage of the Challenge, with some focusing on early￾stage innovations, and others involving later stage projects, which have more clearly specified outcomes and a group of beneficiaries. A CEA framework therefore needs to be flexible and requires the cost analysis to be modified depending on the characteristics of the GC. There should be two layers considered in CEA: the Challenge level and the Grantee level. 1. Challenge level measures the cost effectiveness of the management of the Challenge, which will measure the average cost of identifying and supporting each grant, the cost of managing and supporting each grant, and the expected leverage of funds from other funders. 2. Grantee level examines the cost effectiveness of the funded grants, measuring where possible the unit cost of the outcomes achieved per beneficiary. Each grantee should follow a common cost accounting methodology in reporting to the GC manager. The GC Manager would then aggregate the CEA at an overall portfolio level where possible. The relative emphasis given to (i) and (ii) will depend on the nature of the GC. If the GC is primarily focused on early-stage innovations, the CEA at grantee level will be lighter touch and more emphasis will be given to measuring the cost efficiency of running the GC and identifying the unit cost of a project with the potential to go to scale. In all cases, CEA would be conducted at the Challenge level. A GC that consists mainly of early-stage grants would feature on the left-hand side of Annex Figure 5. The cost efficiency of the GC is defined as the unit cost of managing the GC. This will include the costs of launching the Challenge, identifying the grants/innovators, supporting the projects, overall management, and reporting to USAID. The costs could be expressed per grant managed and by value of managed grants. The cost effectiveness of the GC will include the management and support costs, and will have some overall metric of the outcome of the GC (e.g., potential beneficiary reach of the grants). On the right-hand side of Annex Figure 5, the GC Manager will measure unit costs of the outcomes achieved per beneficiary reached with the outcomes measured against standard indicators where possible. When possible to conduct CEA, it should be conducted over four stages during which time initial estimates of unit costs will be measured and refined: • The design phase is defined as the period prior to implementation of the GC. This stage would normally be completed by USAID, although there have been a number of cases where this phase is supported by the future GC Manager or a third party. The phase is defined to ensure that the design of the Challenge includes the specification that a CEA will be undertaken by the GC Manager, and the key parameters of the Challenge (including the size of budget for grants and GC Management, as well as a theory of change as set out in Annex Table 11). • The inception phase is defined as the contractual starting period for the GC Manager and would normally last for up to 1 year and would include: (i) a refining of the theory of change and the development of the results framework for the Challenge; (ii) the release of the first call for grant proposals; and (iii) the preparation for implementation of a program for completing CEA. • The implementation period is the period up to and including the completion of the GC Manager’s contract with USAID and the completion of all reporting obligations on the GC to USAID. • The ex-post processes would normally be conducted by a third party, but it could involve a researcher who was involved in data collection during the implementation of the GC. This stage involves verification and validation of the outcomes as measured during the implementation and testing the theory of change. Annex Table 11 summarizes the key processes that will be undertaken during these four stages with data collection being undertaken during all four stages. 160 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ANNEX FIGURE 5: CEA DECISION TREE 1. GC meets all basic cost benchmarking standards for a cost-effective challenge? 2. GC expected to contain mainly late stage projects? 3. Activities known and measurable from the call for project proposals? 5. Beneficiaries identifiable and outcomes measurable? 4. Outcomes known and measurable? N N N N N Y Y Y Y Y Design Inception Implementation Collect more info/alter the GC CEA should be lighter touch and focus on the cost efficiency of challenge CEA to measure cost efficiency of challenge only CEA to measure cost effectiveness overall CEA to measure unit cost of outcomes per beneficiary CEA to measure unit cost of outcomes CEA at grant level to proceed Ensure sufficient capacity and budget for CEA at grant level USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 161 ANNEX TABLE 11: SUMMARY PROCESSES AND INDICATORS BY STAGE OF GC STAGE: PROCESS: DESIGN INCEPTION IMPLEMENTATION EX-POST Challenge Definition Outline theory of change. Define key outcomes Define focus of CEA: Challenge and/or grantee level. Identify cost benchmarking for Challenge. Define key cost effectiveness measures. Refine theory of change, impact pathway, assumptions. Select indicators. Grant size, eligibility, and selection criteria. Complete cost benchmarking. Test validity of ToC and Challenge. Activity Costing Coordinator appointed to track costs across all parties. Call for proposals to specify the requirement and prepare activity costing for grants and managing the fund. Define activity cost format for grantees based on template. Periodic grants reporting. Periodic portfolio report on activity cost analysis. Measurement of Effectiveness at Challenge / Grant Level Budget for Challenge management. Budget for grants. Results frameworks for grants. Define/identify key data gaps at activity and outcome level. Define reporting format for grantees to GC Manager. Define GC Portfolio to GC Manager. Review of assumptions and impact pathways. Reporting by grantees on beneficiary reach and results. Reporting by grantees on outcomes achieved. Portfolio assessment by GC Manager. Evaluation of outcomes. Impact evaluation (as needed). Validation of ecosystem/ spillover impacts. Data Collection and Validation Key Data Total fund size. Average Grant Size. Estimated number of beneficiaries. Total Fund Management Cost. Define reporting format for grantees of GC. Key Performance Indicators: Estimated unit cost per output. Estimated unit costs per beneficiary reached. Forecast number of beneficiaries. Support/validation of grantees results measurement processes. Expenditure/ contribution reports disaggregated by cost category. Key Performance Indicators: Grants: unit costs per output/outcome. Grants: unit costs per beneficiary. Management costs per grantee. Portfolio analysis of activity costs. Comparison of unit outcome costs with comparators. Key Performance Indicators: Cost per beneficiary outcome achieved. Cost per beneficiary outcome achieved of comparable interventions. Sections A11.2.3 to A11.2.5 describe the key CEA actions required at each of the four stages of the GC cycle shown in Annex Table 11. It should be noted that all data will be updated during the grant cycle. For example, the average grant sizes will change during the implementation phase. 162 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION A11.2.3 STAGE 1: DESIGN DEFINING CHALLENGE AND KEY OUTCOMES The expected result of this stage is to have a clear understanding of the impact pathway or theory of change for the GC at an aggregate level with the key outcomes against which cost effectiveness can be measured. There should be an overall budget that would set out the allocation for GC management and the distribution for grants. GC Best Practice187 recommends that there needs to be an articulation of the Challenge Statement in the design phase before the Challenge is launched. This should be based on problem research, drafting the hypothesis or concept note, and reviewing the barriers of implementation. There should, in most cases, be a goal statement with a target outcome which defines the Challenge. The Challenge Statement may not have identified the solution to the Challenge and therefore the activities and type of grants may not be known at this stage. The Challenge Statement should be supported by a one-page theory of change, key outcomes, and budget. This theory of change should set out the key assumptions for the impact pathway to hold true and an initial mapping of the ecosystem which would identify: • The potential applicants (innovators) to the Challenge and the competitive landscape. • The profile of the end beneficiary where relevant,188 and estimated potential market/uptake. • The institutional barriers and assumptions facing implementation. COST BENCHMARKING Early in the design stage, once the Challenge is defined and key outcomes are identified, there is a need to undertake some benchmarking and assess the GC costs against benchmarked parameters where this data is available (see Appendix 2.2). There are two elements of benchmarking: 1. Cost benchmarking of a GC – The overall cost of the GC can be benchmarked using the efficiency precedents from other comparable 187 USAID (2014) An Introduction: Grand Challenges for Development. 188 Note this should generally be an individual or household and may not apply at all GCs. 189 DALY. Disability-Adjusted Life Years. 190 Monetization of outcomes is an emerging field, and where possible validated measures relevant to the sector should be used. If no validated measures are available, then consistent measures should be agreed. Examples provided (e.g., DALY) here are illustrative. GCs or interventions within the sector. Some key indicators that should be used are: – Average grant size. – Gross measure of a dollar per direct beneficiaries. – Direct cost (sum of total grants) to indirect cost ratio (remaining administrative costs). – No. of human resources and costs employed to manage a GC. – Share of monitoring, evaluation and learning budget in the total GC budget. – Percent of budget leveraged from sources other than GC. (including contribution from grantee). 2. Cost benchmarking of a grant funded by the GC – Each of the grants can be benchmarked for their costs. GCs should make the grantee report on their cost benchmarking targets right from the beginning, or set an expected range for all grantees to comply. – Minimum cost per outcome target (such as reduced DALY,189 increased learning achievement, increased production).190 – Share of total grant budget in GC budget. – Share of externally leveraged budget in the total grant budget. – Cost at scale (what will be the projected cost when scaled up) in terms of the proportion of the current cost of piloting compared to the reduced cost when scaled-up. (Note: time should be specified, e.g., 10–15 years to scale up). ACTIVITY COSTS The GC design document and call for proposals to manage the Challenge should specify the requirement that CEA is conducted by the GC Manager during implementation. The GC manager should supply a methodology and key indicators during the inception phase. The Manager should also demonstrate the capability of undertaking the tasks or partnering with a firm that has the capacity to collect outcome and cost data. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 163 MEASUREMENT OF EFFECTIVENESS At the design stage there should be some preliminary or indicative parameters from the GC Manager to provide CEA indicators based on: • Total Challenge budget. • Budget for Challenge management. • Budget for grants. • Total fund size. • Average grant size. • Estimated reach/no. of beneficiaries. (Note: Challenge manager to indicate direct or household). • Total fund management cost. It is expected that any benchmark information on the unit cost of the activities and outcomes envisaged should be set as an initial baseline for the implementers. The budgets should be set in standardized formats which would identify costs for Challenge management including the itemization of costs for the design, launch, and screening of the grants from the subsequent costs of monitoring and supporting grantees. EXPECTED CEA OUTPUTS AT STAGE 1 • Draft theory of change. • Define CEA priorities: Challenge level or grant level. • Budget breakdown and percent of total Challenge disbursed in grants. • Reach ‘ballpark’ estimates of unit cost per beneficiary. • Estimate the cost benchmarks for the GC as well as grantees (see checklist in Appendix 2.2). A11.2.4  STAGE 2: INCEPTION OF GC The inception phase could last up to one year for the implementer of the GC depending on how much of the design of the Challenge has been completed prior to being contracted. The focus of the inception phase is for the GC manager to ensure that all systems are in place for managing the relationship with the grantees/innovators and reporting to USAID. It would normally include a process of further scoping the nature of the Challenge and defining the eligibility and selection criteria for the innovations. There may be a need to undertake further benchmarking of costs during the inception phase. DEFINING CHALLENGE AND KEY OUTCOMES The first task of the GC Manager on CEA would be to further define the objectives and scope of the Challenge and refine the Theory of Change. It would involve specifying the objectives of the Challenge and generating a results framework. The GC Manager would define the approach for: • Launching and marketing the Challenge. • Undertaking the calls to identify innovations. • Selecting grants. The inception phase would normally include the period up to the selection of the first round of innovations. The GC Manager would establish if there is an emphasis on early-stage or late￾stage innovations or a mix of the two, as this will determine the scope of the CEA to be undertaken. A methodology for conducting the CEA would be agreed with USAID as part of the process of agreeing a theory of change and results framework. ACTIVITY COSTINGS AND INDICATORS Challenge Management The GC Manager would need to set up a simplified management reporting system, which would cost the key input activities of managing the Challenge separated into a minimum of three budget lines (see Appendix 1.1): • Launch, Marketing and Selection of the grants. • Support for innovations: advisory activity, capacity building, monitoring. • Management and reporting on the Challenge to USAID and other donors. Grantee Level As the individual grantees progress from the inception phase, they would be expected to report regularly (annually) against standard simplified headings: • Income sources to show: value of grant, cash contribution of grantee, debt and in-kind contributions from grantee; other sources of income from other donors. • Costs separated into one-off capital investment costs and recurrent activity costs. 164 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION • Standard definitions and measures of depreciation/amortization and sunk costs. Reporting by the grantee to the GC Manager would be conducted under strict data confidentiality procedures to protect potential commercial sensitivity. MEASUREMENT OF EFFECTIVENESS While in the inception phase, project activities with the grants may not have started. It is therefore crucial that the GC Manager defines the key activity cost headings for the grantees to report on. The GC manager would also provide guidance for the grantees to prepare an outline results framework for each of their grants. It is likely that each grantee would require some initial advisory support to define the nature of the individual Challenge and construct a results chain which is consistent with standard indicators and activity cost headings. Reporting formats and frequency including establishing procedures for defining the project baselines would be required. EXPECTED CEA OUTPUTS AT STAGE 2 • Grant eligibility and grant selection criteria. • Refined theory of change for the Challenge. • Results framework for each grantee. • Reporting template for each grantee with standardized activity cost headings and standardized indicators. • Reporting template for GC Manager including financial reporting split between Challenge management, support to grantees, and grant disbursement. • Revised cost benchmarking completed. A11.2.5  STAGE 3: IMPLEMENTATION DEFINING CHALLENGE AND KEY OUTCOMES There may be circumstances when the Challenge is redefined and the objectives modified during a mid-term review, but in general the key outcomes and indicators would be the subject of refinement only and could include a review of some key targets and assumptions on the theory of change and results framework. ACTIVITY COSTINGS AND INDICATORS If the GC consists primarily of early-stage projects: • Reporting on grant cost effectiveness will be light touch. The focus is on measuring cost efficiency of managing the Challenge. If the GC consists of later-stage projects: • CEA at grant level may be measured and the GC Manger will need to provide ongoing support to grantees to ensure the quality of reporting and ensure consistency in methodology and reporting across the grants. The GC Manager should compile a regular portfolio report on cost effectiveness (minimum annually) for USAID on both the results of the grants and the Challenge. MEASUREMENT OF EFFECTIVENESS During this phase the GC manager would be expected to conduct ongoing monitoring of the grants and validation of the results at output and outcome level against a common template. This will include validation and verification of the reporting by grantees and ad-hoc support to grantees on results frameworks and reporting to USAID. It could require spot checks and project visits to validate the reporting of results and ensure consistency for the portfolio analysis. Early-stage Challenges should focus their cost analysis on the efficiency of the Challenge management processes, including unit costs of running calls for proposals and identifying innovative projects that have potential. For later stage projects, there will be more reviewing of the intermediate outcomes of the projects and validation of the activity costings presented. In order to minimize the administrative burden on reporting it would be expected that the full reporting on the portfolio would be limited to an annual report, possibly augmented by quarterly updates of progress. EXPECTED CEA OUTPUTS AT STAGE 3 • All grantees to be reporting annually on results measurement processes. • Expenditure/contribution reports disaggregated by cost category. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 165 • Grants: unit costs per output/outcome cost breakdown by intervention. • Grants: unit costs per beneficiary. • Management costs per grantee supported. A11.2.6 STAGE 4: EX-POST There is almost always a considerable lag between the disbursement of a grant and the achievement of the project outcomes at scale. Some grants may only begin to reach maturity and realize outcomes at the end beneficiary level after five years or so, which is often beyond the disbursement period of the grant or lifetime of the GC.191 There should therefore be some validation of the Challenge outcomes at the end of the disbursement period and preferably two years after the closure of the grant disbursement, depending on the nature of the Challenge. USAID needs to be pragmatic here. Some grants may not have had time to reach sustainability after 2 years, but it is important to ensure that institutional memory of the Challenge is not lost. This ex-post analysis will ensure that some assessment of the sustainability and verification of the estimates of cost effectiveness (which may have been partly based on expected or predicted outcomes) takes place. This ex-post analysis also enables the opportunity to validate any potential market system impacts as a result of the innovations and spillover effects. Ex-post analysis could involve the commissioning of an independent impact evaluation to validate the estimates of the outcomes reported. What is critical here in conducting any such impact evaluation is that the researcher is able to trace the cost and benefit history of the grant based on the GC data that has been compiled using standardized activity costing. This leaves the primary function of the evaluator/ researcher to be validating beneficiary outcomes. EXPECTED CEA OUTPUTS AT STAGE 4 • Portfolio analysis of activity costs per intervention. • Updated analysis of the portfolio-level outcomes of early-stage projects. • Cost per beneficiary outcome achieved for late￾stage projects. • Cost per beneficiary outcome achieved of comparable interventions. 191 This could be built into the agreement with the grantee. For example, the Africa Enterprise Challenge Fund (www.aecfafrica.org) contracts projects for 6 years to include 3 years of reporting following disbursement of the grant to ensure the outcomes can be captured once projects are mature.. 192 Evaluation workshop with USAID 12.22.2020. A11.3  CEA RESOURCING AND TEMPLATES Once CEA has been implemented with the GCs, USAID will gradually gain more intelligence on benchmark GC management costs, and also on the unit costs of activities and outcomes from the grants. In this way the GCs will be gradually improving the quality and relevance of the benchmarked data as more CEAs are completed. It is important that this culture of CEA is adopted and instilled across all GCs so that the collection of ost and output data becomes a routine part of GC Management. As highlighted above, the investment and emphasis made in CEA will depend very much on the nature of the Challenge. For those Challenges with a lot of early-stage innovations, many of which will not go to scale, the GC should not invest resources in the data collection on outcomes or be overly concerned with the activity costs of the grants, although these GCs should still be run on a cost-efficient basis with benchmarked costs for the GC manager in terms of the launch, identification, and grantee support costs. Central to CEA is the need to adopt some simple and common practices in cost accounting that do not require a major investment in staff resources. The expensive part of the data collection process is measuring the outcomes of the Challenge, especially if a full impact evaluation is commissioned. The cost side is much more about working towards standardized cost accounting systems applied by the GC Manager and Grantee. It is understood that USAID has a rule of thumb that 5-7 percent192 of a program should be allocated for Monitoring Evaluation and Learning. Given that this proportion is generally not met, the cost of CEA should not be a barrier. It is more about instilling a culture within the GCs that CEA is a key part of the implementation of a Challenge, and so overcoming the inertia caused by the disincentive inherent in the collection of cost data. It is best practice within USAID to contract outcome evaluations to a third party in order to keep a level of separation from the GC Manager, therefore validation of cost data could be included in these evaluations. USAID needs to develop some standardized cost categories for GC Management and the reporting templates of the grantees to the GC manager. Appendix 2.1 sets out a proposed outline of the 166 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION template for the GC Manager and Appendix 2.2 for the GC Grantee. The box below provides an example of cost categories that have already been developed by USAID for the education sector. ANNEX BOX 1 USAID EDUCATION SECTOR COST CATEGORIES 1. General operations, management, and reporting. 2. Assessments and evaluations. 3. Higher education/pre-service teacher training. 4. In-service teacher training. 5. Teaching and learning materials. 6. System strengthening. 7. Private sector engagement. 8. Parents/community engagement. 9. Safe schools and infrastructure. 10. Grants, scholarships, and cash transfers to individuals/families. 11. Grants to organizations. 12. Other. Source: USAID. A simplified cost accounting categorization is proposed as a base for all GCs. The grantee is suggested to split costs associated with development and investment of the innovation from costs associated with delivering the innovation to beneficiaries. In some early-stage projects, the delivery costs may be a small proportion of this sub￾total. For late-stage grants, a much higher proportion of recurrent delivery costs would be expected. The key ratios that the GC manager should be tracking are: • GC Management costs as a percent of total GC costs. – Launch and marketing. – Grant screening/due diligence. – Support to grantees including monitoring. – Admin and reporting to USAID. 193 Evaluation workshop with USAID 12/22/2020. • Grants as percent of total GC costs. • Grant breakdown: – Development/investment costs. – Recurrent costs: - Grantee activity cost. - Other direct support to beneficiaries. - Admin and monitoring. A11.4 NEXT STEPS From the consultation undertaken for this paper, it is clear that there is strong interest both from the GC Managers and within USAID to develop a standardized approach to CEA analysis. As highlighted above, CEA should not be viewed as ‘nice to have’ but as an essential part of GC management. As highlighted by many, ambitions should be set modestly so as to ensure that ‘the perfect does not become the enemy of the good.’ 193 In addition to adopting the guiding principles and framework put forward in this paper, the following next steps are recommended: 1. USAID agrees in principle that CEA becomes an integral part of undertaking a GC and that an initial CEA becomes part of the design of all future GCs. 2. CEA is undertaken at all stages of the GC cycle with particular attention to agreeing CEA priorities in the inception phase of the GC. 3. GC managers need to demonstrate CEA capability and present a methodology in the call for GC tenders/proposals. 4. USAID develops a cross-sectoral working group to expand the current standard indicators of outcomes and standard costs, following the lead taken in the education sector. 5. USAID develops a database of benchmark unit costs at activity and outcome level for each sector, and GC Management costs. 6. USAID develops its own cadre or network of expertise to validate CEA analysis from the GCs and consolidate learning on best practice. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 167 APPENDIX 1.1 COST EFFECTIVENESS TEMPLATE – CHALLENGE LEVEL TOTAL COMMIT￾TED TOTAL DISBURSE￾MENT YEAR 1 YEAR 2 …YEAR N Total budget for GC 1.1 USAID 1.2 Others 1 Total Grantee Contributors 2.1 Cash Contribution Grantee 2.2 In-kind contribution Grantee 2 Total 3 Total Challenge Value (1+2) GS MANAGEMENT COSTS TOTAL COMMIT￾TED TOTAL DISBURSE￾MENT YEAR 1 YEAR 2 …YEAR N 4.1 Admin/Reporting 4.2 Marketing outreach 4.3 Screening/Due Diligence 4.4 Support to Grantees 4.5 Other 4 Total GRANTEE COSTS TOTAL COMMIT￾TED TOTAL DISBURSE￾MENT YEAR 1 YEAR 2 …YEAR N 5 Total Capital Costs 6 Grantee Activity Costs 7 Other 8 Total (5+6+7) Notes: Total Budget: Commitment should include size of budget agreed/contracted with USAID and other partners and other finance secured. Disbursement should include all funds paid to GC for each year. Disbursement cannot exceed Commitment. The difference reflects funds not spent. Grantee Contributions • On the income side, all contributions to the implementation of the GC should be included. This will include the cash, debt, and in-kind contribution of the grantees, and any revenue from the grantee that is ploughed back into the implementation of the GC. • In-kind contributions may include labor and management time provided pro-bono by the grantee but accounted for in project costs under 6. 168 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION APPENDIX 1.2 COST EFFECTIVENESS TEMPLATE – GRANTEE TOTAL COMMITTED TOTAL DISBURSEMENT 5 Grantee Costs Capital 5.1 Investment in plant + equipment 5.2 Other development costs Total TOTAL COMMITTED TOTAL DISBURSED YEAR 1 YEAR 2 YEAR N 6 Grantee Recurrent Costs 6.1 Direct Activity Costs 6.2 Training etc. 6.N Other delivery to beneficiaries 7 Other 7.1 Monitoring Costs by Grantee 7.2 Management Costs by Grantee 7.3 Other 8 Total N.B.: 5.1 will include all fixed investment costs which could include plant and equipment. Inputs used in activities including labor will be included in recurrent costs. Key Ratios to consider: 1= Total Donor support to Challenge 3= 1+2 Total Value of Challenge 4 GC Management Cost 4+8 Total GC Management + Total Grants=Total Expenditure on GC 4/(4+8)*100 GC Management Costs as a percentage of total GC expenditure 2/8 Funds Leveraged by Challenge Grantee development/investment costs are separated from recurrent costs to reflect that some Challenges may have high short term initial development costs and only be sustainable in the longer term. APPENDIX 2.1 COST BENCHMARKING TEMPLATE 1. Name of the GC: 2. Implementation Modality: (i) Directly by USAID (ii) For profit contractor (iii) Non-profit partner or contractor Is there any additional donor funding partner to the GC other than USAID? a) Yes b) No If Yes, who are the other funding agencies? USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 169 FUNDING AGENCY AMOUNT PERCENT OF FUND EXPECTED N.B. This is donor funding to the GC not additional funding raised by individual grantees. 3. Key outcome indicators & targets if applicable: INDICATORS TARGET 194 For the total Grand Challenge value, there is also an option to benchmark against the program cost, i.e., what would have been the estimated program cost had the modality been a traditional program instead of a Grand Challenge. APPENDIX 2.2 COST BENCHMARKING OF GC (DESIGN STAGE) Please complete the table below based on the information available from the secondary data sources. The shaded columns are to be filled based on the proposed GC. This may not be possible for a number of GCs that have a range of different types of projects and grantee outcomes. The purpose of this exercise is to ensure that the GC remains within the effective cost parameters comparable to similar GCs. MIN MAX AVERAGE BENCHMARKED FIGURE194 ESTIMATED GC FIGURE COMMENTS DOES THE GC FIGURE MEET BENCHMARKING? REMARKS 1. Total Grant Value of GC. 2. Average grant size (total GC budget/no. of anticipated grants). 3. Gross measure of a dollar per direct beneficiaries (e.g., cost per pupil trained). 4. Gross measure of a dollar per outcome (e.g., cost per life saved). 5. Direct cost (percent of grants against total GC budget). 6. No. of human resources employed to manage a GC. 7. Share of MEL budget in the total GC budget. 8. Percent of budget leveraged from sources other than GC. Note: A direct beneficiary is defined as in receipt of service or use of product from grantee. This does not include other members of the household. 170 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION APPENDIX 2.3 COST BENCHMARKING OF A GRANT FUNDED BY THE GC (INCEPTION/IMPLEMENTATION STAGE) Each of the grants can be benchmarked for their costs where this is appropriate and the exercise should be completed by the GC Manager working with the grantee. Please complete estimated details for individual grants to evaluate their anticipated cost effectiveness. These benchmarking assessments could be used to rank grantees based on their cost effectiveness while awarding the grant. MIN MAX AVERAGE BENCHMARKED FIGURE GRANTEE SPECIFIC FIGURE COMMENTS DOES THE GC FIGURE MEET BENCHMARKING? 1. Gross activity cost per direct beneficiaries (e.g., cost per pupil trained). 2. Cost per outcome (such as reduced DALY, increased learning achievement, increased production). 3. Share of GC budget in the total cost (percent funded through GC). 4. Cost at scale* (what will be the projected cost per outcome when scaled up). * Cost at scale is the predicted cost of the intervention/innovation when scaled up. A projection can be made using the costing at the pilot stage. The cost at scale measure has the potential to be used to determine scaling/replication decisions. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 171 Photo: Global Devlab/USAID 172 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ANNEX 12  STATUS OF COST-EFFECTIVENESS ACROSS GCS GC CURRENT APPROACH TO COST EFFECTIVENESS KEY OUTCOME INDICATORS (THAT COULD BE USED FOR CEA) DATA AVAILABILITY AND USE KEY LESSONS LEARNED & OPPORTUNITIES All Children Reading (ACR GCD) A common results framework applicable for all grantees includes indicators applicable to all grantees that can be utilized to generate cost effectiveness estimates. No specific considerations for Cost effectiveness. Cost effectiveness reporting is done using anecdotal evidence and/or logical statements. Number of training beneficiaries. Number of direct beneficiaries reached by the grants. Reading scores. A comprehensive results framework was only introduced during the third phase, which is currently under implementation. Grantee specific budgets. Results framework, on its own, will not be enough to provide cost effectiveness estimates. Need to have a cost effectiveness design for the entire grand challenge to be able to counter the geographical and innovation specific differences. Need to identify cost indicators that are applicable to all grantees and set up a mechanism to report. Fighting Ebola No specific considerations related to CEA. There is no results framework available for the Grand Challenge. Standard effectiveness indicators related to PPE. The only source of information on cost effectiveness is the implementation report submitted by each grantee, which vary widely in terms of their length, contents and formats. Grant specific budgets. Missed opportunity to compare the innovations with existing cost effectiveness indicators. As a result, some of the innovations such as advanced PPEs were too expensive to scale. Securing Water for Food (SWFF) Full Social Cost Benefit Analysis undertaken195 co-authored with USAID (Ku McMahan) published in Jan 2020. Net present values of innovations/ grants calculated. Value of user benefits quantified. All costs calculated. Full SROR data compiled on costs and user benefits. Good model to follow where benefit stream can be monetized. 195 Usmani, S and McMahan, K (2020) The Economic Benefits of Supporting Private Social Enterprise at the Nexus of Water and Agriculture: A Social Rate of Return Analysis of the SWFF Grand Challenge. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 173 GC CURRENT APPROACH TO COST EFFECTIVENESS KEY OUTCOME INDICATORS (THAT COULD BE USED FOR CEA) DATA AVAILABILITY AND USE KEY LESSONS LEARNED & OPPORTUNITIES Saving Lives at Birth (SL@B) No specific consideration of CEA at design stage. Independent Evaluation (Duke University) conducted in May 2020 which concluded that impact estimates for most grantees were premature given their earlier stage of growth (and evidence level) in the scaling pathway. Evaluation recommended updating the ToC and more clearly defining program goals and measurable indicators of success. Some cost effectiveness analysis completed for 5 of the innovations. 3 published papers of 3 individual CEAs conducted by Duke are available, plus a lesson learnt doc - available here: Duke Univ. Theory of change and “impact framework” produced, with 10 core impact indicators & 5 additional process indicators. (e.g., # of lives saved, or DALYs or QALYs196 improved by fielded innovations). Very limited CEA data at portfolio level. Full CEA undertaken by Durham University. for one innovation (Bempu) compared to other programs Case study only. Duke used an impact model (which enables forecasting/modelling of impact that innovators will have by 2030) to conduct their innovation￾specific CEAs. Costing tool developed by Duke. Case study approach of CEA has good potential. 196 DALY- Disability-adjusted Life Year. QALY- Quality-adjusted Life Year. 174 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION GC CURRENT APPROACH TO COST EFFECTIVENESS KEY OUTCOME INDICATORS (THAT COULD BE USED FOR CEA) DATA AVAILABILITY AND USE KEY LESSONS LEARNED & OPPORTUNITIES Powering Agriculture (PAEGC) Summative Evaluation conducted in April 2020. In 2018, PAEGC collaborated with FAO to prepare a report on ‘Costs and Benefits of Clean Energy Technologies in the Milk, Vegetable and Rice Value Chains’. It presents a methodology to analyze the costs and benefits of some of the technologies that could supplement or replace fossil fuel use, drawing on a selection of PAEGC innovator case studies. Financial and economic NPVs were calculated on the basis of case study data. A follow-up Food and Agriculture Organization (FAO) led study in 2019 developed a methodological approach for comprehensive CBA of energy technologies at a country level with PAEGC-based case studies for Kenya, Tanzania, the Philippines, and Tunisia. Increases in agricultural productivity. Kilowatts (kW) of clean energy generation capacity installed. Tons of carbon dioxide equivalent (tCO2e) reduced as a result of innovators’ field activities. Amount of USD savings in energy costs saved in the value chains. No. of beneficiaries (e.g., farmers, households, agribusinesses). Amount of USD investment catalyzed. Summative evaluation (2020) includes case study data on a sample of projects including: Increases in agricultural output and incomes. Reductions in emissions of greenhouse gases. Good case study material but no systematic cost analysis across the program. Combatting Zika and Future Threats No specific consideration related to CEA in design or implementation of the program. No program level results framework, although some attempts were made to ask projects to report against a list of standard ‘program level indicators’ (e.g., funds leveraged, extent of scale up). Amount of funds leveraged. Project level reporting against output/ outcome indicators (but of little relevance to CEA, outside of ‘funds leveraged’). Very challenging to assess cost effectiveness of activities with highly diverse portfolio of projects and where impact cannot be easily quantified. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 175 GC CURRENT APPROACH TO COST EFFECTIVENESS KEY OUTCOME INDICATORS (THAT COULD BE USED FOR CEA) DATA AVAILABILITY AND USE KEY LESSONS LEARNED & OPPORTUNITIES Scaling Off Grid Energy No specific consideration related to CEA in the design. Some limited attempts to compare the number of expected connections to the amount of funding provided were abandoned due to lack of quality data and relevance. There is limited harmonized reporting between projects led by the various partners. Number of expected connections. Amount of investment catalyzed. Project-level data only available for projects managed by the Lab, in individual reports, but there are variations between grantees. Annual reports have data on total number of expected connections and investment catalyzed. Difficulty to assess the cost effectiveness of activities which have an impact cannot be easily quantified (such as the impact of donor coordination, market acceleration activities…). Creating Hope in Conflict Seen as very important by key informants so that finite resources can be more effectively. Log frame includes some indicators of relevance, but no framework developed for measuring cost effectiveness and only limited guidance [not seen] has been developed for innovators on the subject – e.g., indicators not yet identified at innovator/grant level. # and percent of seed and TTS innovators who have leveraged additional funding through smart partnerships to support scaling and sustaining their innovation. # and percent innovators reporting increase of cost efficiency of humanitarian assistance not specified how defined. Low targets. percent seed innovators progress to Transition to Scale grants. # and percent of seed-funded innovations that establish proof of concept. # of end users (disaggregated) with access to improved humanitarian products or services. # of intermediaries. (disaggregated) using/implementing innovative humanitarian products or services in their communities. Reporting against the log frame is clear and data is disaggregated. Data is available up to Milestone 2 – June 2020 including broken down in detail for end users with access and for implementers. CHIC was launched in 2018 and R1 projects are still being implemented so data / results still at an early-stage. Budget breakdown on GCC costs (e.g., split between fund mgt, technical assistance and grants) available on request. Note: LF states that ‘The milestones for cost efficiency will be revisited once CHIC has more systematically gathered quantitative data from innovators. The achievement data in the log frame indicates that MEL systems are robust and aggregate project level data effectively. Although cost effectiveness is important to its overall objectives, it does not seem to have a robust framework for measuring this. [Note: the RFP for developmental evaluation consultancy services, issued Dec 2020, identifies one of the key activities to be ‘support [GCC] to test the program’s VFM and provide guidance and/or validation on its framework and approach to VFM]. 176 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION GC CURRENT APPROACH TO COST EFFECTIVENESS KEY OUTCOME INDICATORS (THAT COULD BE USED FOR CEA) DATA AVAILABILITY AND USE KEY LESSONS LEARNED & OPPORTUNITIES Making All Voices Count Fund approach to CE not known, but there is evidence that the fund itself was not cost effective: Final Evaluation, 2018 on FM costs: ‘management, equipment, office and admin costs accounted for 29 percent; and ‘problems with coordination across consortium partners and limited synergies over the program life had led to lower-than-expected value for money’. Sida meta-evaluation report 2018: Limited achievement at outcome and impact levels. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 177 ANNEX 13  INSIGHT MEMO: LEARNING FROM THE INTEGRATION OF GENDER AND SOCIAL INCLUSION (GESI) INTO USAID GRAND CHALLENGES 197 USAID, 2012, Gender Equality and Female Empowerment Policy. 198 USAID: Gender and Social Inclusion Analysis; Uganda, Aug. 2017. https://www.usaid.gov/sites/default/files/documents/1860/Gender_Social_Inclusion._ Final_Report_08.23.17.pdf. EXECUTIVE SUMMARY The main purpose of this insight memo is to generate findings on how gender equality and social inclusion (GESI) has been addressed by USAID Grand Challenges (GC), provide insights on how GESI could be better integrated into future GCs, and make recommendations. The USAID GCs focus global attention and resources on specific, well-defined international development and humanitarian problems and testing innovative approaches, processes, and solutions to solve them. GESI was not a prescribed focus of GC design, but the meta-evaluation provides an opportunity to learn from experience to date. The analysis has been guided by USAID’s definition of GESI: “Gender Equality concerns women and men, and it involves working with men and boys, women and girls to bring about changes in attitudes, behaviors, roles and responsibilities at home, in the workplace, and in the community. Genuine equality means more than parity in numbers or laws on the books; it means expanding freedoms and improving overall quality of life so that equality is achieved without sacrificing gains for males or females”.197 Social Inclusion is defined as, “the process of improving the terms on which individuals and groups take part in society – improving the ability, opportunity, and dignity of those disadvantaged on the basis of their identity”.198 SUMMARY FINDINGS Across the three GC phases, design, implementation and results, four of the nine GCs have considered GESI: Creating Hope in Conflict (CHIC), Securing Water for Food (SWFF), Saving Lives at Birth (SL@B), and Powering Agriculture (PAEGC). In two GCs, All Children Reading (ACR GCD) and Making All Voices Count (MAVC), GESI was implicitly considered through targeting poor, disabled people or other social groups as beneficiaries. There was no evidence of more than minimal consideration of GESI in the other three GCs, i.e., Combating Zika and Future Health Threats (Zika), Fighting Ebola (EBOLA), and Scaling Off-Grid Energy (SOGE). Design: There were good examples (e.g., CHIC, SL@B, SWFF, PAEGC) where analysis of poverty, rights, access to services (e.g., health, education, water, etc.) by disadvantaged groups, and the influence of conflict and insecurity was considered in the design, alongside highlighting poor people’s perspectives to different degrees. Implementation: GC consideration of GESI was stronger in implementation than design, and awareness and integration of GESI have increased iteratively over the course of implementation for many. Some GC MEL frameworks (CHIC and SL@B) incorporate good data disaggregation to monitor progress and report differential impacts. Otherwise, GESI does not feature prominently in GC MEL frameworks and processes. Results: Three out of nine GCs (CHIC, SL@B, and SWFF) have demonstrated clear disaggregated results, but otherwise the presentation of results on GESI is very limited. Those GCs that had strong integration of GESI including in the MEL frameworks, 178 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION have been able to report differentiated results for women particularly, but the extent to which the GCs, as a whole, have contributed to both bridging gender disparities and poverty reduction is not visible. SUMMARY INSIGHTS Insights, set out in the memo against each set of findings, highlight the importance of: • Contextual analysis that integrates GESI, demonstrating the systemic barriers and challenges that different disadvantaged groups face. • Theories of change that include specific GESI perspectives and develop clear assumptions of how poor or disadvantaged groups would benefit. • Specifying and differentiating target groups. • Defining and specifying terms related to GESI. • Understanding and raising the awareness and knowledge of grantees about GESI in a given context. • Developing monitoring frameworks and tools that integrate GESI-related indicators and targets, in order to monitor progress, and report the differentiated results and impact. RECOMMENDATIONS Headline recommendations, unpacked in the memo, are directed principally to partner GC managers but guidance and oversight from USAID is also required. • Embed GESI from the outset (during the design). • Develop a robust Theory of Change which integrates GESI.199 • Design a MEL framework which reflects GESI and makes different groups visible. • Specify the target beneficiary groups. • Present results with disaggregated data in terms of gender and different social groups. • Understand grantee capacity on GESI, raise awareness and knowledge of grantees on what the terms gender equality and social inclusion mean in a given context and provide support. • Develop tools and guidance which help integrate GESI considerations at all stages of the GC. 199 It is recognised that USAID ADS 205 does not require GESI to be incorporated into theories of change: ‘The impact of gender should be discussed in the Project Description section, and when possible included in the required Annex presenting a logic model or graphic display of the theory of change.’ 200 USAID, 2012, Gender Equality and Female Empowerment Policy. A13.1 INTRODUCTION The main purpose of this insight memo is to generate findings on how gender equality and social inclusion (GESI) has been addressed by USAID Grand Challenges (GC), provide insights on how GESI could be better integrated into future GCs and make recommendations. The USAID Grand Challenges for Development focus global attention and resources on specific, well-defined international development and humanitarian problems, promoting innovative approaches, processes, and solutions to solve them. GESI was not a prescribed focus of GC design and GESI was not included in the terms of reference for the GC meta-evaluation or the insight memos it is producing. USAID recognized the value of learning about how GCs have considered GESI, which led to GESI being selected as an insight memo theme. The insight memo drew on data collected for the meta-evaluation: document review, key informant interviews, and the grantee survey, all of which included evidence on GESI within the GCs. The GC documents reviewed included external evaluations, internal reports and published materials, and GC web pages. Interviews were conducted with USAID and partner GC managers, donor partners, and others closely involved in GC implementation, such as those providing technical assistance to grantees. The grantee survey included specific questions on GESI to obtain the perspective of innovators who had received GC funding. This evidence was analyzed against the lines of inquiry selected for this insight memo, to generate findings and insights from the GCs against the core phases of design, implementation, and results. The analysis has been guided by USAID’s definitions of GESI: • “Gender Equality concerns women and men, and it involves working with men and boys, women and girls to bring about changes in attitudes, behaviors, roles and responsibilities at home, in the workplace, and in the community. Genuine equality means more than parity in numbers or laws on the books; it means expanding freedoms and improving overall quality of life so that equality is achieved without sacrificing gains for males or females”.200 • Social Inclusion is defined as, “the process of improving the terms on which individuals and USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 179 groups take part in society – improving the ability, opportunity, and dignity of those disadvantaged on the basis of their identity”.201 • The memo uses the concept of Gender Equality and Social Inclusion to mean improving access to livelihood assets, services, and opportunities for all different groups, including women, men, poor, disadvantaged, and excluded groups. GESI supports more inclusive policies, structures, and mindsets or behaviors, and increases the voices and perspectives of all these different groups. A13.2  FINDINGS AND INSIGHTS The review has examined the extent to which the GCs have integrated issues of gender equality and social inclusion into (i) the design (including selection), (ii) implementation (grant management/ technical support and monitoring and evaluation), and (iii) results, to generate insights for future GCs. Our findings and insights are presented under each stage of the GC cycle. A13.2.1 GESI IN DESIGN This section looks at the findings and the insights generated from the experience of GESI integration in the design of the GCs portfolio to answer the following key questions: 201 USAID: Gender and Social Inclusion Analysis; Uganda, Aug. 2017. https://www.usaid.gov/sites/default/files/documents/1860/Gender_Social_Inclusion._ Final_Report_08.23.17.pdf. 202 According to the PAEGC Mid-term review findings, during the 1st call, diverse organizations were selected (NGOs, private sector, foundations) but most funded organizations did not conduct market analysis to identify demand for their products by poor people. However, during the second call - funded organizations show ‘good knowledge of local context in relation to target group/living conditions, local market, product demand.’ Lines of enquiry: How was GESI considered during the design stage, e.g., was any gender or social analysis incorporated into the contextual analysis? How has GESI been considered in the grantee selection criteria? KEY FINDINGS • Nearly half of the GCs (four out of nine) considered GESI at the design stage, as set out in Annex Table 12. These four are CHIC, SWFF, SL@B, and PAEGC, where GC design has integrated analyses of poverty and access to services and benefits by disadvantaged groups. In particular, PAEGC targeted organizations that developed innovative solutions that benefited vulnerable and poor populations affected by food insecurity.202 • In three other GCs, GESI has been implicitly considered through targeting poor people, disabled people, or other social groups as beneficiaries, but with little analysis of the barriers and challenges these groups may face in accessing services and benefits (ACR GCD, MAVC and Zika). For the remaining two GCs (EBOLA and SOGE), there is no or minimal evidence of GESI being considered at the design stage. ANNEX TABLE 12: LEVEL OF GESI INTEGRATION IN GC DESIGN NAME OF GC LEVEL OF INTEGRATION FOCUS /SECTOR CHIC High Humanitarian SL@B High Health (pregnant women and newborns) PAEGC High Agriculture SWFF High Water ACR GCD Medium Education (disability focus) MAVC Medium Governance Zika Medium Health (Zika) EBOLA Low Health (Ebola) SOGE Low Energy Note: Rating was informed by the presence, quantity and quality of evidence about how GESI had been considered in problem and context analysis and how GESI had been integrated into the grantee selection process. 180 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION • While GESI was a requirement from the outset for CHIC, where consideration of marginalized groups and gender analysis were required for each application submission, this was not the case for other GCs. With PAEGC, although GESI was not a requirement, USAID and Sida203 worked together to ensure gender analysis was included in the selection process and in the advice to the winners. The emphasis placed on gender and poverty was increased from the first to the second round of calls for proposals. • How GESI was framed or perceived by a GC depended to some extent on the specific focus of the challenge. For example, SL@B focuses on women and newborn health. Therefore, as would be expected, GESI has been central to the program. The program was specifically designed to support innovations that empower pregnant women and their families to practice healthy behaviors and be aware of, and access, healthcare.204 In contrast, SOGE focuses on the number of electricity/energy connections which will be created and does not also focus on the type and characteristics of the people who would most benefit from them, despite the relevance of beneficiary type (e.g., women-headed households, elderly) on poverty outcomes. • Reaching and funding non-traditional development actors205 was seen as an important way of achieving diversity of grantees and seven GCs took specific steps to do this206 - ACR GCD, CHIC, PAEGC, SWFF, SL@B, SOGE, and Zika. This was despite some of these GCs (ACR GCD, SOGE, and Zika) not having robustly considered GESI in their fund design. GESI can be a feature of reaching non￾traditional actors (for example, women small￾203 Swedish International Development Cooperation Agency. 204 UK FCDO Business Case for SL@B. 205 ‘Non-traditional development actors’ are those who would not usually access USAID funding. They include organizations based in the communities that would potentially benefit from the innovation, private sector innovators and partners, and research and academic institutions in target (i.e., low or middle income) countries.How and how successfully GCs have done this is explored in Section 4.6 of the meta-evaluation. 206 As explored in section 4.6 of the meta-evaluation report. holder farmers in SWFF and actors from affected communities in CHIC). • Survey responses indicate that most grantees see addressing gender equality and social inclusion as a fundamental means by which their project can have social impact. Addressing social impact was described as a central part of projects by 91 percent of respondents to the grantee survey conducted for the meta-evaluation, but a smaller proportion (75 percent) saw improving lives of women or other disadvantaged groups as central. Conversely, 7-8 percent of grantees disagreed that social impact, improving the lives of women and others and ensuring better outcomes for marginalized groups was central to their project, indicating that these projects were not tackling poverty and GESI as a central concern. KEY INSIGHTS The following key insights arise from the review of factors which have contributed – or are likely to contribute - to successful integration of GESI in future GC design, and lead to recommendations below. • A clear interest by donor partners and an encouragement or requirement for GESI integration positively influences the priority given to it by GC managers. Although there is insufficient evidence from an analysis of program documents of the extent to which donors are using their influence early enough in the design stage to steer GCs towards GESI consideration, there are cases where donor partners have encouraged the integration of GESI or made it a requirement. For example, in CHIC, the focus on GESI of the UK and Dutch Strongly Agree Agree Neutral Disagree Strongly Disagree “Addressing social impact in a central part of our project” “Improving the lives of women and girls and/or other disadvantaged groups in a central part of this project.” “Ensuring better outcomes for marginalized groups in a central part of our project” 80% 11% 24% 2% 4% 4% 17% 3% 4% 51% 7% 4% 4% 20% 65% ANNEX FIGURE 6: SURVEY RESULTS IN RELATION TO GESI USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 181 governments, as co-donors, helped to ensure its integration at the design stage: there was explicit consideration of marginalized groups in design. Gender analysis was required for each application submission, and all innovations were required to explicitly and systematically integrate gender equality considerations at all stages of the project (including budgetary provisions), with gender￾disaggregated data included in project reporting. • GESI integration was the strongest where poverty analysis was strong (e.g. CHIC, SL@B, SWFF, PAEGC). Contextual analysis that integrates gender and social analysis is paramount at the beginning, to help understand fully the issues around GESI that need to feed into design (and, subsequently, implementation). Conversely, a lack of research into gender/social inclusion analysis at the beginning to understand the extent of barriers, challenges, and disparities seems to have handicapped considerations of GESI in other GCs. • Where context analysis at the design stage is informed by the voices of those affected by a problem it is likely to be more robust. For example, in the case of CHIC, the voices of those affected by conflict have informed program design and contributed to the GC’s strengths in GESI. • Across the GCs, there is little evidence of clear design related to the expected contribution programs could make to reducing gender disparities in access to, control over, and benefit from the products or services provided through the GC.207 From documentary evidence such as information for applicants, GC focus is often limited to what can be done to include women rather than understanding how to explicitly incorporate women and other disadvantaged groups’ voices and experiences into project design and use that to influence decisions about implementation. What is often absent in GC reports is a systematic analysis of the differences between men and women in how they might benefit from the GC, with qualitative insights on their 207 It is important to reiterate that GESI was not a requirement for all GCs. 208 The ToC describes the relationships between the problem (context) being addressed and the strategies being used to achieve the changes sought. It depicts the causal pathway, outlining the project from its inputs through to achieving the expected impacts from the program. In broad terms, ‘a theory of change is an on-going process of reflection to explore change and how it happens – and what that means for the part we play in a particular context, sector and/or group of people.’ (James, C. 2011. Theory of Change Review: A Report Commissioned by Comic Relief. London: Comic Relief). ‘Having an explicit ToC allows us to check whether a failure, if any, was due to the fundamental theory or due to an operational and implementation failure. The key point is that this definition helps tackle the common problem relating to theories of change: when organizations assume that change in a society revolves around them and their program, rather than around a range of interrelated contextual factors of which their program is part.’ Valters, C. 2015. Theories of Change: Time for a radical approach to learning in development https://www.odi.org/publications/9883-theories-change-time-radical-approach-learning-development. 209 With Sida as one of MAVC’s donors, the intervention’s programmatic focus on gender equality was influenced by Sweden’s policy on gender equality. This states that women and girls’ access to and use of ICT is central, and this was one of the three principles underpinning MAVC. GESI was also explicitly stated in the results framework. However, from the documents reviewed, although there is some discussion of gender disaggregation of data, it proved difficult in practice due to the ethical importance of citizen anonymity to promote engagement and ensure the safety of individuals (an issue raised by the Institute of Development Studies, a member of the fund manager consortium). Additionally, GESI was rarely visible in other progress reports. voice, access, and control, the differential power structures/imbalances, and the structural barriers faced by disadvantaged groups. This again takes us to the importance of gender and social analysis at the outset, as observed earlier. • Theories of change208 or intervention hypotheses that include GESI perspectives or assumptions of how the poor or disadvantaged would benefit from a GC are helpful to framing design in a way that supports GESI. In SWFF, for example, there is explicit mention of how and why poor people are expected to benefit: “The hypothesis assumes that the poor will benefit from increased efficiency and/or profitability, or indirect economic benefit by improving food security.” This was not supported by a more detailed explanation of how this was expected to happen, or who ‘the poor’ are. This would have made the theory of change stronger. • When the targeted people/users are described in general terms (e.g., “citizens”) rather than more specifically (e.g., by age, gender, social group), it weakens GESI integration. In several GCs, not considering GESI was a missed opportunity to have an impact on poverty and exclusion since women can benefit from access to solar energy in different ways from men, and potentially with more impact. Goals and objectives were framed broadly in terms of citizens, institutions, or technologies. For example, MAVC, framed as a governance program capitalizing on the transformative potential of innovation and technology to amplify the voices of citizens, lacked explicit disaggregation of who the citizens were.209 Similarly, SOGE focuses on the number of connections which were to be created rather than the people benefitting from them. • Targeting disadvantaged groups, including women, as beneficiaries or end users may not be sufficient in itself to reach them or for them to benefit from an innovation, if there is no clear analysis of either the barriers and challenges these groups face, nor how a GC intends to overcome them. SWFF saw gender 182 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION as an important aspect of acceleration support and reaching target beneficiaries. At an annual convening a gender adviser with a consultant from UN Women held sessions with each SWFF innovator on how gender considerations could be integrated into their projects. • Terminology in relation to “gender” may be simply used as another way of referring to “women”, and some of the language used to define social inclusion may be insufficiently explained. Terminology such as “hard to reach” populations or “marginalized children” and “children living with disability” was used in project documents without reference to processes of social analysis to identify the composition of the disadvantaged groups. Generally, it is useful to specify who these disadvantaged groups are, as well as their gender, to better understand the disparities among the groups and the differential impact which innovations might have on their lives. A13.2.2 GESI IN IMPLEMENTATION This section looks at the extent to which GESI has been considered in the implementation phase, such as grant management and technical support, and monitoring, evaluation and learning (MEL), guided by the following key questions: Lines of enquiry How has GESI been considered in the provision of grant management and technical support? How has GESI been considered in the program MEL? KEY FINDINGS Our overarching finding is that GC consideration of GESI in the implementation phase was stronger than at the design stage. Five GCs had integrated GESI well into their implementation phases as demonstrated in Annex Table 13. Additionally, we found that: • Awareness and integration of GESI increased iteratively over the course of implementation in some GCs. There are a number of cases where attention to GESI, and its integration into management processes, evolved over the period of program implementation (see section on fund management and technical support below). • Taken as a whole, GESI has not been a prominent feature of MEL indicators and processes in GCs, although some have incorporated data disaggregation well, to monitor differential impacts (see section on MEL below). FUND MANAGEMENT/TECHNICAL SUPPORT: There are several examples of GCs increasing their focus on GESI integration from design and over the course of implementation, often due to donor influence as set out below: • PAEGC integrated GESI more explicitly from the second call for proposals, raising awareness of the need for grantees to integrate women into the production chain or as users of the innovation. • ACR GCD’s third phase was targeted towards countries that lagged behind in the overall performance of the education sector, in terms of promoting reading skills among children with disabilities, particularly those with hearing- and vision-related disabilities. ANNEX TABLE 13: LEVEL OF GESI INTEGRATION IN GC IMPLEMENTATION NAME OF GC LEVEL OF INTEGRATION FOCUS /SECTOR CHIC High Humanitarian PAEGC High Agriculture SWFF High Water SL@B High Health (pregnant women and newborns) ACR GCD High Disability/Education MAVC Medium Governance ZIKA Low Health (Zika) EBOLA Low Health (Ebola) SOGE Low Energy Note: Rating was informed by presence, quantity and quality of evidence about how GESI had been considered in the provision of grant management and technical support and program MEL. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 183 • SWFF explicitly gave preference to female innovators from LMICs in its fourth call.210 This was clearly due to the influence of Sida, together with the Dutch Government, as is clear from the SWFF Mid-Term Review where there is a clear call for round four (R4) applicants to make visible efforts to address poverty and gender. • In SWFF, PAEGC, and MAVC there is evidence that Sida and USAID worked together to ensure gender analysis in the selection process and in the advice to the winners. • In PAEGC and SWFF, practical tools and guidelines have been developed to help grantees develop their understanding of gender equality. Sida led this, commissioning a study on gender in both funds which made clear recommendations for gender to be made visible in grantee selection criteria, in the advice provided to applicants and grantees, and in reporting indicators. Several guidance documents aiming to raise awareness on how to integrate gender in six areas of work were produced within PAEGC.211 For SWFF, Sida produced guidance on gender mainstreaming and three gender impact summaries, as well as providing the services of a gender expert. It is not sufficiently clear from the reports the extent to which these guidelines have been useful to those intended to use them, and whether the GC managers have convened the grantees to discuss these guidelines and ensure they are being understood and used to integrate GESI considerations effectively. • Within MAVC, gender was a higher priority for Sida than for other donors and seen as important for achieving transformational change through the program. While during implementation, different conditions and opportunities for men, women, boys, and girls had been analyzed and integrated into the results framework, where innovations were working with technology, research ethics demanded user anonymity, making it hard (and contested to track, log and therefore report gender splits in practice. • In CHIC, a strong gender focus in implementation was clear from the outset, influenced in part by the UK and the Dutch, and involved i) marketing the challenge to women’s and inclusion groups as part of its promotion strategy; ii) having gender 210 According to the Evaluation of Sida’s Global Challenge Funds - Lessons From a Decade Long Journey (by Triple Line – 2018), local innovators now constitute over 50 percent of the grantees in the program and 35 of the innovations are women-led. 211 i. Integrating Gender in the Deployment of Clean Energy Solutions for Agriculture; ii. Gender Responsive Product Development; iii. Integrating Gender in the Financing of Clean Energy Solutions; iv. Integrating Gender in the Marketing of Clean Energy Solutions to Agricultural Users; v. Integrating Gender in Monitoring and Evaluation of Clean Energy Solutions; vi. Integrating Gender in the human resources process. 212 For ACR, the results framework integrates the terms such as marginalized children, but no definition of marginalization was included. There is a plan to evaluate the results against a few indicators that concern children living with disability and marginalized children. balance in selection panels; and iii) establishing clear output measures in terms of the number of women-led innovations and awards to teams from conflict-affected communities, stating a preference for applications from these specific groups. • Similarly, Grand Challenges Canada drew attention to the importance of gender integration on the SL@B program. MONITORING, EVALUATION AND LEARNING Some GC MEL frameworks (CHIC and SL@B) incorporate good data disaggregation to monitor progress and report differential impacts. Otherwise, GESI does not feature prominently in GC MEL frameworks and processes. • SL@B and CHIC had strong follow-up/monitoring of GESI during implementation, capturing and reporting disaggregated data against indicators by factors such as gender, age, and disability. In most GCs, outcome-level results are not consistently disaggregated by gender and other disadvantaged groups. Results are often reported for beneficiaries described as ‘people’, ‘communities’, and ‘innovators’, where gender and social inequalities are invisible. • GC results frameworks mostly do not to integrate GESI by having clear goals/objectives, indicators, and targets which could demonstrate differential changes for different groups. A few GCs included references to GESI in the results framework, but still referred to terms like “marginalized children” and “children living with disability”, or “disadvantaged groups” etc. (e.g., ACR GCD212). KEY INSIGHTS Insights are based on the meta-evaluation review and other experience of evaluating and managing funds and working with grantees, designed to aid consideration of GESI in future GC implementation - grant management, technical support, and MEL. • It cannot be assumed that GC grantees understand or appreciate the need to build gender equality and social inclusion 184 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION into implementation in a given context. It is therefore important to gauge understanding and build awareness and increase knowledge of these complex concepts and understand their application in the context in which they are working, for example by understanding the policies, structures, behaviors, and social norms, including in a particular sector, which shape different needs or barriers to access. • Grantees need support to develop and apply a GESI lens and build GESI into their programing, for example through training, convening, and sharing learning with grantees. This includes supporting them on how to carry out gender and social analyses to understand the differential power structures/imbalances, the structural barriers faced by disadvantaged groups, the behaviors which discriminate women, men, boys and girls, and other social disadvantaged groups, and the differences between men and women in how they might benefit from an innovation. SWFF produced gender guidelines in order to support grantees on this topic. • Identify entry points within programs where one can make tangible changes to strengthen GESI, and where measurable progress can be achieved, such as in targeted funding round focus, marketing strategies, or the selection process. • Grantees need support to develop monitoring and evaluation frameworks, which include indicators and targets for GESI, and on how to monitor progress to learn and adapt their program for greater differentiated GESI results. This is crucial as without a strong monitoring and evaluation framework, it is not possible to know clearly who an innovation is reaching and with what outcome. To illustrate, SWFF measured benefits from innovations including improvement in income resulting from innovation use for both women and poor people. This revealed that these groups were benefitting less than all respondents, which would not otherwise have been apparent. • Support can be provided through practical tools such as guidelines and checklists disseminated through appropriate channels (for example as provided by CHIC to applicants and by SWFF during implementation). A13.2.3 GESI IN RESULTS This section looks at the extent to which GESI has been considered in the results reported of the GCs portfolio. The following review questions have been asked to that end: Lines of enquiry What are the reported results in terms of gender equality and social inclusion in the GCs? To what extent has innovation contributed to better outcomes for marginalized groups? KEY FINDINGS • Three out of nine GCs (CHIC, SL@B and SWFF) have demonstrated clear disaggregated results, but otherwise the presentation of results on GESI is very limited. This relates directly to the absence of explicit inclusion of GESI in most of the GCs results frameworks. (See Annex Table 14 for a GC overview). • Those GCs which had strong integration of GESI including in the MEL frameworks, have been able to report differentiated results (for women particularly) as the examples below demonstrate while those which have not considered GESI at the design and implementation stage were clearly not able to demonstrate any socially differentiated results. Examples of strong GESI consideration include the following: • In two of the funds (CHIC and SL@B) there is clear evidence of disaggregated results and impact. For example, CHIC details an increased number of lives saved and lives improved and presents disaggregated data (number of of lives saved – Total: 112 of which 63 female, 59 male; 63 adults, 29 adolescents, 30 children; and number of of lives improved – Total: 37,195 of which 18,628 female, and 18,567 male). Similarly, SL@B innovations targeted the causes of maternal and newborn death and saved over 11,500 maternal and newborn lives; 58 (39 percent) awards were made to innovations led by women, with proportionally more female leadership amongst high income grantees (44 percent compared to 26 percent in LMICs); and some SL@B awards enabled innovators to focus their efforts on reaching poorer and more disadvantaged populations. • Women and vulnerable groups have been positively impacted from SWFF-supported innovations but not to the same degree as other beneficiaries. (Outcomes measured are in relation to agricultural productivity, income, access to low interest rate loans and USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 185 water availability).213 The SWFF final evaluation recommends further investigation into uneven benefits from innovations for women and poor people and advises that attention needs to be given to poor end-users to ensure greater benefit from innovations and improved income. Despite the lower impact for women and the poor, there was encouraging progress made in gender integration by SWFF-supported innovations. According to the evaluation, “more emphasis was placed round after round on integrating women at all levels of innovations, whether as Customers/End-Users or as Managers and Executives of the innovations themselves.” • The ACR GCD components that were targeted at people living with disability gained greater traction and were also successful in producing innovations that help people living with disability to read more easily. For example, ACR GCD grantees have developed learning tools that are suitable for people with difficulties (hearing impairment and learning difficulties). Grantees included NGOs and small and medium enterprises working with people with disabilities had access to the funds. By widening the pool, the GC could cover greater geographies. Gender was not considered within disabled groups targeted by ACR GCD innovations and no disaggregated data on differential impact (girls, boys, women and men) exist as yet. 213 Final Performance evaluation report, August 2020. KEY INSIGHTS • Uneven results underscore the importance of strong design that takes GESI into consideration in the theory of change, the objectives and development of MEL frameworks with clear disaggregated indicators, as well as in monitoring progress, using GESI specific indicators/targets, to demonstrate differentiated results. • Lack of disaggregated results in some GCs does not mean lesser achievement of results compared to the GCs with GESI consideration. Rather, it means those with disaggregated results have been able to demonstrate the differentiated impact of GCs on the different social categories and therefore the GC’s impact on poverty, gender equality, and social inclusion. A13.3 CONCLUSIONS Although GESI was not inherent or a focus in the design of GCs, nearly half of GCs have considered GESI - some explicitly and others implicitly - during the design and implementation phases in particular. The level of ambition on GESI has increased during the course of implementation. Encouragement and development of tools and support by USAID and partner donors has had some influence on stronger consideration of GESI during implementation. Although a few GCs have been able to demonstrate clear differentiated impacts, GESI has generally, however, not been a prominent feature in MEL frameworks and processes, and this seems to have affected the limited presentation of differentiated impact across the portfolio at an aggregate level. ANNEX TABLE 14: LEVEL OF GESI INTEGRATION IN GC RESULTS NAME OF GC LEVEL OF INTEGRATION FOCUS /SECTOR CHIC High Humanitarian SL@B High Health (pregnant women and newborns) SWFF High Water PAEGC Medium Agriculture ACR GCD Medium Disability/Education MAVC Medium Governance Zika Low Health (Zika) EBOLA Low Health (Ebola) SOGE Low Energy Note: Rating reflects the extent to which GC results and outcomes have been achieved for women and other disadvantaged groups. Where results have not been reported in this way at all, the rating is low as there is no evidence for the impact the GC has had. 186 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Insights highlight the importance of: • Contextual analysis that integrates GESI, demonstrating the systemic barriers and challenges that different disadvantaged groups face. • Theories of change that include specific GESI perspectives and develop clear assumptions of how poor or disadvantaged groups would benefit. • Specifying and differentiating target groups. • Defining and specifying terms related to GESI. • Understanding and raising the awareness and knowledge of grantees about GESI in a given context. • Developing monitoring frameworks and tools that integrate GESI-related indicators and targets, in order to monitor progress, and report the differentiated results and impact. • Recommendations of how GESI can be strengthened in GCs design and implementation, to enable clear reporting of differentiated results and therefore demonstrate the impact of future GCs on poverty reduction and gender equality and inclusion, are presented below. A13.4  RECOMMENDATIONS: LEARNING POINTS FOR FUTURE PROGRAMING These recommendations are principally directed to GC Partner Managers but oversight, guidance, and direction from USAID GC Managers, and at a strategic level from USAID Policy teams are also needed, to create the framework, capacity and tools needed for meaningful integration of GESI into GCs. • Embed GESI into GCs from the outset, during the design. Contextual analysis should demonstrate the systemic barriers and challenges that disadvantaged groups face within the overall challenge being addressed by the GC. These particular barriers and challenges also need to be addressed if the GC is to reach disadvantaged groups effectively. They include policies, unequal power structures, inequalities, and entrenched social norms, as innovations reach and affect various social groups and genders differently. This analysis is important and should feed the objectives and strategies of a program. This analysis would also inform how and why GESI is relevant to a particular program or sector and enable sufficient resources to be secured to integrate GESI considerations into the GC. Centrally produced guidance for GCs on how to integrate GESI would provide them all with a good basis for action and a clear understanding of the expectations and ambitions which USAID has for GESI within its GC programs. • Ensure that GCs have a robust Theory of Change which integrates GESI. Design frameworks need to identify clear pathways for change based on clearly articulated assumptions, which take into account GESI. It is important that the ToC demonstrate the pathways through which different disadvantaged groups, such as poor people (defined), women, and disabled persons (e.g., hearing impairment or other) would benefit from the GCs. • Design a MEL framework which reflects GESI and makes different groups visible. It is paramount to develop and design results frameworks that include clearly defined GESI￾related objectives, indicators, and targets and demonstrate specific changes for different social groups, and by gender, to ensure that GCs are working to reduce disparities, and contributing to developmental aims. These principles should apply results frameworks for both the GC as a whole, and for individual grantees. As illustrated earlier, this makes the impact on different groups visible and enables steps to be taken where they are being left behind. • Specify the target beneficiary groups. As programs impact differently according to gender and other social groups, it is critical that GCs specify their target ‘beneficiaries’ as clearly as possible, instead of formulating them in general terms such as ‘people’ or ‘citizens’, etc. GESI integration is weakened when targeted people USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 187 are described in general terms rather than more specifically (e.g., by age, gender, social group). Specified targeting also helps to establish a clear understanding of potential differential impact on each specified group. Related to this, terminology matters in relation to GESI. Terms used must be clearly defined for applicants and for grantees. • Present GC results with disaggregated data in terms of different social groups and genders and support grantees to collect data accordingly, to feed in (e.g., with training or budget). It is important to be able to understand the extent to which a GC as a whole and individual innovations have made a difference on the lives of different target groups and their institutions. This should be supported by underlying analyses of the barriers and challenges faced by the different groups and how the innovations have addressed them. This provides a full picture on gender and social inequalities, and frames the changes achieved for each group. • Understand grantee capacity on GESI, raise awareness and knowledge of grantees on what the terms gender equality and social 214 Tools and guidance on gender equality have been developed by many donors, such as Sida, USAID and UK-AID; and other tools on social inclusion are available from INGOs (e.g., Oxfam) which future GCs can access to help as a guide or checklist on how to integrate GESI at the design phase in future GCs. • inclusion mean in a given context, and provide support. Support may include training, convening, and learning with partners on how to carry out gender and social analyses to understand the differential power structures/ imbalances, the structural barriers faced by disadvantaged groups, and the differences between men and women in how they might benefit from a program. Grantee support on GESI can be provided both by the fund manager, in the course of managing the grants, and through technical assistance for scaling and acceleration. • Develop tools and guidance214 that help integrate GESI considerations at all stages of GC design and implementation, from application, selection, project planning, and implementation to monitoring, evaluation, and learning. GCs should develop and disseminate these products to intended users (e.g., selection panels, applicants, and grantees) and provide training in how to use them. 188 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Photo: Global Devlab/USAID USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 189 ANNEX 14 CASE STUDIES A14.1  METHODOLOGY FOR CASE STUDIES 215 Evaluation Report, Qysas (Stories): An Arabic Leveled Digital Library, Implemented by Little Thinking Minds in Jordan SEPTEMBER 2017, https://www.littlethinkingminds.com/img/Qysas_Final_Report.pdf. The case studies provide detailed studies of four grantees that demonstrate key learning about thematic priorities for the meta-evaluation: scaling and sustainability, partnering with USAID Missions, and using MEL to support acceleration and scaling. Case study topics were selected from grantees shortlisted by USAID Grand Challenge Managers. Evidence was largely qualitative and collected through in-depth interviews with grantees and USAID, and from secondary sources including project reports, documents, and public information, such as grantee websites and media articles. More detail is provided in Annex 6.5 A14.2  LITTLE THINKING MINDS: LOCAL AND RELEVANT APPROACHES LEAD TO DEEPER IMPACT A14.2.1 CASE STUDY SUMMARY Grantee overview: Little Thinking Minds (LTM) is a company that produces educational resources that aim to enhance children’s skills and learning outcomes and increase their connectedness to the Arabic identity. The company was founded in 2004 by Rama Kayyali and Lamia Tabbaa to create educational Arabic content for preschoolers in the form of videos and apps. The collaboration of LTM, Integrated Services–Indigenous Solutions (IS), and Jordan Education Initiative (JEI) strengthened the project portfolio by bringing in diverse and unique perspectives and added to the research rigor of the project. LTM products are used by native and non-native Arabic speakers in over 200 schools and by 80,000+ students across the region. Project achievement and sustainability: The grant from All Children Reading Grand Challenge for Development (ACR GCD) in 2014 enabled LTM to develop a child-centered Arabic early grade reading and learning platform, Qysas. LTM and Integrated Services have further collaborated in a follow-on iteration of the original ACR GCD project that is still student-centered but led by teachers and complemented by classroom-based activities. The Qysas has now transitioned from being a digital library of regionally sourced award-winning books to a much wider-scale iteration, ‘Let’s Live in Harmony’, implemented in partnership with Jordan Ministry of Education, to scale results and to ensure wider impact. Key learning: • Local leadership, understanding, and use of language were all key to the innovation’s success. • Having a MEL partner from the application stage onwards and making good use of data enabled continuous learning and adaptation. • GC funding brought a private sector business model into the public sphere. This and the role of the project’s implementing partner, led to successful collaboration with and uptake by government. • The ACR GCD grant led to LTM developing learning platforms for children with learning difficulties and becoming more inclusive. A14.2.2  ACR GCD GRANTS TO LITTLE THINKING MINDS Little Thinking Minds was successful in winning a grant with ACR GCD in 2014 (Round 2). The details are given below:215 • Project Name: Qysas (Stories): An Arabic Levelled Digital Library for Every Classroom • Challenge & Solution: Class discussion books in Jordan teach students overwhelmingly how to read whole words rather than first introduce vowel and letter phonics. This approach to 190 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Arabic teaching has not produced results and poses complicated demands on young children in acquiring literacy. Qysas’ interactive and animated early class Arabic literacy app has over 125 e-books and a levelled reading platform that automatically provides feedback and advice through a child-centered and diversified learning approach. • Location Implemented: Jordan • Competition Entered: Grant Competition 2014 • Status: Winner • Collaborators/ Partners: Integrated Services – Indigenous Solutions, a research/MEL organization, and Jordan Education Initiative and Abdul Hameed Shoman Foundation, both non-governmental organizations. • Duration: Jan 2015 – July 2017 • Students in the Qysas project, at Jordanian public schools, accessed the platform during literacy group sessions held at their school. The platform was also available to parents for home use with their children on digital devices. A14.2.3 KEY LEARNINGS The focus on local language was a key to local acceptance. LTM implemented the Qysas (Stories): An Arabic Leveled Digital Library for Every Classroom project to address the lack of levelled Arabic-language reading materials available to early primary school students in Jordan. LTM is a Jordanian women-owned for-profit business that creates engaging, digital literacy tools to improve children’s Arabic reading skills. The project aimed to increase early grade literacy skills in Arabic— specifically, oral language and vocabulary, fluency, and comprehension—by providing self-paced, interactive, electronic reading materials that supplemented ongoing classroom instruction. Being local to the region, the project under the leadership of LTM was successfully able to mobilize resources to develop local and relevant content for children. Selected books represented a range of reading levels and topics and all were sensitive to gender and culture. LTM worked with regional publishers to select 145 engaging books to develop into e-books loaded onto the Qysas platform. Students logged on to the platform using a unique username and password ensuring that each student’s unique usage data was captured through the platform. Each ebook had corresponding audio voice-overs to allow students to listen as they read and quizzes to assess students’ comprehension. LTM grouped the e-books into seven levels, and all students began at level one. Qysas by LTM also helped to increase student engagement. A qualitative study by Integrated Services found that the Qysas platform boosts academic confidence and encourages children to attend school. A scalability assessment was conducted on the Qyas platform and LTM scored highest of all the grantees assessed. The average statistic for the Arab region is that a child reads one book per year, according to a study conducted by Integrated Services. In this program, in the space of one academic year, children read 125 books, which was a huge achievement. (LTM even had to add an additional 25 books to meet the pace at which the children were reading.) LTM have demonstrated that once children have access to engaging and appropriately levelled reading materials, their reading levels increased dramatically: as they put it, “reading is an issue of access, not culture”. “The average statistic for the Arab region is a child reads around 17 minutes per year, or one book per year. In this program in the space of the academic year, kids read 125 books, and before the year was finished, and Little Thinking Minds had to add an additional 25 [books] to meet the pace at which the kids were reading. So, moving from one book per year, to 150 books per year, it became clear that lack of reading habits arises not because of the Middle East’s lack of a reading culture but an issue of access.” LTM interviewee Having a learning partner right from the application stage contributed to continuous learning and adaptation in the innovation process. LTM collaborated with Integrated Services, an organization specializing in monitoring, evaluation and learning (MEL), to maximize the impact of development projects throughout the Middle East and North Africa. The collaboration strengthened the project portfolio by bringing diverse and unique perspectives into the project and added to the research rigor of the project. “QYSAS is the only Ed-Tech program that has been controlled, trialed, and as rigorously assessed”. Case study interviewee LTM focused on developing engaging content, while Integrated Services looked at dissecting the project from evaluation, testing, literacy acquisition and pedagogical angles, which contributed richly to the overall goal of ACR GCD towards developing Edtech USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 191 for literacy. 2014, the year the grant was launched, was also the period of a huge influx into Jordan of Syrian refugees. A very large number of children were out of school and did not have basic literacy. Around this period, there was a lot of research being done in the education sector on the poor performance in the early grades of children in Jordan and nearby countries in the MENA region and the difficulties of learning Arabic. Integrated Services identified the set of challenges evidenced by the research and worked with LTM to see how together they could address these challenges, developing a proposal to develop a simple tech product-level Digital Library. Notably, while the other grantees of ACR GCD in this round tended to hire or subcontract MEL firms for their specific tasks for baseline and end-line data collection, Integrated Services offered continuous monitoring, learning and refinement to the Qysas project. ACR GCD was instrumental in making a business model which was active in the private sphere open and accessible to the public sphere. Without ACR GCD, students in public schools might not have been able to use the reading facilities that they are accessing now. Prior to ACR GCD, LTM’s business model was to work with private schools on a subscription basis. Either the schools or individual students subscribing to access a wide range of materials. Through the ACR GCD grant, the approach was successfully scaled up and, with the Jordan Education Initiative (JEI) as a partner, run in public schools. Private sector companies often have excellent content but are often deterred from working with government and ministries due to concerns about unreliability and unpredictability. With strong backing of USAID through ACR GCD, LTM could successfully collaborate with a government-based organization like JEI. The project implementers agreed that LTM has benefited greatly from the public-private partnership (PPP) model enabled by ACR GCD and the support of solid agencies like USAID and World Vision (ACR GCD’s flagship and managing partner). “All children reading, was particularly flexible in working with private sector actors who have really excellent content but have fears of entering into the working with the ministries/Governments as it’s an unreliable space for them.” Case Study Interviewee 216 Due to technical and implementation challenges during the first year of implementation, the project was extended through the 2016–17 academic year. Specifically, in countries in the MENA region that require strong partnerships with the government to achieve scale-up, this unique ACR GCD combination strengthened LTM’s projects sustainability and ability to scale. LTM found that innovation demands continuous learning, adaptation and transformation. This included responding to problems with network and internet connectivity, but LTM was flexible enough to transform its product from online to offline depending on the requirements. Although the platform was originally designed to be used online, LTM also programmed an offline option to accommodate internet connectivity challenges at the schools. Participating schools provided students with opportunities to use the tablets and the Qysas platform either during class or after school during literacy group sessions. These sessions were held twice a week for 45 minutes, at which time students could listen to and read along with e-books and take the corresponding quizzes. The data and evidence generated on the use, effectiveness and impact of the technology at the student level was a key to wider expansion of the technology. LTM implemented the Qysas project over two academic years with two cohorts of Grade 2 students at ten intervention schools. Implementation began in selected schools with the first cohort of Grade 2 students during the 2015–16 academic year with teachers and JEI interns responsible for overseeing the literacy groups. Student usage data was downloaded onto portable memory devices during school visits and teachers and project staff monitored literacy group sessions to check on student progress and verify that students’ reading abilities matched the level of e-books they were reading on the Qysas platform. A new cohort of Grade 2 students was selected from the same schools in the second year.216 LTM used data and learning from year 1 to improve the Qysas platform and stabilize the implementation model. Additionally, in the second year, only teachers, not JEI interns, led literacy groups, opting to do so at the end of the school day as part of an unstructured literacy strengthening class. “The transition during COVID-19 of the program to an online platform, based on the request of the ministry that was done completely pro bono at cost of the two companies interest to continue having kids read, particularly when they’re at home”. Case Study Interviewee 192 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Although not initially targeted at inclusive learning, through the ACR GCD grant Qysas was encouraged to develop learning platforms for children with learning difficulties. The books that were originally chosen for Qysas, represented diversity in gender roles, tried to showcase the different roles that women and men can play and represented women and men on equal terms. “The way the program is designed, if you have a learning disability, [it can] be used regardless of your reading ability.” Case Study Interviewee This was built on when Qysas was further developed through the “Let’s live-in harmony” project, when there was particular attention on inclusion and social cohesion for Syrians and Jordanians. Bringing content for deaf children is also under active consideration by the team. Further, Qysas allows children with learning difficulties or poor reading ability to use it, through an audio function. ACR GCD required applicants to have strong local partnerships. This prompted LTM’s collaboration with Jordan Education Initiative which opened doors for LTM. JEI is rooted in provision in the public sector and strengthened LTM’s relationship with the USAID Mission in Jordan. Further, its connection to Queen Rania of Jordan, as one of ten education initiatives personally supported by her, gave them traction with the Government and others. JEI facilitated LTM’s relations with Ministries and enabled project set-ups in the intervention schools. A14.2.4 KEY ACHIEVEMENTS AND SUSTAINABILITY PROSPECTS The Qysas program has the potential for scalability across the Arab world and has demonstrated that there is a market in the MENA region for benchmarked, levelled literacy programs that generate tangible literacy results.217 LTM’s business model for the public sector is currently being scaled at the Ministry of Education in Jordan. In the RiseUp Summit in Cairo, 2018, Little Thinking Minds raised $1.265 million in Series A funding, the funding round led by the largest Egyptian VC firm, Algebra Ventures, joined 217 https://www.globalinnovationexchange.org/innovation/little-thinking-minds. 218 https://www.menabytes.com/little-thinking-minds-series-a/. 219 https://www.arabnet.me/english/editorials/entrepreneurship/investment/little-thinking-minds-raises--1-265m-led-by-algebra-ventures. 220 https://www.entrepreneur.com/article/350439. 221 https://magnitt.com/news/little-thinking-minds-classera-partner-47670. by recently launched Dubai-based Mindshift Capital that invests in women-led companies, and Saudi’s Al Turki Group.218,219 LTM has expanded across the MENA region and has raised funding for Qysas. LTM now has offices in Jordan, Saudi Arabia, and the UAE, and was recently declared the winner of the first edition of the Access Sharjah program run by Sharjah Entrepreneurship Center (Sheraa). Launched late in 2020, Access Sharjah is a global platform that connects digital startups to relevant public and private entities in Sharjah and across the UAE, and it led to LTM being awarded a $100,000 equity￾free grant to further their work in the fields of knowledge and culture.220 Additionally, LTM has signed a partnership agreement with Classera Inc., the international e-learning solutions provider with the biggest market share in LMS (Learning Management System) for the K-12 segment in the Middle East Market and LTM’s flagship product, I Read Arabic will be integrated into Classera’s innovative e-learning solution for K12, making it accessible to a global user base of over two million users.221 LTM and Integrated Services have collaborated on a follow-on iteration of the project that remains student-centered but is led by teachers and complemented by classroom-based activities. The focus of the Digital Library is social cohesion for Syrians and Jordanians, teaching a sense of self, community, family and self-expression through storytelling, complemented with teacher-led classroom-based activities to reinforce concepts taught in the stories. The project has thus evolved from being a self-paced digital library to a teacher-run activity that supports literacy in the classroom while promoting social cohesion through stories, storytelling and activities. The Qysas has now transitioned from being a digital library of regionally sourced award￾winning books to the current iteration (Let’s live in harmony) and in partnership with the Jordan Ministry of Education, Qyas now has been developed into an independent library corresponding to grades K through three. “And, to the extent that ministries are engaged from the get-gi in the design, it also builds ownership.” Case Study Interviewee USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 193 The Qysas curriculum is now approved by the MoE and, through being engaged in design from the outset, Ministry officials have a sense of ownership 222 This case study was based only on secondary literature review. 223https://markets.businessinsider.com/news/stocks/gradian-health-systems-wins-saving-lives-at-birth-award-to-improve-maternal-newborn-healthcare-in￾zambia-1002224292, 224https://www.gradianhealth.org/press-release-gradian-health-systems-wins-saving-lives-birth-award-improve-maternal-newborn-healthcare-zambia/. 225 A CE label is a declaration of compliance with the relevant or applicable health, safety, and environmental protection legislations for products sold or manufactured within the European Economic Area. 226 https://dukeghic.org/wp-content/uploads/sites/20/2020/06/E-SL@B-Final-Evaluation-Report.pdf. 227 https://www.usaid.gov/sites/default/files/documents/1864/Adam-Lewis-Innovator-Spotlight-508.pdf. 228 https://www.healthynewbornnetwork.org/news-item/fifteen-innovators-get-funding-saving-lives-birth/. for the Let’s Live in harmony project. This has built greater momentum towards scaling up. A14.3  GRADIAN: SCALING TAKES MUCH MORE THAN A GOOD INNOVATION222 A14.3.1 CASE STUDY SUMMARY Grantee overview: Gradian is a non-profit medical technology company that works to transform the impact of medical equipment in low-resource hospitals around the world. In July 2017, Gradian’s project – a partnership that includes the Society of Anesthesiologists of Zambia (SAZ), Tropical Health & Education Trust (THET), Sonergy Diagnostics, the Churches Health Association of Zambia (CHAZ), and the Nick Simons Foundation won a USAID, ‘Saving Lives at Birth’ (SL@B) Grand Challenge Transition to Scale grant to scale up the Universal Anaesthesia Machine (UAM) – the world’s only internationally-certified anesthesia machine designed to work without electricity and medical oxygen.223 224 Project Achievements and Sustainability: Gradian received a SL@B TTS grant to improve surgical and obstetric care in Zambia by scaling their CE-certified225 Universal Anesthesia Machine (UAM) pairing it, for the first time, with a simulation-based course in 2017.226 Scaling achievements since then have been significant. Having already contributed to thousands of safer surgeries in Zambia and nearly 20 other African countries, the UAM is a now proven source of anesthesia for surgery in hospitals facing unreliable power and oxygen, such as those in rural districts of Zambia. The UAM has been installed in over 500 health facilities in almost 30 countries since its first introduction. Key Learning: • The scaling process was strongly supported by Gradian’s prior experience in the market and pre￾existing partnerships with local stakeholders, especially the Government of Zambia. • A strong MEL plan (including theory of change) that produced data on lives saved helped Gradian to effectively advocate for scaling up. • Prioritizing local capacity building and outreach helped Gradian to connect to communities and create demand. • Also, key was partnering with local and national governments, as many of the hospitals that buy the organization’s products were government-funded public facilities. A14.3.2  SL@B GRANTS TO GRADIAN Saving Lives at Birth sought innovative ideas to leapfrog conventional approaches in three main domains: (1) technology; (2) service delivery; and (3) “demand side” innovation that empowers pregnant women and their families to practice healthy behaviors and be aware of and access health care during pregnancy, childbirth, and the early postnatal period, especially the first 2 days after birth. Prior to winning the ‘Transition to Scale’ award with SL@B, USAID Missions have funded the procurement of Gradian’s equipment for several years, as a direct customer in Zambia and a project donor in Ethiopia and Guinea.227 Gradian Health Systems (New York, USA) was selected under the Transition-to-Scale Award nominees in July 2017 to scale up specialized training and technology provision for reliable anesthesia to improve surgical and obstetric care, 228 one of only 15 awards selected from more than 550 applications. 194 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION • Challenge and Solution: Gradian’s Saving Lives at Birth project sought to overcome the challenge of high maternal and newborn mortality rates and limited access to surgical care, such as Caesarean sections, by equipping 33 hospitals in four provinces of Zambia with the UAM, providing specialized clinical courses on obstetric and pediatric anesthesia and offering a technical workshop on devices required for safe surgical and obstetric care. The goal was to commercialize and scale this program throughout the country, by focusing on technology, training and services. • Location Implemented: Zambia • Collaborators / partners: Society of Anesthesiologists of Zambia (SAZ), Tropical Health & Education Trust (THET), Sonergy Diagnostics, the Churches Health Association of Zambia (CHAZ), and the Nick Simons Foundation A14.3.3 KEY LEARNINGS Gradian’s experience in the market prior to receiving the grant from SL@B and pre￾existing partnerships with local stakeholders, especially the government in Zambia aided the scaling process. Before receiving their SL@B grant, Gradian had already identified Zambia as a priority market for the UAM. The Zambian government had just launched a National Surgical, Obstetric, and Anaesthesia Strategic Plan that outlined the need for reliable and safe anesthesia (Republic of Zambia Ministry of Health, 2017). Gradian had seen some momentum from one￾off sales in the country. There was both need and demand for the UAM and for capacity building for the anesthesia workforce, but a lack of infrastructure was a barrier to effectively reaching end users. At the time, Zambia’s funding for medical equipment procurement was limited, so the SL@B grant allowed Gradian to equip Zambian Ministry of Health-selected hospitals with UAMs and ongoing simulation-based training (in centralized locations and on-site at the hospitals) for healthcare providers. Through the experience in Zambia, the Gradian team identified a need for more extensive capacity building for clinicians using the UAM. To that end, Gradian developed a three-day simulation-based training curriculum for its products to augment 229 https://dukeghic.org/wp-content/uploads/sites/20/2020/06/Building-for-Sustainability.pdf. 230 https://markets.businessinsider.com/news/stocks/gradian-health-systems-wins-saving-lives-at-birth-award-to-improve-maternal-newborn-healthcare-in￾zambia-1002224292. 231 https://dukeghic.org/wp-content/uploads/sites/20/2020/06/E-SL@B-Final-Evaluation-Report.pdf. 232 https://www.usaid.gov/sites/default/files/documents/1864/Adam-Lewis-Innovator-Spotlight-508.pdf. the one-day on-site orientation that traditionally accompanied the sale of any UAM unit.229 A strong MEL plan (including theory of change) that produced data on lives saved helped Gradian to effectively advocate for scaling up. As noted, Gradian’s goal was to commercialize and scale their UAM program throughout the country, by focusing on technology, training and services.230 They maintained a strong MEL mechanism that generated valuable data and stats on the impact. As a result, they were able to sell the technology to the government as well as the commercial sector. Gradian prioritized local capacity building and outreach that helped them to connect to communities and create demand. The UAM has a built-in oxygen concentrator that generates its own medical-grade oxygen from room air (without an external oxygen source) and comes with a three-year preventative and corrective maintenance warranty with local biomedical technicians and spare parts, as well as on-site and online simulation￾based training for anesthesia providers delivered by local trainers.231 In addition to the technology, their implementation model had three other elements that enhanced their scaling potential:232 1. Technology: To make world-class devices that meet international quality standards and have features designed specifically for low-resource settings. 2. Training: To provide on-site user training every time we install a product, including simulation scenarios and proctored cases. 3. Customer Service: To provide reliable after-sales support for all users and customers through a distribution network of local companies, entrepreneurs, and technicians. Gradian partnered with local and national governments, as many of the hospitals that buy the organization’s products were government-funded public facilities. Gradian targeted the Government not only as a policy partner but as a potential customer of their products. Team members see relationship-building with governments and alignment with national strategies as critical to the success of their work USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 195 and recommend that innovators who want to work with governments engage them from the beginning of the project. Early engagement with partners can help innovators figure out where the gaps are in the health system and how to fill them.233 A14.3.4 KEY ACHIEVEMENTS AND SUSTAINABILITY PROSPECTS Having already contributed to thousands of safer surgeries in Zambia and nearly 20 other African countries, the UAM is a proven source of anesthesia for surgery in hospitals facing unreliable power and oxygen, such as those in rural districts of Zambia. The project is equipping hospitals with the UAM and providing specialized capacity building opportunities for all anesthesia providers, including a mobile, simulation￾based course pioneered by partners at the Society of Anesthesiologists of Zambia (SAZ) and Johns Hopkins University School of Medicine. Gradian is working with Tropical Health and Education Trust (THET), the Churches Health Association of Zambia (CHAZ), and the Ministry of Health to train local biomedical technicians on equipment and nearly 10 other devices needed for safe childbirth.234 The UAM has been installed in over 500 health facilities in almost 30 countries since its first introduction. Gradian has trained more than 1,000 clinical users of the UAM in the process of scaling this innovation. A pilot study conducted in Northern Nigeria found no malfunctions and far fewer complications than other forms of general anesthesia, in addition to significant savings on oxygen costs. The same research found that UAM was the preferred method for administering anesthesia to patients. A study that assessed potential failures of the machine concluded that UAM is a reliable and safe anesthesia workstation.235 • Project and subsequent achievements are significant, despite challenges experienced related to installation sites and infrastructure in Zambia: • Gradian installed the UAM in 32 hospitals and trained more than 70 anesthesia providers during 233 https://dukeghic.org/wp-content/uploads/sites/20/2020/06/Building-for-Sustainability.pdf. 234 https://www.usaid.gov/sites/default/files/documents/1864/Adam-Lewis-Innovator-Spotlight-508.pdf. 235 https://www.usaid.gov/sites/default/files/documents/CII_Global_Health_Innovation_Index_.pdf. 236 https://dukeghic.org/wp-content/uploads/sites/20/2020/06/E-SL@B-Final-Evaluation-Report.pdf. 237 https://dukeghic.org/wp-content/uploads/sites/20/2020/06/E-SL@B-Final-Evaluation-Report.pdf. 238 https://dukeghic.org/wp-content/uploads/sites/20/2020/06/Building-for-Sustainability.pdf. 239 https://dukeghic.org/wp-content/uploads/sites/20/2020/06/Building-for-Sustainability.pdf. 240 https://www.usaid.gov/sites/default/files/documents/CII_Global_Health_Innovation_Index_.pdf. 241 https://dukeghic.org/wp-content/uploads/sites/20/2020/06/E-SL@B-Final-Evaluation-Report.pdf. 242 https://dukeghic.org/wp-content/uploads/sites/20/2020/06/E-SL@B-Final-Evaluation-Report.pdf. the SL@B funding period. As a result, Gradian is beginning to see this model generate public and private sector demand for anesthesia care capacity, the UAM, and Gradian’s simulation-based training in alignment with Zambia’s first-ever National Surgical, Obstetric, and Anaesthesia Strategic Plan.236 • The achievement during the SL@B grant period validated the combined UAM and training package and helped to catalyze scale to other countries in East Africa.237 • The team has since scaled this model to Tanzania, where they won a tender with the Ministry of Health for UAMs and have strong relationships with academic institutions. Gradian now has a simulation lab at several medical education institutions across the country.238 • The team is working on new tools for markets where there is a high concentration of out￾of-warranty machines, such as the Gradian Advantage Plan that would provide refresher training and an extended service warranty at a tiered pricing model.239 • In 2018, Gradian launched Comprehensive Care Ventilator, another critical care product.240 • Furthermore, in December 2019, Gradian won a World Bank Global Financing Facility (GFF) award (approximately $4 million) to reduce maternal and newborn mortality. The GFF award will support the UAM scale-up in Tanzania and Sierra Leone, which is estimated to increase access to safe anesthesia deliveries and surgeries by 200,000 per year.241 SL@B provided acceleration support to Gradian, and it gave Gradian and their local partners the opportunity to field test and refine a simulation-based product training curriculum and methodology that has revolutionized Gradian’s model and increased their potential for impact. The funding, networking opportunities, and the tailored support received through the program catalyzed opportunities for scaling this model within Zambia and across other countries, such as Tanzania and Uganda.242 SL@B also supported Gradian to engage physicians from Sierra Leone, Zambia, and 196 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Tanzania to design, formalize, and accredit a new simulation based UAM training curriculum. The curriculum incorporates key anesthesia and critical care concepts, coaching on core techniques, and an intensive set of real-world simulation scenarios.243 The support from SL@B was highly helpful to Gradian to move the UAM innovation along with an increasing understanding of the market Gradian is targeting or working in, providing reference letters, nominating and/or connecting Gradian for speaking / conference opportunities (e.g. Grand Challenges Meeting, Unite for Sight, and others), pitch support 243 https://www.usaid.gov/sites/default/files/documents/CII_Global_Health_Innovation_Index_.pdf. 244 https://dukeghic.org/wp-content/uploads/sites/20/2020/06/E-SL@B-Final-Evaluation-Report.pdf. 245 https://dukeghic.org/wp-content/uploads/sites/20/2020/06/E-SL@B-Final-Evaluation-Report.pdf. from professional coaches (e.g. preparation for the DevX pitch competition), highlighting UAM in communications pieces (blogs, briefs, social media), and promoting through media channels).244 SL@B funding and USAID support has been significant. Gradian indicated in a research study by Duke University that SL@B helped them to connect with potential partners for scale (non￾funding). Further, the technical advice provided by USAID Mission(s) was also instrumental in their journey to scale.245 A14.4  EKITABU: KEEPING IMPACT AT THE CENTER TO ENABLE SCALE UP A14.4.1 CASE STUDY SUMMARY Grantee overview: As the digital revolution spread across Africa, more and more devices were being supplied to schools yet growth in hardware was not being matched by useful and accessible learning content. It was this situation which led to the creation of eKitabu in 2012. Established in the U.S. but based in Kenya, eKitabu offers over 350,000 e-books plus digital content to 14 African countries through an app and e-library that are usable offline and apply open standards. In 2016, the Ministry of Education (MoE) in Kenya approached the EdTech startup to create accessible content for the country’s new digital literacy program, which involved the rollout of more than a million tablets and laptops to learners in primary schools across Kenya. According to MoE data, deaf children made up more than half of all learners with disabilities in Kenya’s public school system, so adapting learning materials for Kenyan Sign Language (KSL) acquisition and literacy were flagged as a top priority. The result was the development of eKitabu’s Studio KSL to help the deaf community and local content creators to integrate KSL videos into early grade reading materials, thereby producing visual storybooks in support of Kenya’s new inclusive education policy. Project Achievements and Sustainability: As an awardee of All Children Reading: A Grand Challenge for Development (ACR GCD), eKitabu has developed video storybooks for literacy and has brought digital content to more than 1,500 schools across all 47 counties of Kenya and 13 African countries. In 2017, as Studio KSL was in development, All Children Reading announced its Sign On For Literacy prize, which sought innovations that increase literacy outcomes for deaf children in low-resource settings. ACR GCD gave eKitabu an opportunity to leverage the prize to take Studio KSL even further, launching an equivalent in Rwanda. It has produced storybooks in both Rwandan and Kenyan sign language, as resources for deaf children and their parents, created employment for deaf young people and generated valuable research. ACR GCD has greatly helped eKitabu to scale both in terms of formalizing e-book creation and producing creative commons license learning materials, which are now part of the Global Digital Library. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 197 Key Learning: • ACR GCD’s prioritization of people living with disability cemented eKitabu’s ability to develop learning materials for children with learning difficulties. • The opportunity for private businesses to apply for ACR GCD grants was pivotal to the success and sustainability prospects of eKitabu. • eKitabu has made impact the focus of innovation, and inclusivity a key factor in design. • Cost-effectiveness analysis helped eKitabu to expand its sale and make a case to attract more resources. • Close communication with the USAID Mission and ACR GCD funding contributed to securing collaboration with the in-country government. • eKitabu has benefitted from improvements in the ACR GCD approach to monitoring, evaluation and learning (MEL). 246 “Born accessible” refers to building accessible books right from the beginning, and building the process into current ebook production workflows, instead of taking apart and updating books post-production to make them accessible. A14.4.2  ACR GCD GRANTS TO EKITABU eKitabu has had three ACR GCD projects as follows: USAID ACR GCD BOOK BOOST: ACCESS FOR ALL CHALLENGE, 2018 Project Name: Driving Local Development of Born Accessible246 Titles in Kenya. eKitabu aimed to catalyze the born accessible book chain in Kenya to create a library of open-source, accessible digital titles, adapting its open standard-based toolkit for local content developers. • Location implemented: Global. • Technology Used: Assistive Technology, Software/Apps. • Amount Award: $162,000. • Status: Winner. • Collaborators/Partners: Digital Literacy Trust. USAID ACR GCD SIGN ON FOR LITERACY PRIZE, 2019 Project Name: Studio KSL: eKitabu launched Studio KSL to help the deaf community and local content creators integrate sign language videos into early grade readers and produce visual children’s storybooks in support of Kenya’s new inclusive education policy. Investing to set up Studio KSL and streamline production with deaf actors was intended to lower the cost of producing quality visual storybooks and help to document regional differences in Kenyan Sign Language. • Location Implemented: Kenya. • Technology Used: Assistive Technology. • Amount Award: Up to $250,000. • Status: Finalist. • Collaborators/Partners: Deaf Ability Initiative and The Kenya Society for Deaf Children. 198 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION USAID ACR GCD BEGIN WITH BOOKS, 2020 Project Name: Open Books Malawi: eKitabu aimed to scale work begun in Kenya and Rwanda through ACR GCD’s Book Boost and Sign On For Literacy prizes to implement the Open Books Malawi initiative. The project is expected to deliver a total of 270 books - 220 in Tumbuka and 50 in Malawian Sign Language - to the Global Digital Library, which already houses more than 120 accessible African e-publications digitized by eKitabu. • Location Implemented: Malawi. • Technology Used: Software/Apps. • Status: 1st Place, Winner. • Collaborators/Partners: Juárez & Associates, Malawi National Association of The Deaf (MaNAD), Malawi Institute of Education (MIE). A14.4.3 KEY LEARNINGS ACR GCD has greatly helped eKitabu to scale both in terms of formalizing e-book creation and producing creative commons license learning materials, which are now part of the Global Digital Library. The founders of eKitabu believe that they would not have come this far without ACR GCD funding. The USAID mission in Kenya in education that introduced eKitabu to the USAID Grand Challenge. Their application was successful, and they were awarded funding in the Book Boost Challenge in 2018, emerging as the winner of the competition. Book Boost did indeed give eKitabu a real boost, in terms of formalizing the process that they used to create eBooks. It also enabled eKitabu to develop a toolkit for local publishers and authors in Kenya and Rwanda that documented and formalized the internal process, resulting in an open-source accessible e-publishing toolkit: toolkit.ekitabu.com. The Book Boost grant also enabled eKitabu to explore opportunities in sign language video, which was accelerated under the “Sign on for Literacy” challenge of ACR GCD in 2019. ACR GCD was thus successful in serving as an incubator for new technologies and new capabilities in organizations like eKitabu. ACR GCD’s prioritization of people living with disability cemented eKitabu’s ability to develop learning materials for children with learning difficulties. In building capabilities to “With ACR’s help, the top three priorities for ekitabu in the past few years have been software development, team growth and country expansion.” Case Study Interviewee achieve impact in literacy, particularly with children with disabilities, ACR GCD had an explicit priority to include children with disabilities which very much resonated with eKitabu. eKitabu’s Studio KSL, supported by ACR GCD, has about 20 sign language storytellers and signers who are deaf, young people in their early 20s, mostly in Kenya, Tanzania, Rwanda, and Malawi, who produce the learning materials, typically in partnership with local disabled persons organizations. The studio is now continuing to operate on a sustainable basis. The opportunity for private businesses to apply for ACR GCD grants was pivotal to the success and sustainability prospects of eKitabu. The project implementers stated that eKitabu has greatly benefited from the Public-private Partnership (PPP) model of ACR GCD, backed by its strong donors and implementing institutions (USAID, the Australian Government and World Vision). This unique combination in ACR GCD has strengthened eKitabu in terms of sustainability and scale, especially making a difference in African countries that require strong partnerships with the Government for innovations to achieve scale-up. eKitabu has made impact the focus of innovation, and inclusivity a key factor in design. The design philosophy in eKitabu has been that “when you design your project for people on the margins, your project works better for everyone”. Inclusive education with a special focus on disabled groups has enabled eKitabu, with its partners of local and international publishers (e.g., African Storybook, Cambridge University Press, Gallup Press), to deliver content to over 1,500 schools across Kenya and 13 other countries in Sub-Saharan Africa, including Rwanda. eKitabu has reached scale across all 47 counties of Kenya. eKitabu is now in the process of securing private sector investment from funders in the U.S. to achieve its main priorities of software development, team growth, and country expansion. eKitabu currently has revenues from a number of contracts with various agencies and device manufactures, including UNICEF, Microsoft, and over 70 publishers within and outside Africa. According to the implementers, partnering with ACR GCD has augmented eKitabu’s value proposition, enabling them to build relationships with key people, get advice and connect with them to enhance impact. The eKitabu team also believes that investing in staff USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 199 with the right skill set and retaining them has also aided them to scale their operations and live up to their expectations. “One of the instrumental supports from ACR GCD beside the grant money was encouragement to focus on impact, particularly for children with disabilities. We have developed numerous learning product for children with disability which we would not have been at the priority if there was no ACR GCD grant” Case Study Interviewee. Cost-effectiveness analysis helped eKitabu to expand its sale and make a case to attract more resources. In one of the cost-effectiveness analyses conducted in the past by the implementers, under a project in DFID’s Girl’s Education Challenge, eKitabu was able to reduce the costs of delivering a set of curriculum materials by over 95 percent. The team is now interested in doing a relatively large-scale quantitative study that looks at not just the impact, but also the cost-effectiveness with solid economic and econometric analysis behind it. Improvements in the ACR GCD approach to monitoring, evaluation, and learning (MEL) has benefitted eKitabu. In the first collaboration with ACR GCD in round 2, there were no components for MEL explicitly in the design but from round 3, ACR GCD incorporated MEL into project design. The ACR GCD’s encouragement that eKitabu focus on impact, particularly for children with disabilities, is perceived to have been of great benefit to the project. The performance-based milestone approach was also described as very effective in keeping projects on track and increasing the productivity of the company. The team now plans to undertake a relatively large-scale quantitative study that looks not just at impact, but also cost￾effectiveness with a solid economic and econometric analysis behind it. Substantial work has been put into designing a Randomized Control Trial, that involves people with disabilities, especially children with disabilities, for whom it is particularly difficult to design and implement effective development programs. One of the take-aways for ACR GCD, discussed with eKitabu during the case study interviews, is that there needs to be additional focus on pathways towards scaling-up, especially through monitoring and evaluation mechanisms including rigorous impact evaluations and cost effectiveness analysis. The USAID DIV program focusing on rigorous evidence of impact and cost-effectiveness, particularly in relation to scaling, offers interesting learning. Another suggestion was to connect ACR GCD grantees so they can learn from each other and strengthen and build collaborations across countries. Close communication with the USAID Mission and ACR GCD funding contributed to securing collaboration with the in-country government. eKitabu’s journey with ACR GCD started with Missions introducing them to the Grand Challenge. eKitabu stated that they have greatly benefited from the relationships with the USAID education officers in the countries where they work. USAID was able to provide credibility with local governments as projects were in alignment with the Mission objectives. Working on a formal ACR GCD project has contributed immensely to the organizations’ credibility and has set relationships off on the right footing. “We appreciate ACR GCD as it helped us to build relationships with people, get advice, connect on values, connect on impact, connect on aspirations, and even on uncertainties or hard problems to resolve.” Case Study Interview eKitabu managed to secure good relationships with the Kenyan government by keeping impact and mutual goals at the center. One of the founders commented that “ACR GCD seemed to be conceived on the idea that public private collaboration is necessary for sustainability and scale. And that resonated well with what we see in education.” Being based in Kenya also helped. In the opinion of the founders and directors, effective engagement with government takes patience, persistence, and a good understanding of the local context. They added that it is important to have respect for government officials and to recognize the work they have been doing for the betterment of its citizens and concluded that if the Government is approached with evidence on impact, there will always be a room for partnership. A14.4.4 KEY ACHIEVEMENTS AND SUSTAINABILITY PROSPECTS eKitabu is geared towards building literacy in contexts with limited access to infrastructures such as power and connectivity. It holds long-term potential for improving literacy across education systems in lower income countries, and especially for children 200 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION with special needs.247 A Digital Literacy Trust report (2018) reported high enthusiasm for reading among children using eKitabu.248,249,250 The major achievements mentioned in the evaluation document of Studio KSL are:251 • Launched Studio RSL through piloting in Rwanda: Prototyped Studio RSL with the national WFD member, Rwanda National Union of the Deaf (RNUD) in May 2019, resulting in two complete RSL storybooks and clear next steps to produce 20 more RSL storybooks and to support RNUD to create a digital version of their RSL dictionary. • Produced and distributed a steady stream of Studio KSL storybooks: Completed 50 Studio KSL storybooks, packaged in EPUB. In the process, documented 400 KSL signs to build glossaries of key vocabulary for each storybook. • Promoted Studio KSL storybooks as a resource for parents of Deaf children and the Deaf community in Kenya: Engaged with 151 parents of children who are deaf during parents’ visiting days at four primary schools for the deaf. • Increased employment opportunities for Deaf Kenyans through Studio KSL: Employ 15 young adults who are deaf in Studio KSL. They inspire us to plot new, creative projects and they deepen commitment to eKitabu’s progress in becoming a more inclusive workplace. • Built a foundation for research on the impact of local sign language 247 https://docs.edtechhub.org/lib/WGYWI78R/download/VSHQLALL/Chuang_Koomar_2020_EdTech percent20Companies percent20Focused percent20on percent20Classrooms percent20and percent20Teachers percent20in percent20sub-Saharan percent20Africa.pdf. 248 https://docs.edtechhub.org/lib/WGYWI78R/download/VSHQLALL/Chuang_Koomar_2020_EdTech percent20Companies percent20Focused percent20on percent20Classrooms percent20and percent20Teachers percent20in percent20sub-Saharan percent20Africa.pdf. 249 https://www.ekitabu.com/wp-content/uploads/2019/08/Technology-for-Inclusive-Education-Pilot-Report-31-AUG-2018.pdf. 250 The study included insights from surveys and interviews on content and device utilization during an eKitabu initiative. After the completion of the pilot, each pupil spent 2 hours per week reading (compared to 0 hours at the beginning of the pilot as they previously did not have access to any storybooks). 251 ekitabu Final Report: https://drive.google.com/file/d/1tpRgtCELfAEjZVqsRxPagXk7vyvjb9U3/view?usp=sharing. 252 https://www.edu-links.org/learning/pivoting-pandemic-ekitabu-provides-sign-language-video-content-students. video storybooks: piloted a sign language communication assessment tool developed by partner Royal Dutch Kentalis, adapting the tool for KSL, including filming the video stimuli with plans to harmonize assessment tools across deaf education projects in East Africa, and set the stage for larger-scale research. During the COVID-19 pandemic, eKitabu supported continuing learning for millions of children in Kenya. After the Government closed all schools in response to the COVID-19 pandemic, the Kenya Institute of Curriculum Development (KICD) requested that eKitabu support deaf students by developing sign language videos to support early grade literacy for deaf children. In a further response to COVID-19, eKitabu rapidly built on these materials and produced Digital Story Time, a daily 30-minute broadcast for children and families. The program reaches four million households via eKitabu’s Youtube channel, eKitabu’s website, and EDU Channel TV (a channel operated under KICD as part of Kenya’s Ministry of Education).252 eKitabu has shown good signs for sustainability. The founders and implementers of eKitabu believe that the initiative will continue to sustain and grow after ACR GCD funding finishes, even if the end of funding might slow expansion of the initiative. eKitabu sees the key to sustainability as sizable private sector revenue (through sales and purchases), ability to attract funding from other donors such as UNICEF, and on-going collaboration with governments in countries of operation. A14.5  WORLD MOSQUITO PROGRAM: A SCALED, SCIENTIFIC INNOVATION A14.5.1 CASE STUDY SUMMARY Grantee Overview: The World Mosquito Program (WMP), a Combating Zika and Future Threats Grand Challenge (Zika GC) innovator, uses Wolbachia-infected mosquitoes to block disease transmission from mosquitoes that carry dengue, Zika, yellow fever, and chikungunya. This approach can be self-sustainable for up to eight years, eliminating local transmission. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 201 Wolbachia is a type of bacteria that blocks virus transmission from infected insects to humans. It occurs naturally in many insects but not in Aedes aegypti, the mosquitoes that carry dengue, Zika, and chikungunya. The World Mosquito Program (WMP) has developed a way to breed Aedes aegypti mosquitoes that carry Wolbachia. Once Wolbachia-carrying mosquitos are released into a community, they breed with wild mosquitoes and pass the bacteria onto their offspring. Over time, the majority of mosquitoes carry Wolbachia, greatly reducing the incidences of mosquito-borne diseases being transmitted to humans.253 Project Achievement and Sustainability: In 2016, when Zika became a public health threat in Brazil and other countries in Central and South America, USAID launched the Combating Zika and Future Threats Grand Challenge. It was through this open innovation effort that the WMP was selected to begin its large-scale pilot deployment of Wolbachia- carrying mosquitoes in Colombia.254 The Zika grant led to the scaling up of a small-scale pilot project in Colombia, covering about 40,000 people in about 3-4 km2, to a large-scale project covering two major cities and a population of 2.5 million people. Numbers increased from rearing and releasing 100,000-200,000 mosquitoes a week to nearly 3 million. In addition to its groundbreaking innovation in biotechnology, WMP has continuously innovated in its efforts to engage with the community and reduce associated costs. The program focuses on capacity building and training in the local community and has achieved high community acceptance and support across its pilot areas. WMP has scaled-up in partnership with government through loan-based buy-in mechanisms in countries like Brazil. It has other important partners that help sustain its global efforts. Key Learning: • The WMP grant from Zika provides a good example of how, in addition to exploring new and innovative ideas, the GC can fund already successful innovation to scale, reach a much larger population, and enable them to sustain for longer. • Keeping impact and evidence at the center has helped the program to learn, adapt and scale. • The internal monitoring, evaluation and learning (MEL) framework that supported USAID reporting helped to streamline results reporting and aided reflection on the learnings at the institutional level. • Capacity building and training of the local community is essential for knowledge transfer and also to build wider acceptance of the technology. • As an institution of academic origin, WMP has faced challenges in bringing the issue of mosquito-borne diseases as a critical issue that needs to be addressed by government. It has addressed this through research and evidence and intends to develop advocacy partnerships. 253 https://www.worldmosquitoprogram.org. 254 https://www.usaid.gov/sites/default/files/documents/CII_Global_Health_Innovation_Index_.pdf. 255 https://www.usaid.gov/sites/default/files/documents/1864/Combating-Zika-and-Future-Threats-Nominee-Summaries-4-13-2017.pdf. A14.5.2  USAID ZIKA GRAND CHALLENGE GRANT TOWMP To stop the spread of Zika and prevent other infectious disease outbreaks, USAID launched Combating Zika and Future Threats: A Grand Challenge for Development (Zika). The $30 million Challenge called upon the global innovator community to generate cutting-edge approaches to fight the Zika outbreak and to help strengthen the world’s ability to prevent, detect, and respond to future infectious disease outbreaks.255 Project Name: Deployment of Wolbachia-infected Mosquitoes to Block Disease Transmission. Problem and Solution: Serious diseases like dengue fever, Zika virus and chikungunya is caused by transmission from mosquitoes to humans which is currently tackled, if at all, largely through insecticides. The approach, which represents a paradigm shift in arboviral disease control, provides a natural, sustainable, cost-effective new tool for preventing transmission of a range of arboviruses. Funding was received from the Zika Grand Challenge to test the approach, which had been proven to work over long-term field tests, in much larger populations in several Latin American communities. Compared with conventional insecticide-based or genetic population suppression control methods that may provide limited, short-term reductions in the mosquito population, once Wolbachia bacteria has established in the local mosquito population, it persists without 202 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION the need for continual reapplication or additional insecticide-based control methods while reducing the risk of infection with dengue, chikungunya and Zika viruses. In addition, residents are not required to change their behavior or participate in ongoing activities after the mosquito releases are concluded. This research, which is the first of its kind in the world, could potentially benefit an estimated 2.5 billion people currently living in arboviral disease transmission areas worldwide.256 • Location Implemented: Colombia. • Collaborators / Partners: Monash University (Melbourne, Australia). A14.5.3 KEY LEARNINGS The WMP grant from Zika provides a good example of how, in addition to exploring new and innovative ideas, the GC can fund already successful innovation to scale, reach a much larger population, and enable them to sustain for longer. The WMP began under the funding and leadership of Gates Foundation and Wellcome Trust, with projects in Vietnam, Indonesia, Colombia, Brazil, and Australia. The Gates Foundation negotiated on behalf of WMP to scale up the program through USAID and paved the way for WMP to connect with the USAID Zika Grand Challenge program. This led to the scaling up of a small-scale pilot project in Colombia, covering about 40,000 people in about 3-4 km2, to a large-scale project covering two major cities and a population of 2.5 million people. “Without the USAID money, we probably wouldn’t have scaled and what the USAID money allowed us to do was to learn and scale in a small amount of time and to do scale-up on community engagement.” Case Study Interviewee USAID funding enabled WMP to expand its scaling and learning experiences and supported the project to greater achievements in relation to release of mosquitoes and scale-up in community engagement. Numbers increased from rearing and releasing 100,000-200,000 mosquitoes a week to nearly 3 million. There is no clear pathway on how Zika and USAID will continue with the program after the end of the scaling agreement. WMP may struggle to scale up the much-needed innovation to other countries. 256 https://www.usaid.gov/sites/default/files/documents/1864/Combating-Zika-and-Future-Threats-Nominee-Summaries-4-13-2017.pdf. 257 https://www.worldmosquitoprogram.org. 258 https://www.usaid.gov/sites/default/files/documents/CII_Global_Health_Innovation_Index_.pdf. Keeping impact and evidence at the center has helped the program to learn, adapt, and scale. Entomological studies conducted to date indicate that breeding has been successful and there is now a high frequency of Wolbachia￾carrying mosquitoes in the areas where they were released. At the end of 2019, WMP completed its deployment of Wolbachia-carrying mosquitoes in Medellin, Colombia. Epidemiological data is still being generated, but in many of the pilot areas, there is very promising evidence of lower incidences of mosquito-borne diseases, especially dengue. Ongoing trials will provide critical data to measure more clearly the impact of WMP’s initial programs.257, 258 According to the project implementers, based on data as of March 2020, the reductions in dengue fever incidence were 56 percent. The project team also felt that the impact of funding from this program, compared to the other programs, has been incomparable and there has been a very good return on the investment. The internal monitoring, evaluation, and learning (MEL) framework that supported USAID reporting helped to streamline results reporting and aided reflection on the learnings at the institutional level. WMP also has its own internal MEL framework and captures data in an online platform which provides readily accessible reports and historical data. The project team said that this helped them to amalgamate data and produce performance and milestone-based reports to USAID effectively and on time. USAID required the program to submit additional annual results reports which are separate from the MEL reports. The program team felt that USAID could better organize and streamline their reporting requirements to avoid duplication and make the process much more straightforward and efficient for grantees. The program has also benefited from research and evaluation that goes beyond the mandated program reporting and evaluations. WMP has shown flexibility to allow external researchers to conduct research on different elements of the program including cost effectiveness. WMP has so far published about 100 peer-reviewed publications about the program over the last 16-17 years and there have been a few studies conducted by independent groups to validate and certify the results. The program has also been able to invent innovative ideas for USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 203 releasing the Wolbachia mosquitoes into the environment via drones, using the USAID funding. USAID commissioned a risk assessment, focused on environmental issues to ensure that biologically controlled efforts to release Wolbachia Mosquitoes do not create any negative impact. Studies proved that there was a negligible risk. In addition to its groundbreaking innovation in biotechnology, WMP has continuously explored ways to implement the scheme effectively in the ground and has innovated in its efforts to engage with the community and reduce associated costs. WMP has evolved its strategies to help inform and gain support from people in communities where Wolbachia mosquitoes are introduced and have achieved high levels of acceptance. WMP has also developed techniques to reduce the cost of releasing mosquitoes towards its target of $1 per person protected in large, urban settings. By evolving its approach, WMP continues to overcome challenges and establish a pathway to scale and major global impact.259,260 For example, during its initial phase, ethics approval for programs under WMP at the household level was secured using a door-to-door process which was extremely time￾consuming. With large scaling-up, a more innovative and efficient mechanism was needed. This was achieved by developing a public acceptance model, which has now become a cornerstone of the whole program, and a crucial tool for defining and tracking community engagement in the project. Capacity building and training of the local community is essential for knowledge transfer and also to build wider acceptance of the technology. WMP trains the local community to release and catch mosquitoes. It also trains local vector control groups in the community to continue monitoring and has developed an online learning platform which is continually being revised and updated to ensure the continuing success of the program. The WMP noted in response to capacity building efforts that “our dream is that is that, you know, in each of the sites, the local vector control groups will be equipped to release mosquitoes and catch mosquitoes without the help of WMP staff and that the online learning platform will assist with the methodologies.” 259 https://www.usaid.gov/sites/default/files/documents/CII_Global_Health_Innovation_Index_.pdf. 260 https://www.worldmosquitoprogram.org. 261 https://www.usaid.gov/sites/default/files/documents/CII_Global_Health_Innovation_Index_.pdf. 262 https://www.worldmosquitoprogram.org. WMP has subsequently been able to scale-up in partnership with the Government through loan-based buy-in mechanisms in countries like Brazil. The Brazilian project functions with a loan￾based buy-in mechanism through funding received from the Gates Foundation and the Wellcome Trust (not USAID). The project has won a number of buy-ins and the government is funding projects in various parts of Brazil. Municipalities take out the loans under guarantee by the Ministry of Health. This provides WMP with a model for scaling and sustainability beyond GC funding, which it is pursuing. WMP also commented that the partnership with USAID has assisted them in developing newer relationships and expanding networks, although it could have benefited more from links to USAID Missions. “We’re looking at a multi-pronged new business development approach, some of it would be engaging directly with governments and having them use loans or other financial mechanisms in order to fund our projects” Case Study Interviewee A14.5.4 KEY ACHIEVEMENTS AND SUSTAINABILITY PROSPECTS Since the establishment of its first project site in northern Australia in 2011, WMP has undergone significant expansion from a small research project to an international, not for profit initiative operating in 12 countries. WMP has three regional hubs – an Asia hub in Ho Chi Minh City, Vietnam, and Oceania hub, based at Monash University in Melbourne, Australia, and a Latin America hub in Panama City (scheduled to be opened in early 2022). Through collaboration and innovation, their global approach can help to protect many thousands of communities from the threat of mosquito-borne disease. In addition to USAID, WMP has other important partners that help sustain its global efforts, including Monash University, the Bill & Melinda Gates Foundation, the Wellcome Trust, and other Government international development donors, including Australia and New Zealand.261,262 Since USAID’s initial investment, WMP has leveraged approximately $50 million in external support and was selected as a finalist in the 2020 MacArthur 204 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Foundation 100&Change competition with an opportunity to win $100 million to further scale.263,264 The impacts highlighted in the Global Health Innovation Index are:265 • The World Health Organization recommended in 2016 that the Wolbachia method be piloted and monitored. • Epidemiological studies have shown dengue incidence is lower in treated communities. • The program has achieved high rates of Wolbachia carrying mosquitos in pilot areas. • Epidemiological impact studies are ongoing in several locations. The approach of the program design is such that after the initial investment, WMP’s method is self-sustaining. WMP has been deploying since 2011; the first deployments were in Australia and it currently works in 12 countries and has 3 regional hubs. Within target countries, WMP is expanding to additional cities with government support. The program has achieved high community acceptance and support across its pilot areas. Currently, the Gates Foundation funding has finished, the USAID funding finishing this year and the Wellcome Trust is just funding a small epidemiological trial, which will finish next year. Now that the technology has been proven, WMP is now looking at a multi-pronged new business development approach, by engaging directly with 263 https://www.usaid.gov/sites/default/files/documents/USAID_2020_CII_ImpactBrief_V8_SPREAD_508_012621.pdf. 264 https://www.worldmosquitoprogram.org/en/news-stories/media-releases/world-mosquito-program-shortlisted-usd100m-grant-macarthur. 265 https://www.usaid.gov/sites/default/files/documents/CII_Global_Health_Innovation_Index_.pdf. governments and having them use loans or other financial mechanisms in order to fund the projects. “Scaling up to the national level brings different challenges for the project in terms of strategy about partnership and in advocacy.” Case Study Interviewee Despite having a scientifically backed technology, WMP has not been able to sell the idea to the extent they would have liked to. An institution of academic origin, WMP has faced challenges bringing the issue of mosquito-borne diseases like Dengue or Chikungunya as a critical issue that needs to be addressed by the nation. In many underdeveloped countries, the government considers other issues to be more critical in nature and tend to neglect the public health concerns related to mosquitos. With this regard, WMP has also been working to increase the global evidence highlighting the criticalities and negative impacts of mosquito-borne diseases. In countries like Mexico and Brazil, there are 10,000s cases of Dengue every year and there is a cost-effective benefit to stop spraying insecticides and move towards introducing Wolbachia mosquitoes. This involves a lot of research and advocacy programs. One of their strategies for the future is to work with Civil Society Organizations and NGOs who are good at advocacy and community mobilization. During the initial phase, they mostly collaborated with universities. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 205 ANNEX 15 SURVEY FINDINGS Please click here to access the survey findings. 206 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Photo: Global Devlab/USAID USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 207 ANNEX 16 PROCUREMENT ANALYSIS GC ROUND / CALL APPLICATION DEADLINE GRANTEES ANNOUNCED MONTHS ELAPSED All Children Reading Grant Competition (2011) 01/31/12 09/2012 7 Grant Competition (2014) 04/10/14 02/03/15 9 Enabling Writers Prize 10/01/14 08/12/15 10 Mobiles for Reading (2014-15) 11/13/14 - - Technology to Support Education in Crisis and Conflict Settings Prize 03/30/15 - - Tracking and Tracing Books Prize 04/01/15 - - Mobiles for Reading (2015-16) 03/09/16 - - EduApp4Syria Prize 04/01/16 03/20/17 11 No Lost Generation Summit Tech Prize 02/2017 - - Book Boost: Access for All Challenge 01/19/18 04/04/18 2 Sign On for Literacy Prize 02/16/18 06/27/18 4 Begin with Books 11/15/19 03/05/20 3 UnrestrICTed 07/20/20 - - Combating Zika and Future Threats Round 1 04/2016 08/10/16 4 Round 2 04/2016 10/12/16 6 Creating Hope in Conflict Round 1 04/12/18 09/25/18 5 Round 2 08/23/19 08/18/20 11 Round 3 11/16/20 TBD - Fighting Ebola Round 1 11/07/14 - - Round 2 12/01/14 - - Making All Voices Count Round 1 11/01/13 04/03/14 5 Round 2 10/15/14 04/17/15 6 Round 3 10/07/15 02/12/16 4 Powering Agriculture Round 1 12/2012 12/2013 12 Round 2 11/2014 11/2015 12 Saving Lives at Birth Round 1 05/09/11 07/13/12 14 Round 2 04/02/12 07/11/12 3 Round 3 03/28/13 - - Round 4 01/08/14 - - Round 5 02/09/15 - - Round 6 02/29/16 - - Round 7 02/24/17 - - Round 8 02/28/18 - - Scaling Off-Grid Energy Round 1 - 11/14/16 Round 2 - 10/25/17 Securing Water for Food Round 1 - - - Round 2 - - - Round 3 - - - Round 4 10/10/16 - - 208 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION Photo: Global Devlab/USAID USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 209 ANNEX 17 TERMS OF REFERENCE USAID GRAND CHALLENGES FOR DEVELOPMENT META￾EVALUATION IPE TRIPLE LINE SCOPE OF WORK (SOW) SEPTEMBER 2020 Scope of Work Contents Background 180 Evaluation Purpose 180 Evaluation Scope & Areas of Interest 181 Primary Areas of Interest 181 Secondary Areas of Interest 183 Data Sources 183 Evaluation Technical Approach 184 Process for conducting the evaluation 184 Methodological and tactical approach 184 Analytical approach 185 Cost effectiveness analysis 187 Uptake strategies 188 Evaluation Deliverables 189 Technical and Contract Oversight 190 Evaluation Team 190 Evaluation Timeline 194 BACKGROUND USAID is committed to using Open Innovation (OI) competitions such as Grand Challenges (GCs), Challenges, and Prizes to mobilize new solutions for critical development objectives. Within USAID, the open innovation journey can be traced to the 2008 Development 2.0 Challenge, which called for innovative mobile solutions to help achieve international development goals. Since 2011, USAID and its partners have launched ten Grand Challenges, funding more than 525 innovators to test, develop, and scale solutions to respond to health and development challenges such as water and agriculture, early grade literacy, off-grid energy access, complex humanitarian crises, and more. GCs have leveraged attention and support for traditionally under-resourced (i.e., have not received adequate funding or attention, or have not found effective solutions) challenges and are increasingly recognized by USAID as a way to diversify both solutions and the partnership base. EVALUATION PURPOSE As USAID looks to advance implementation and expand the use of Open Innovation Competition methods, USAID’s Center for Development Innovation (CDI) recognizes the opportunity to systematically reflect on more than eight years of GCs to generate an actionable evidence base. The evaluation will primarily be conducted to: • Assess the historical impact of and the sustainability of results achieved through the Grand Challenge approach, understand the drivers and inhibitors of effectiveness, and identify practical strategies that optimize future impact (i.e., Did I get what I asked for and did this help me address my problem?). Please note: Desired results of GCs generally include the explicit objectives of supporting innovations as well as objectives to catalyze external funding for and awareness of particular development issues. • Develop practical strategies, systems, and frameworks to measure the impact, results, and uptake of GCs across diverse contexts, stages of innovation, and multi-partner programs. • Assess the feasibility of measuring the cost effectiveness of GCs and of comparing the cost effectiveness of GCs to program alternatives. Pending results of the feasibility assessment, develop practical and technically-sound framework/guidance to measure the cost effectiveness of GCs going forward. Please note - the evaluation will not require retrospective cost effectiveness analysis or GCs. • Assess strategies used to scale innovations by GCs and their effectiveness & better understand the inhibitors and enablers of scaling innovations. • Develop a practical method of measuring the effectiveness of investments intended to strengthen innovation ecosystems. Using this framework, evaluate the effectiveness of previous GC ecosystems investments as feasible. • Assess attempts to engage USAID Missions and OUs in GCs and identify promising practices 210 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION to support the adoption of GC approaches across the Agency. • Document and understand the advantages and disadvantages of the various GC partnership, governance, and operational models and inform GC strategies that maximize future management effectiveness. • This forward-looking evaluation will generate actionable strategy recommendations in a range of formats that are useful to the evaluation audiences and include practical, adaptable measurement frameworks and recommendations that can be applied by USAID and its partners to guide investment decisions, and advance the design, management, and measurement of GCs. EVALUATION AUDIENCES The evaluation will primarily inform decision makers who fund and manage GCs, and deliverables should be developed to inform these audiences. Stakeholders include: • USAID OUs involved in managing GCs. • USAID’s Center for Development Innovation (at the U.S. Global Development Lab). • USAID Leadership, USAID Missions, and Bureaus. • Donors & partners that currently co-fund/manage or have co-funded/managed GCs. • Other policymakers and decision-makers considering open innovation challenges also can apply evaluation findings and recommendations, for example, those in developing countries governments. EVALUATION SCOPE & AREAS OF INTEREST This evaluation will consider GCs that USAID has engaged as a partner, including: • Saving Lives at Birth. • All Children Reading (ACR GCD). • Powering Agriculture. • Making All Voices Count. • Securing Water for Food (SWFF). • Fighting Ebola. • Combating Zika and Future Threats. • Scaling Off-Grid Energy (SOGE). • Ensuring Effective Health Supply Chains. • Creating Hope in Conflict. • The evaluation will be grounded in a utilization￾focused approach that generates data that stakeholders need and will use for decision making. The evaluation areas of interest proposed below reflect input generated through two participatory workshops with USAID GC stakeholders. Sub-questions for each area of interest reflect points of interest voiced by USAID GC stakeholders. Final evaluation questions will be refined, narrowed, and shaped with input from USAID and GC partners through one-to-two participatory workshops (number contingent upon participant availability) to be led by the selected evaluator. PRIMARY AREAS OF INTEREST IPE will review all available data from the GCs and recommend which of the following primary areas of interest are feasible given availability of data. The exception is Area 3 (Cost effectiveness), which will be tackled as described below in pages 8-9 of this SOW. AREA 1: Retrospective Assessment of GC Results A. What results have been achieved and sustained across GCs and what factors support, inhibit, and explain the success and sustainability of GCs and their innovations? – What types of outcomes have been achieved by GCs in support of which development objectives? Which objectives, sectors, and contexts are GCs best and worst positioned to achieve success in, particularly compared to traditional development models? – To what extent have GCs been used to address systemic issues? How can GCs be used to address systemic challenges going forward? – To what degree has GC grant making matched GC investment criteria, and whether appropriate criteria were selected to reflect intended outcomes? – How can GC partners better tailor expectations and align investments and resources to meet realistic objectives? – What unintended consequences have resulted from GCs, good or bad, and what can we learn from them? – Have GCs applied appropriate combinations of resources and approaches to support specific GC objectives? Which resources/approaches are both necessary and sufficient to reach which aims, and which resources/approaches are not? B. What is the right balance of early and later-stage innovations to reach which types of objectives? USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 211 AREA 2: Developing Frameworks and Strategies to Measure the Performance and Impact of GCs How should GC stakeholders define and measure results, impact, and program effectiveness across GCs? What frameworks, methods, and learning strategies can best support measurement going forward? a. How should success be measured across stages of innovation, particularly early-stage innovations and R&D investments? b. How can GCs measure long-term outcomes and incentivize post-award reporting? c. Practically and operationally, how can GC partners better structure and coordinate data collection and measurement efforts? What systems and processes will best support such efforts? d. How should USAID coordinate the collection of data across USAID-supported GCs going forward? AREA 3: Measuring Cost effectiveness How feasible is it to measure the cost effectiveness of previous and future GCs and to compare the cost effectiveness of GCs to traditional program models? • (Contingent on feasibility assessment) Methodologically and technically, how can we measure the cost effectiveness of GCs going forward? What practical frameworks/approaches can be developed to measure GCs? Please Note: Pending the results of the feasibility assessment, the evaluator will be expected to develop frameworks/ practical methodologies. AREA 4: Scaling Innovations Which GC innovations have reached scale (either for sustainability and/or for impact), and what strategies and factors contributed to or inhibited scaling? What are the characteristics of the innovations that reach scale? • How can IPE categorize different pathways to scale and appropriately integrate pathways to scale into programing? AREA 5: Investing in Ecosystems How effectively have GCs invested in ecosystems strengthening, and what have been the results of these investments? • How can GCs more effectively invest in longer￾term ecosystems strengthening going forward? • How can the effectiveness of ecosystem investments be measured? • What types of partners/innovators are best suited to support which ecosystems components and stakeholders? • What objectives and components, such as sourcing, testing, and scaling, are ecosystems (and related value-chain, cluster) systems most relevant to support? AREA 6: Engaging USAID Missions and OUs To what extent have GCs effectively engaged USAID Missions and OUs, and what lessons can be learned from engagement efforts to date? • How can GCs be integrated into USAID and Mission programing going forward? AREA 7: Governance, Partnership Models, and Operational Effectiveness How have GCs been managed/governed differently and what models can be described? What are the advantages and disadvantages of different models? • How do different approaches and structures influence results? • What can we learn about partnership models and GCs, specifically relating to value/shared value; governance structures/models; decision-making processes? • How have different types of GC partners used their comparative strengths to maximize operational effectiveness and development impact of GCs, and how can partnerships be formed and managed to maximize comparative advantages going forward? SECONDARY AREAS OF INTEREST In collaboration with Catalyst and USAID, IPE will review all available data to prepare recommendations on whether analysis is possible and the value of prioritizing the following areas in the. AREA 8: Acceleration Which GC acceleration strategies are most effective and at what stages of innovation? What other acceleration strategies might GCs pursue that could best support innovators? • What are the advantages and disadvantages of providing in-house versus external acceleration support? 212 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION AREA 9: Procurement and Reducing Barriers to Funding for Non-Traditional USAID Partners How can USAID and GC partners further reduce barriers to procurement efficiency and to the participation of diverse innovation? • To what extent are GCs actually more efficient than other USAID funding mechanisms? • Are certain types of innovators better positioned to succeed in GCs than others? Why? • What are the advantages and disadvantages of in￾house versus external grants management? AREA10 Catalytic Effects: To what extent have GCs catalyzed funding, development of other solutions, follow-on funding, and awareness? • Do GCs generate more funding than USAID can working alone? DATA SOURCES In addition to primary data collection, the evaluation will collate and leverage existing data to the degree possible; potential secondary data sources include historical program documents, previous evaluations, and project and monitoring data. The evaluation will not seek to duplicate prior evaluative efforts, and will leverage existing learning products such as: • Powering Agriculture: Midterm Evaluation and internal evaluation. • Sida: Evaluation of Sida’s Global Challenge Funds. • ACR GCD – Assessment. • SWFF – Midterm and Final Evaluations. • SOGE - Developmental Evaluation. In order to retrospectively evaluate the results and effectiveness of GCs, the evaluator will be expected to develop a framework to compare contextually and structurally diverse GCs across multiple areas of interest. EVALUATION TECHNICAL APPROACH PROCESS FOR CONDUCTING THE EVALUATION The process for conducting the evaluation was structured around key stages: a. Inception phase: Establish utilization-focused priorities for the evaluation; workshop to review intervention model with USAID as a basis for finalization of evaluation framework and questions; finalize methodology, engagement plan, data collection approach and timeline. Preparation of Scope of Work (SoW). b. Stakeholder engagement and data collection: Testing of KII and survey instruments prior to commencing data capture; preparation of data capture tools for recording data in a format conducive to efficient analysis against key evaluation questions; KIIs with key stakeholders identified during inception (including USAID OUs involved in GCs, management agents, partner donors). Up to three field visits are proposed which will enable engagement of USAID Missions, ecosystem stakeholders, field-based management agents and GC-supported innovators. c. Interpretation and synthesis: As data is captured, the evaluation team will continuously look for interesting issues arising that merit further investigation; methods used in analysis will include portfolio analysis, theory of change analysis and comparator program analysis. d. Validation and reporting: Opportunities for validation will be both formal and informal. Formal validation processes will include the preparation of monthly interim insight memos which will present emerging findings for comment and review by relevant stakeholders; and formal opportunities are also included for feedback on key reporting deliverables. This stage also covers the preparation and facilitation of a stakeholder learning workshop to encourage uptake of key recommendations. METHODOLOGICAL AND TACTICAL APPROACH The evaluation will be grounded in a utilization￾focused approach, consistent with IPE Triple Line evaluation methods and principles that ensure their evaluation methodology is effectively targeted at specific client needs. Essential to ensuring the use of the evaluation results is to define the term ‘use’ by identifying who will use what data to make which decisions. Discussions in the inception phase with key users (relevant USAID OUs, GC partners and stakeholders) will contribute to a methodology that supports a focused evaluation process and report. IPE TripleLine will ascertain what types of evidence are considered credible, and who needs to be interviewed and consulted in the design stage. USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 213 This will create consensus and buy-in around the evaluation process and will enhance ownership and use of the report’s findings. The evaluation will take a mixed methods approach and data collection will be mainly through document review, key informant interviews, sample surveys, focus group discussions and workshops during field visits to a sample of GC implementation countries. In order to retrospectively evaluate the results and effectiveness of GCs, the evaluator will be expected to develop a framework to compare contextually and structurally diverse GCs across multiple areas of interest. Primary data collection is to involve testing of key informant interview (KII) and survey instruments prior to commencing data capture; preparation of data capture tools for recording data in a format conducive to efficient analysis against key evaluation questions; KIIs with key stakeholders identified during inception (including USAID OUs involved in GCs, management agents, partner donors). Up to three field visits will be undertaken which will provide the opportunity to triangulate information from document review and interviews as IPE TripleLine provides opportunities to engage directly with U.S. Mission personnel, ecosystem stakeholders, GC in-country management and support teams and a selection of grantees. The selection of countries will be discussed with USAID and will be made based on criteria such as concentration of GC presence, accessibility of location for convening workshops of grant holders, representation of GCs across the countries. Analytical approach IPE will use an approach to analysis which comprises three main components: i) an intervention model analysis which interrogates the design, implementation and emerging impact of each of the 10 GCs in the portfolio and ii) a portfolio analysis to identify differences and similarities between GCs within the portfolio. These two components of analysis will also inform iii) the assessment of the feasibility of cost effectiveness analysis, and the preparation of a practical framework for measuring impact and cost effectiveness. The overall approach to this evaluation is informed by IPE Triple Line’s detailed understanding of the fund management processes and the factors that enable or constrain successful achievement of program outcomes and impact provide the basis for a robust framework for analysis of each stage in the fund management cycle (design and launch of calls, pipeline development, selection, contracting, financial and non-financial technical support and monitoring, evaluation and learning). IPE TripleLine also understands the challenges of identifying innovative solutions to difficult development problems and supporting the progression of promising innovations along pathways to impact at scale. It is understood that USAID-supported GCs may include ecosystem support components designed to catalyze support and funding for scaling such as promoting adoption of proven social benefit innovations by government agencies or linking private sector innovators to potential market system investors. The GCs may also engage USAID Missions and OUs to support ecosystem development, ensure coherence with other programs. Figure 2 presents a generic GC intervention model that IPE TripleLine will adapt and refine in consultation with USAID and GC partners during the preparation of the statement of work. IPE TripleLine will use this generic model as a framework to ensure consistency in appraisals of the contextually and structurally diverse GCs. The preparation of the feasibility assessment and prioritization memo will include a mapping of the evaluation questions against this intervention model and will be split into three focus areas: a. Program design: This will focus on the contextual factors informing program design; the rationale for selection of the GC approach itself; the definition of program objectives and any targets set; the theory of change or equivalent, to demonstrate how the proposed project inputs and activities IPE Triple Line expected to lead to the achievement of objectives; the proposed management and governance arrangements (including for the provision of fund management, technical assistance and acceleration services); and plans for ecosystem development and the engagement of USAID Missions and other OUs. b. Implementation: This will focus on the actual implementation processes at all stages in the fund management cycle from the design of calls through to monitoring, evaluation and learning, as outlined above and in Figure 1. c. GC achievements: This will focus on the results, outcomes and emerging impacts of the GCs in the portfolio, using existing evidence from evaluations, where available. 214 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION ANNEX FIGURE 7: GENERIC GC INTERVENTION MODEL PROGRAM DESIGN IMPLEMENTATION ACHIEVEMENTS Shared PROGRAM OBJECTIVES (among partners) ‘Ecosystem’ development support USAID/ partner Mission & other Operating Units Thematic Objectives Theory of Change Result framework Budget Governance arrangements Stages of innovation Scaling objectives Geographic focus Ecosystem support Linkage with other Program Engagement of related agencies FUND MANAGEMENT Design of calls Pipeline and launch Selection Contracting Funds and Tech support Comms & Outreach M&E and Learning SUSTAINABLE OUTCOMES AND IMPACT AT ‘SCALE’ • Raised awareness • Enabling environment PROGRAM APPRAISAL & DESIGN (including context analysis & problem identification) 266 Disability Adjusted Life Years www.who.int. 267 https://www.globalinnovation.fund/practical-impact-assessment/. The same framework will be used to guide data capture from document review, key informant interviews and focus group discussions. Initially, this data will be used to prepare descriptive profiles of the design, implementation and achievements of each GC. These descriptive profiles will be structured to reflect the components of the intervention model to enable comparative analysis within the portfolio. The descriptive profiles will be further developed and refined in the early stages of the evaluation and will be shared with USAID and the managers of the GCs for validation to ensure their factual accuracy. This will ensure that subsequent evaluation and analysis will be based on a consistent and factually accurate understanding of the structure and performance of the GCs within the portfolio. For each GC, the evaluation will investigate how design and implementation factors have supported or constrained performance and this will feed into general learning about which approaches work best and under which conditions. The use of the USAID GC model will also be compared to other approaches used to address similar challenges to those addressed by the GCs in the portfolio. The identification of comparator programs will be considered as part of the evaluation feasibility and prioritization process. Portfolio and comparative analysis A portfolio analysis process will enable comparisons of the basic characteristics of each GC to identify commonalities and differences. Characteristics included in the analysis will include: defined objectives, stages of innovation, geographies, budget, number of grants/investments, type of service provider for fund management, technical assistance and acceleration, projected lifetime and provision of ecosystem support. Basic data will be included in a preliminary analysis to inform the feasibility assessment and prioritization memo for validation. The portfolio analysis tool will be further developed and validated during the preparation of the SoW. Cost effectiveness analysis IPE TripleLine will complete a cost effectiveness assessment (CEA) of the GCs. CEAs takes a similar approach to cost benefit analysis CBA by using alternatives to income for impact measurement such as DALYs266 which is based on the number lives improved/saved; or measures such as the Practical Impact Assessment (PIA) used by the USAID supported Global Innovation Fund.267 This CEA will cover some simple and measurable aspects such as the unit cost of reaching/treating an end beneficiary and measuring the efficiency in the use of USAID funds in the management of the GC. To achieve this, the CEA will separate the costs and benefits of the GC Fund Manager (FM) from the cost and benefit stream of the portfolio of grants (the Fund). Cost effectiveness of Fund Manager The FM will have a range of costs associated with management components of the GC including: launching the challenge, selecting the grants, contracting, managing and supporting the grants, and measuring the results. Care needs to be taken in comparing GCs as some may transfer activities USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 215 to third parties (for example results measurement may be included in the GC grant or may in other cases be performed by the FM). The overall FM cost is usually described as the FM administrative cost and expressed as a percentage of total funds under management. Typically, a large proportion of the FM cost is in the early stages of the fund in launching and selecting the projects and so the proportion of administrative costs generally declines as the fund matures. FM costs also vary greatly depending on the nature and intensity of non-financial support provided to grantees. Cost effectiveness of the GC Fund The GCs under review cover a range of different sectors with different outcomes and impact, some of which will produce a measurable monetary benefit (e.g., increased farm yields from Powering Agriculture) whereas others generate other, less measurable types of end benefit including improved voice and accountability (MAVC), conflict resolution (Creating Hope in Conflict), improved health outcomes (Fighting Ebola). The feasibility assessment would undertake the following desk review for each of the GCs with appropriate interviews of the FM and use of benchmark data. • Map the current results/impact chain and define the end beneficiaries. This will be undertaken as part of the GC intervention model analysis described above, taking care to note the critical assumptions at each stage. Where possible, results frameworks (log frames), results chains or theories of change that have been developed by the FM or GC grantee will be used. • Review the data collected by the FM and/ or grantees for the results/impact chain for each of the GCs. Of particular interest will be how end beneficiaries are reached and measured and the assumptions made by each FM. The approach of the team would be to work with each of the GC Fund Managers in a collaborative way to understand their approach to results measurement and data collection. • Outline a cost effectiveness framework for each GC Fund. This would include, where possible, the definition of common units of impact measurement. One option would be to develop a DALY framework for the health and education projects, a CBA tool for the agriculture, water and energy projects and/or to use the PIA tool which could in theory be used for all GCs. This tool takes all measures of impact into a common unit. The practical impact unit is a function of the scale of benefit (how many people), the breadth of impact (life changing power) and the probability of success. This assumption￾based model defines impact achieved by Year 10 in a common unit equivalent to one person getting a one-time benefit equivalent to 100 percent of annual income. • Assess the extent to which net impact as currently measured or calculated is directly attributable to the GC and outline the extent to which further data collection is required to validate the estimates of impact. An overall evaluability assessment based on the data available would be undertaken as part of the feasibility assessment. • Define the recommended approach, inputs, timing, costing and the respective merits of undertaking detailed cost effectiveness study for each of the GCs. This would need to look at what might be considered a ‘good enough’ estimate of impact based on validated assumptions versus the time and cost of a detailed longitudinal impact study which may assess the counterfactual using a control group. The potential trade-offs between depth within a few GCs and breadth across the portfolio will be presented. UPTAKE STRATEGIES The findings and recommendations will be both conceptual, and highly practical, with a focus on enabling immediate utilization. The findings of the evaluation will be shared with key stakeholders. Stakeholders will be able to reflect on, debate, and adapt learnings through a series of presentations, workshops, discussion forums and other mechanisms. During the inception stage IPE TripleLine will explore with USAID and other key users the best formats that this series should take. The evaluation team should retain the option of presenting its own independent recommendations, the discussion process enables those involved in taking forward recommendations to participate in verifying the viability of the recommendations within their own specific operating environment. In order to ensure uptake of key lessons and recommendations of dissemination and knowledge sharing approaches, IPE TripleLine will plan and deliver a Learning Workshop for GC stakeholders. 216 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION EVALUATION DELIVERABLES All deliverables will be submitted to Catalyst for technical review prior to submission to USAID. Before being accepted, deliverables will be approved by USAID and the Catalyst technical review team (see next section). • Inception Briefing: A concise (~2 pages) briefing capturing key takeaways of a multi￾stakeholder evaluation kick-off workshop. • Evaluation Feasibility Assessment & Prioritization Recommendations: Analysis of the availability and usability of secondary data, the feasibility of collecting primary data, and the practical and trade-offs involved in prioritizing evaluation questions. This assessment will include recommendations for how USAID should prioritize evaluation areas of interest (Concise narrative document - up to 10 pages). • Evaluation Statement of Work (SOW): The Evaluation SOW will include the following components. – Background & Context. – Final Evaluation Questions. – Evaluation Design, Methodology, and Sampling Strategy (Including a list of secondary data source). – Description of Evaluation Deliverables. – Detailed Evaluation Work Plan. – Data Management Protocol (describing how data will be securely and ethically managed across the data management life cycle). – Staffing Plan. – Risk management strategy, describing identified risks to implementation and proposed approaches to mitigation and specific related support from Catalyst and USAID. • Evaluation Instruments: Any data collection tools (such as surveys, KII protocols, etc.) that will be used to collect data. • Cost Effectiveness Analysis (CEA) Feasibility Report: A technical assessment of the feasibility of evaluating the cost effectiveness of Grand Challenges and of comparing Grand Challenges to other development mechanisms. Narrative report up to 30 pages (excluding Annex). • Data Collection and Analysis: The evaluation team is expected to independently conduct data collection and analysis, and will be expected to securely and ethically collect, process, and store data. USAID will support the collation of secondary GC performance data and documents, and will introduce evaluators to connect with GC stakeholders as needed. Additionally, the evaluator must adhere to and adhere to USAID standards when collecting data, creating a final evaluation report, and publishing data sets on USAID’s Development Data Library (DDL). The evaluator will be required to comply with USAID policy regarding open data, gender-disaggregation of data, and other. • Targeted Interim Insights Memos: In order to support continuous stakeholder engagement, the evaluator will submit concise memos drawing from interim findings. Learning can be captured in visual presentations or written memos, depending on which formats are determined most engaging and useful. • Cost effectiveness and Impact Measurement Frameworks: Pending the CEA feasibility analysis, the evaluator shall develop practical frameworks to measure the cost effectiveness of GCs. The evaluator shall also develop adaptable measurement frameworks that can be deployed to measure the performance and impact of future GCs. • Final Evaluation Report, Presentation, and Targeted Knowledge Products: The evaluator shall submit a final narrative evaluation report that complies with the requirements of USAID’s Evaluation Policy. The evaluator will also create concise and engaging knowledge products for targeted audiences and will develop a PowerPoint/ visual presentation to capture key findings. • Plan and Deliver Learning Workshop for GC stakeholders in order to foster the utilization of evaluation recommendations. TECHNICAL AND CONTRACT OVERSIGHT As part of the project management, IPE TripleLine will undertake quality assurance on all evaluation outputs across key process, normative and technical criteria. TripleLine will participate in reviews twice per month via conference call during which they update the Catalyst and USAID team on progress and identify any risks. The IPE TripleLine contract will have the following technical and contractual oversight: USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 217 Courtney Roberts, Moonshot Global/Catalyst, lead for technical oversight. Courtney will review, comment and request necessary revisions on methodology and all technical deliverables. She will recommend approval by Catalyst and USAID. Stephen Rahaim, Resonance/Catalyst, contract lead and technical oversight contributor. Stephen will review, negotiate and approve any changes to scope, timeline or the contract. He will approve all deliverables in coordination with USAID and will approve IPE invoices for payment. He will review and comment on all technical deliverables. Lorin Kavanaugh-Ulku, USAID Open Innovation Competitions lead, primary point of contact. Lorin or her designee will review, comment and request revision to technical deliverables. She will approve all technical deliverables and act as the primary point of contact between IPE/Catalyst Scott Jackson, USAID, Scott will provide technical support and coordination to USAID’s oversight of the project. He may, if designated, approve, comment and request revision to technical deliverables. Catalyst may add additional technical oversight to the team in a review and coordination capacity. EVALUATION TEAM The evaluation team should offer both subject matter and evaluation expertise and will leverage both to develop practical and actionable strategic recommendations. Catalyst will approve any staffing changes, additional or alternative, team members. Approval is required prior to any additional or alternative team members billing the contract. The IPE TripleLine evaluation team will include: Martin Wright – Team Leader (Fund Management / Agriculture / Irrigation & Water Supply) Martin has more than 30 years of experience in international development, including more than 20 years in grant fund management and complex evaluations of funds supporting innovation. He was a Senior Evaluator on the meta-evaluation of 10 of Sida’s global challenge funds, a key team member on the evaluation of the Global Innovation Fund, and he has led or directed several complex evaluations covering open innovation and human centered design (Amplify), higher education (Strategic Partnerships for Higher Education Innovation and Reform, SPHEIR), and forest governance in Indonesia (Forestry, Land-use and Governance, FLAG). He currently has a leading role in the provision of monitoring and evaluation services on an earth observation technology development program funded through the UK Space Agency. Martin has a background in the management of major challenge funds, as Team Leader for large DFID-funded grant programs, including the Global Poverty Action Fund (GPAF) and the Civil Society Challenge Fund (CSCF), and brings technical expertise in agriculture, irrigation, and water supply. He previously worked as a Team Leader on a USAID-funded groundwater irrigation and water supply project in Timor, Indonesia, including a period of direct hire on a Personal Service Contract. His experience of managing grant programs also covers education, health, post-conflict and post-disaster recovery, natural resource management, and rights￾based approaches. Clarissa Poulson – Senior Evaluator (Governance / Education / Humanitarian) Clarissa is a senior evaluation expert and specialist in challenge and innovation funds. With Katharine May, she presented an award-winning paper and led a session focused on the evaluation of grand challenges at the UK Evaluation Society 2019 conference, and she also chaired a session at the American Evaluation Society conference in 2018. Recent relevant experience includes as senior evaluator on the evaluation of the DFID Amplify program focused on open innovation and the use of ideo.org’s human-centered development approach and leading the External Evaluation of the Humanitarian Innovation Fund. She currently leads an evaluation for GSMA, its partners and donors, specifically Sida, to better understand the mechanisms, results and impact of the Strategic Partnership for Digital Transformation in contributing to the achievement of ‘a digitally enabled 2030 development agenda’; and she holds a senior role in a long-term evaluation focused on higher education (SPHEIR). Clarissa previously led fund management work on risk and performance management for both the GPAF and CSCF DFID-funded grant programs. Katharine May – Senior Evaluator (Health / Governance / Open Innovation) Katharine is a monitoring, evaluation, and learning expert and has designed and reviewed monitoring and evaluation frameworks for projects and programs with a focus on governance, accountability, and adaptive management. Recent relevant experience includes leading on adaptive approaches for an evaluation of the Amplify fund, and as Methods Advisor to an evaluation of DFID’s Institutions 218 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION for Inclusive Development program in Tanzania. Previously, Katharine led on the management of health-focused grants within the fund management teams working on the GPAF and CSCF programs, drawing on a background in NGO health sector programs. Together with Clarissa, Katharine co￾facilitated a session on the evaluation of grand challenges at the UK Evaluation Society 2019 conference in London. Sudhanshu Joshi – Senior Evaluator (Education) Sudhanshu is a senior education expert and Athena Infonomic’s Director of Operations based in Washington D.C., providing their team with an important coordination link with D.C.-based stakeholders. With his extensive experience in education programing, he brings relevant experience of grant program fund management and design, and experience-based understanding of monitoring and evaluation principles, tools, and practices. He has expertise in the development of results frameworks and evaluation of multi-component programs, utilization-focused evaluations, formative and process evaluations and the application of DCED approaches to monitoring and results measurement and verification of development impact. Sudhanshu brings knowledge of a range of development fields including sustainable livelihoods, agriculture, natural resource management, small-scale irrigation and agricultural production, post-conflict and post-disaster recovery and disaster risk reduction. Dr Shoa Asfaha – Gender and Climate Expert (Civil Society / Environment / Climate) Shoa is a monitoring and evaluation, gender and inclusion, forest governance and climate change specialist, with more than 25 years’ experience in international development advising voluntary sector organizations, research institutions, governments, UN institutions and the donor community. She has a wide experience in helping agencies on how to integrate gender and social inclusion in their programs and brings a substantial track record working on civil society challenge funds. Shoa has worked in a wide range of contexts, including the Congo Basin, East, Central, and Southern African countries, and South Asia and Indonesia. Having worked on project implementation with communities (men, women) at program and strategic levels with a cross section of organizations, she understands the challenges and complexity of development and gender and social inclusion issues, and expectations of multiple stakeholders. Julian Ratcliffe – Analyst and Data Manager Julian played a key role in data collection, document review and data management for the recent GIF evaluation and brings strong knowledge of issues related to gender and inclusion. He has experience in a variety of research roles and has strengths in narrative analysis, the evaluation of qualitative data and argument, in structural and systems analysis, and in contextualizing information to provide far-reaching, unique, and creative insights. He has brought high-level research skills and expertise to a range of different approaches to promote social and economic development working across many sectors. David Smith – Cost Effectiveness Analysis (Inclusive business) With a background in economics and agriculture, David has 16 years’ experience in the design and implementation of evaluations, at project and program level, focusing on the results monitoring and evaluation of challenge funds. He has been team leader for numerous complex evaluations of multi-country, multi-donor funded programs implemented through a range of modalities and for a different client types including foundations, private sector clients, and bilateral and multilateral donors. He led the teams evaluating Sida’s portfolio of challenge funds, and the GIF evaluation. He has a strong background of experience in monitoring and evaluation of private sector development programs and the development of methodologies for impact assessments. He led IPE Triple Line’s support for monitoring and results measurement for the Africa Enterprise Challenge Fund (AECF), evaluations and MEL support to the Argidius Foundation, TradeMark East Africa, MasterCard Foundation, Shell Foundation, and a review of DFID’s Business Innovation Facility. Matthew Kentridge – Innovation and Scaling Strategy Matthew is a strategy consultant with more than 20 years’ experience specializing in program and project USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 219 management, operating model and organizational design, and growth and transformation strategies. He recently worked with IPE Triple Line both on the evaluation of Sida’s portfolio of 10 global challenge funds, and the evaluation of the GIF. He is a long￾standing adviser to DFID, working with the Research and Evidence Department and the Emerging Policy and Innovation Capability Unit on a range of topics including digital and innovation strategy and how to harness technology for development. He has also advised on the development of a concept and initial business plan for a proposed challenge fund focused on digital access in selected countries across Africa, Asia, and South America. In addition to DFID, Sida, and GIF, Matthew has worked with Omidyar Network, numerous government departments in the UK and South Africa, and a wide range of private sector companies, and led projects to promote innovation across the NHS in the UK. Promila Bishnoi – Quantitative Methods Adviser Promila is an experienced monitoring and evaluation expert. She brings to the team her background in economics and experience of applying quantitative methodologies to a wide range of thematic areas including health, education, rural sanitation, and water supply. 220 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION EVALUATION TIMELINE The evaluation will be carried out according to the activity timeline below. MAR-20 APR-20 MAY-20 JUN-20 JUL-20 AUG-20 SEP-20 OCT-20 TASK 2 9 16 23 30 6 13 20 27 4 11 18 25 1 8 15 22 29 6 13 20 27 3 10 17 24 31 7 14 21 28 5 12 19 26 Kick Off / Inception Meeting with USAID / GC Stakeholders 4 Inception Briefing Due 9 Evaluation Feasibility Assessment / Prioritization Recommendations Memo Due 30 Prioritization Workshop with USAID / GC Stakeholders 3 Draft Evaluation SOW submitted 24 Submit Evaluation SOW 1 Draft Evaluation Instruments submitted 8 Pilot test instruments 15 Finalization of Evaluation Instruments (following pilot testing) 20 Primary Data Collection 7 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION 221 MAR-20 APR-20 MAY-20 JUN-20 JUL-20 AUG-20 SEP-20 OCT-20 TASK 2 9 16 23 30 6 13 20 27 4 11 18 25 1 8 15 22 29 6 13 20 27 3 10 17 24 31 7 14 21 28 5 12 19 26 CEA Feasibility Assessment due 7 Interim Insights Memos on selected topics Submission of Draft Report, Draft Impact Measurement and Cost effectiveness Frameworks 4 Participatory review of drafts deliverables with USAID/GC Stakeholders Feedback on deliverables due from USAID/GC Stakeholders Feedback Integrated and Deliverables go to Design Final Deliverables submitted 8 Participatory Stakeholder Learning Workshop 222 USAID GRAND CHALLENGES FOR DEVELOPMENT META-EVALUATION