Photo: Bashar Shglila STRENGTHENING CIVIL SOCIETY GLOBALLY (SCS GLOBAL) PERFORMANCE EVALUATION Prepared under Contract No: GS-10F-0033M / Order No. 7200AA18M00016, Tasking N030 This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. DRG LEARNING, EVALUATION, AND RESEARCH ACTIVITY II STRENGTHENING CIVIL SOCIETY GLOBALLY (SCS GLOBAL) PERFORMANCE EVALUATION APRIL 2020 Prepared under Contract No.: GS-10F-0033M / 7200AA18M00016, Tasking N030 Submitted to: Brandy Witthoft, USAID COR Submitted by: Susan Abbott (Senior Evaluation Expert) Contractor: NORC at the University of Chicago 4350 East West Highway, 8th Floor Bethesda, MD 20814 Attention: Renée Hendley Telephone: 301-634-9489; E-mail: Hendley-Renee@norc.org DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │i TABLE OF CONTENTS ACRONYMS III EXECUTIVE SUMMARY 1 1. EVALUATION PURPOSE AND EVALUATION QUESTIONS 5 EVALUATION OBJECTIVES 6 EVALUATION QUESTIONS 6 2. PROJECT BACKGROUND 6 3. METHODS AND LIMITATIONS 7 QUALITATIVE APPROACH 7 LIMITATIONS 8 4. FINDINGS, CONCLUSIONS & RECOMMENDATIONS 8 EVALUATION QUESTION 1 9 EQ1—How has the SCS Global LWA been effective in advancing USAID priorities in CSM programming? 9 EVALUATION QUESTION 2 20 EQ2—Were the design and resource allocation within SCS Global LWA conducive to USAID’s needs in CMS programming? 20 EVALUATION QUESTION 3 25 EQ3—How has the SCS Global LWA utilized co-creation during the development and implementation of Associate Awards? 25 EVALUATION QUESTION 4 28 EQ4—How unique is SCS Global from similar DRG mechanisms? 28 CONCLUSIONS 30 RECOMMENDATIONS 34 38 59 77 80 89 96 99 ANNEX A: CASE STUDIES ANNEX B: CONCEPT NOTE ANNEX C: EVALUATION QUESTIONS ANNEX D: DATA COLLECTION INSTRUMENTS ANNEX E: LIST OF PERSONS INTERVIEWED ANNEX F: CONFLICT OF INTEREST DISCLOSURE ANNEX G: OTHER DOCUMENTS CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │ii FIGURES Figure 1: SCS Global LWA Objectives 1 Figure 2: Leader Award Activities 1 Figure 3: The SCS Global Consortium of 21 partners, plus USAID 5 Figure 4: Goals and Objectives of SCS Global LWA 7 Figure 5: USAID DRG Development Objectives 9 Figure 6: List of SCS’s Associate Awards 9 Figure 7: Iterative Learning between Leader and Associate Awards 11 Figure 8: KII Feedback and Analysis of the Google Analytics of NGOConnect.net 14 Figure 9: Leader Objective—Key Deliverables and Tools 15 Figure 10: Useful Parameters for Co-Creation 25 Figure 11: Strengths and Weaknesses of Pre-Award and Post-Award Co-Creation 28 Figure A1: Mural in Zambia of African leaders Nelson Mandela, Kofi Annan, and Kwame Nkrumah 39 Figure A2: Youth playing soccer in downtown Djibouti 40 Figure A3: Photo of Djibouti Mural 42 Figure A4: Outside Wall of the Ministry of Health in Djibouti 43 Figure A5: Cohort of Youth Lead interns outside of FHI 360 office 48 Figure A6: Youth Leader Fellow at the World Bank in Lusaka with her Mentor 50 Figure A7: Coalition led by the Kenyan Media Council (includes other key Kenyan voices in media self-regulation) 56 CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │iii ACRONYMS AA Associate Award BBI Building Bridges Initiative CDCS Country Development Cooperation Strategy CLA Collaboration, Learning, and Adaptation COP Chief of Party CSM Civil Society and Media CSO Civil Society Organization DAI Development Alternatives International DRG Democracy, Human Rights and Governance DRG-LER Democracy, Human Rights, and Governance Learning, Evaluation, and Research Activity DRL U.S. State Department’s Bureau of Democracy, Human Rights and Labor FAFT Financial Action Task Force Toolkit FHI 360 Family Health International GEPT Global Elections and Political Transitions HRSM Human Rights Support Mechanism ICNL International Center for Non-Profit Law JTSR Journey to Self-Reliance KII Key Informant Interview LWA Leader with Associate Awards M&E Monitoring and Evaluation MERL Monitoring, Evaluation, Research, and Learning NDI National Democratic Institute NGO Non-governmental Organization OGP Open Government Partnership PRECAD Programme de Renforcement des Capacités des Associations de Djibouti RDMA Regional Development Mission for Asia CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │iv SADES-K Safeguarding Democratic Space in Kenya SCS Global Strengthening Civil Society Globally SI Social Impact WSR Whole Systems in the Room USAID United States Agency for International Development YPAT Youth Programming Assessment Tool CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │1 EXECUTIVE SUMMARY As part of the Democracy, Human Rights, Governance (DRG) Learning, Evaluation, and Research II (DRG-LER II) Activity, USAID requested NORC to design and conduct a performance evaluation of USAID’s Strengthening Civil Society Globally (SCS Global) Leader with Associate Awards (LWA), a Cooperative Agreement implemented by Family Health International 360 (FHI 360), which began in May 2016 and ends in May 2021. SCS Global’s total Leader with Associate Awards (LWA) ceiling funding is $2,500,000; the estimated LWA’s Associate Awards (AAs) ceiling was originally $200,000,000,1 and, based on Mission demand to date, was recently increased by $300,000 for the remainder of the award. To date, SCS Global has made 18 AAs in 18 countries, cumulatively totaling $199,138,173.2 The principal objective of SCS Global is to develop and implement a new approach to Civil Society and Media (CSM) programming that offers USAID and its DRG partners an evidence-based way of supporting and creating robust, context￾appropriate, and forward-thinking CSM activities that advance DRG objectives and thereby advance other development goals (education, health, economic growth, environment, among others). This performance evaluation examines: 1) how effective SCS Global has been under the Leader award at fostering leadership, identifying key DRG trends and challenges in the civil society and media sectors, and demonstrating thought leadership to address those challenges across the broader sector; 2) the extent to which SCS Global is meeting the needs of the Agency in the CSM sector; and 3) how effective the LWA has been at integrating technical leadership and products into its 18 AAs, how SCS Global has served Missions, and how the mechanism was capable—or not capable—of adapting, to date. FINDINGS NORC’s evaluation team’s overarching finding is that the SCS Global implementing partner, FHI 360, has effectively and ably managed Leader Objective 1.2 and the AA Objective, which comprises the majority of the overall SCS Global LWA mechanism’s funding. However, FHI 360 as the Prime Contractor of the SCS Global LWA falls short in the execution of the activities and deliverables that 1 This figure is an estimated amount. There is no guarantee regarding the magnitude of AAs in dollars or number of awards. The actual total of AAs funding originally was set to not exceed this amount, though USAID has indicated that it may increase the amount. 2 A full list of the AAs and their respective total estimated costs are in the Annex and below in a Table. Figure 1: SCS Global LWA Objectives LEADER OBJECTIVE 1.1: Develop and promote effective approaches to the strengthening of civil society and independent media actors. LEADER OBJECTIVE 1.2: Effective management of SCS Global LWA. AA OBJECTIVE: Provide technical assistance and support to a wide range of civil society and media actors and organizations to enhance participation, inclusion, transparency, and accountability across sectors. Resource Hub: www.NGO Connect.net SCS Global Update Newsletter Implementation Tips for USAID Partners Research and Learning Agenda New and Emerging Trends Thought Leadership and Learning Forums Figure 2: Leader Award Activities CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │2 fell under Leader Objective 1.1 (see Figure 2). As elaborated on more in the report, it is not clear, for all the effort that FHI 360 has put into Objective 1.1., that the deliverables and outputs were useful to the wider CSM/DRG community or that they are improving the delivery of USAID/DRG programming. A key theme that emerged from the evaluation is that USAID is still seeking the best practices and lessons that can directly inform its CSM programming. A central challenge for both USAID and FHI 360, and perhaps by default the rest of the SCS Global Consortium (FHI 360 refers to consortium members as partners), is that USAID knows what it does not want in current Leader Award performance, but doesn’t have a ready-made prescription for what this looks like. Rather, USAID/DRG looks to FHI 360 and its management of the Consortium of partners, to be bold, creative, and forward thinking and to come to them with a vision and a clearly articulated and innovative workplan and strategy for delivering knowledge products and effective learning approaches that will strengthen civil society and media. Instead, the current implementation of the SCS Global LWA Mechanism has placed the overwhelming focus on management of a set of AAs and has not yet found the right footing or creative vision for delivering the set of tools, knowledge products, and learning activities that underpinned the initial vision that USAID/DRG had for SCS Global. In short, the evaluation finds that while the management and administration of the AAs is strong, current approaches used and products developed for SCS Global as outlined in the Activity Purpose3—“to identify and develop evidence-based effective approaches to strengthen civil society and media”—have fallen short of USAID/DRG expectations and that a refined and different vision for the delivery of the results anticipated as part of the LWA’s Objective 1.1 is warranted. The chief takeaway that the evaluation team noted was that FHI 360 did “develop effective approaches to civil society strengthening,” but they did not succeed in strengthening civil society and media more broadly through the development of tools that could be used by the Missions and other organization or creating spaces for CMS and DRG program to engage in dialogue. KEY CONCLUSIONS The conclusions are grouped together in accordance with the four evaluation question areas: HOW HAS THE SCS GLOBAL LWA BEEN EFFECTIVE IN ADVANCING USAID PRIORITIES IN PROGRAMMING? 1. The SCS Global LWA has been effective in advancing USAID priorities in CSM programming chiefly because it has provided a useful mechanism to manage the AAs and enables both the DRG Center and local USAID Missions to support and bring the CSM programming to life. 2. Under FHI 360’s leadership, the SCS Global LWA has been less effective, compared to the aforementioned components, with the fulfillment of the activities outlined under Objective 1.1. 3. The program has built a solid foundation upon which FHI 360, USAID, and the wider SCS Global consortium can build. WERE THE DESIGN AND RESOURCE ALLOCATION WITHIN THE SCS GLOBAL LWA CONDUCIVE TO USAID’S NEEDS IN CSM PROGRAMMING? 4. The design and resource allocation of SCS Global, as envisioned and developed by USAID, offered a fresh approach to civil society and media support. 5. Leader Objective 1.1, which includes thought leadership through the development and dissemination of new toolkits, approaches, and methodologies to support CSM programming, fell 3 See SCS Global Results Framework in the Appendix. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │3 short of expectations to provide significant innovation and learning. (Note, the metric or standard against which the evaluation team considered the thought leadership aspects of Objective 1.1 were the qualitative feedback provided by the KIIs and an evaluative review of the knowledge products, events and platforms outlined in Figure 2 and 9 in this report) 6. A chief shortcoming in the SCS Global LWA was a perceived imbalance of funds managed and held by FHI 360 as compared to other partners in the consortium, as well as resources available for local partners and the intended beneficiaries of the funding. 7. Over-focus on deliverables coming from the project Leader in Washington appeared to have stymied more active participation across the consortium and network. HOW HAS THE SCS GLOBAL LWA UTILIZED CO-CREATION DURING THE DEVELOPMENT AND IMPLEMENTATION OF ASSOCIATE AWARDS? 8. SCS Global has used co-creation, notably both pre-award and post-award, and in program implementation to create a robust effort to support locally led development. 9. FHI 360 has incorporated and championed co-creation tools with inconsistent results. 10. The LWA has encouraged local collaboration, learning, and adaptation of each AA. HOW UNIQUE IS SCS GLOBAL FROM SIMILAR DRG MECHANISMS? 11. SCS Global is a unique mechanism that allows greater flexibility to do more cross-sectoral programming than some other mechanisms. RECOMMENDATIONS The Evaluation Team’s overall recommendation is for USAID to continue this global civil society strengthening initiative to build on and advance the networks, results, and products produced in this first phase of the SCS Global LWA mechanism. The program, despite faults in the thought leadership of objective 1.1, has built a strong foundation for the future. Some specific recommendations to enhance the opportunities for thought-leadership and collaborative programming are: 1. Plan a pause-and-reflect process within SCS Global and USAID using the same principles of co￾creation that are espoused at the country level. This is an important time to figure out next steps to take what FHI 360 has developed and improve upon it. In the remaining program time, all members of the consortium should be invited to attend a facilitated learning event that uses the same principles of co-creation espoused in the start-up of the AA. This meeting should have an eye toward capturing lessons learned and a learning agenda that would support the consortium of partners involved in SCS Global and for use by the DRG CSM community. A learning focus might include: − A follow-up global survey targeting all SCS Global partners, beneficiaries, and USAID leads in Washington, D.C., and the Missions who took part in the LWA − Reflection on SCS Global’s thought leadership and knowledge management role − Stock-taking of SCS Global experience with co-creation, in which FHI 360 can identify a set of strategic best practices drawn from its program experience that will inform future iterations of the initiative. USAID Missions are keen to get feedback and advice on co￾creation as a tool and process, and SCS Global could provide a real value added. − Collaborative learning mechanisms for USAID and FHI 360—along with the rest of the SCS Global Consortium—to work together in a partnership around learning CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │4 − Discussion about best practices and desired channels and formats for information dissemination related to SCS Global. Use this pause and reflect for developing user-centered design approaches to solicit feedback and strategic direction on the NGOConnect.net and determine how to better communicate with and deliver thought leadership and knowledge products to the DRG community; dissemination strategies for USAID may differ from those required to reach local partners and the wider DRG community, which should be taken into consideration. 2. Deepen the work around thought leadership (Objective 1 of SCS Global) with a focus on improved outputs in the final year of the program and making it a greater priority in future iterations of the program. This recommendation stems from the pronounced difference between the effectiveness and perceived impact of Objective 1 and Objective 2 and questions the evaluation team had about how to best improve the delivery and execution of the Leader Objectives. The program has stimulated significant learning that can serve civil society and media organizations but needs to be communicated and discussed both within and outside of the program. In this way, there are short-term and long-term improvements that can be made. 3. Revise the design and resource allocation under the Leader Award. USAID and FHI 360 should sit together to come up with a plan to better meet the needs of USAID’s DRG programming. One of the notable observations from the Evaluation Team is that FHI 360 seems to have an outsize role in both the Leader activities as well as in the AAs; the role or purpose of FHI 360’s 20-member consortium is unclear, as they are being underutilized. 4. Facilitate knowledge management and learning systems for the 21st century DRG practice. CSM programming has long faced a challenge with regards to how to improve the uptake of evidence in program design and implementation. The evaluation team suggests that SCS Global and FHI 360 spend the remaining time on the award to facilitate through the Consortium a sector-wide discussion of the problem and its possible solutions. 5. Improve program reporting to target the needs of country stakeholders and USAID. Program reporting to USAID for the LWA needs to be streamlined, accessible, engaging. Shorter, more action-oriented reports that capture learning and key insights about the CSM space and DRG goals is needed. 6. Stress opportunities for interpersonal interaction in the program delivery. Currently, much of the interactions among Associates and USAID is facilitated only through FHI 360, even though many FHI 360 consortium members are USAID partners on other programs and provide direct feedback in those capacities. There is a need for community-wide conversation, participation, and group learning events (like an annual SCS Global Conference) that would bolster the type of engaged learning this LWA has the potential to inspire. 7. Improve functionality of NGOConnect.net to make it more accessible and user friendly for civil society and media organizations around the world. SCS Global has a chance to curate and cultivate an online learning platform that could be useful to many in the CSM and DRG community. The need is there, and this LWA has the potential to provide informed insight on what users can use and incorporate into their programs. A more contemporary site with improved design, functionality, and organization would be beneficial to the SCS Global Mechanism and others in the DRG community. USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │5 1. EVALUATION PURPOSE AND EVALUATION QUESTIONS As part of the DRG Learning, Evaluation, and Research II (DRG-LER II) Activity, USAID requested NORC to design and conduct a performance evaluation of USAID’s Strengthening Civil Society Globally (SCS Global) Leader with Associate Awards (LWA) Mechanism, which began in May 2016 and will be completed in May 2021. SCS Global total Leader Award funding is currently $2,500,000, and the estimated LWA’s Associate Awards (AAs) amount was originally $200,000,000; based on Mission demand to date, however, this was recently increased to $300,000 for the remainder of the award. To date, SCS Global has made 18 associate awards, with LWA funding in the amount of $199,138,173.4 SCS Global is designed to identify and develop effective approaches to strengthen civil society and media in support of democracy, human rights, and governance (DRG) and other development results in open and politically restrictive environments. The LWA focuses primarily on design and learning as well as fostering technical leadership and brokering connections between specific LWA interventions to enhance outcomes and achieve economies of scale. Some specific activities include generating practical tools and/or knowledge useful to most LWAs, as well as other related efforts. The award is implemented by FHI 360, with a consortium of partners including: Internews, Social Impact (SI), and Development Alternatives International (DAI); 13 other partners; and 4 network partners. Figure 3: The SCS Global Consortium of 21 partners, plus USAID This performance evaluation comes at a time when widely acknowledged closing spaces for civil society and threats to Internet freedom are on the rise—governments around the world regularly engage in repression of civic and political freedoms and acts of surveillance. In the countries where USAID operates, governments also increasingly “shut down the internet” as a response to civic action. The collective use of digital tools, data, and infrastructure has created an atmosphere in which technology often outpaces the legal and regulatory customs, norms, and policies that govern freedom of expression and access to information. Most recently, repressive regimes have used the global COVID-19 pandemic to exacerbate their controls over free speech and other political rights. All of these trends have created a need for programs like SCS Global to strengthen independent media and civil society in the face of difficult obstacles. Despite the worrisome political, economic, and social environments where many CSM programs take place, there are some opportunities for SCS Global in terms of how the mechanism can be used to better support civil society’s role as advocates, watchdogs, and service providers—all increasingly important in a context where civil society is under threat. 4 See Figure 5 for a full list of the AAs. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │6 EVALUATION OBJECTIVES NORC’s performance evaluation examines how effective SCS Global has been under the LWA agreement at fostering leadership, including identifying key DRG trends and challenges in the Civil Society and Media (CSM) sectors, and demonstrating thought leadership to address those challenges across the broader sector. It assesses the extent to which the SCS Global LWA is meeting the needs of the Agency in the CSM sector. Finally, it assesses how effectively the LWA award has integrated technical leadership and products into its 18 AAs, how SCS Global has served Missions, and how the mechanism was capable—or incapable—of adapting to Mission and operating unit needs.5 EVALUATION QUESTIONS The following overarching evaluation questions are addressed in this evaluation report. Annex C includes a full listing of evaluation questions and sub-questions. 1. How has the SCS Global LWA been effective in advancing USAID priorities in CSM programing? 2. Were the design and resource allocation within SCS Global LWA conducive to USAID’s needs in CMS programming? 3. How has the SCS Global LWA utilized co-creation during the development and implementation of AAs? 4. How unique is SCS Global from similar DRG mechanisms? 2. PROJECT BACKGROUND To implement its CSM approach, FHI 360 built from complementary USAID programs (such as the Civil Society Innovation Hubs, the Open Government Partnership (OGP), and the Community of Democracies) to allow civil society and media in open and closed spaces to collaborate, coordinate, and act in new and more effective ways. According to the initial vision of the LWA program design, the results of the LWA activities are meant to inform approaches under AAs and in international development more widely, potentially influencing the entire field of civil society and media promotion. This LWA was envisioned to enable more consistency in USAID support to CSM, demonstrate value-added to CSOs and media, and broaden development approaches aiming to achieve both DRG and other development outcomes. SCS Global’s theory of change lays out a connection between CSM strengthening activities and increased capacity for achieving DRG and civil society objectives: “If evidence-based effective approaches to strengthen civil society and media in support of DRG and other development results in both opening and closing environments are identified, effective approaches to the strengthening of civil society and independent media actors are developed, and the SCS Global Leader with Associate Award is effectively managed, then civil society and independent media organizations will be strengthened to advance DRG objectives and they will also advance other development goals (education, health, economic growth, environment, etc.).” The overall goals and objectives of SCS Global are presented in Figure 4 below. SCS-Global incorporated the following cross-cutting programming principles as stated in the award: 1) coordination; 2) sustainability; 3) gender, youth, and other inclusive development considerations; and 4) monitoring, evaluation, and learning. 5 Annex B provides the Concept Note and a more detailed Evaluation Matrix used to guide this study. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │7 Figure 4: Goals and Objectives of SCS Global LWA 3. METHODS AND LIMITATIONS NORC’s approach to the SCS Global performance evaluation entails a mix of mutually reinforcing qualitative methods that reflect the project logic and research questions being addressed. Qualitative analysis focused heavily on feedback from USAID’s DRG Center and USAID Mission staff and on information and evidence gleaned from three country case studies (produced from in-depth field visits to Djibouti, Kenya, and Zambia; see Annex A) to provide concrete examples that illustrate in greater detail the process and results of the project. NORC’s approach to selecting the appropriate methodology was based on the USAID Evaluation Policy, its experience conducting evaluations in the field, and budgetary restrictions. NORC conducted the SCS Global performance evaluation in a participatory manner, finalizing the evaluation design in consultation with USAID and the SCS Global Leader Implementing Partner and implementing the evaluation in close coordination and/or consultation with USAID staff in Washington and Mission offices as well as FHI 360. QUALITATIVE APPROACH NORC’s evaluation of SCS Global relied on qualitative data collection and analysis, including: • Desk-top review of materials related to the SCS Global LWA, such as the Leader Award Agreement; select AAs; co-creation information; and documents primarily related to the LWA and select AAs: Performance Management Plan (PMP); Monitoring, Evaluation, Research, and Learning (MERL) Plans; CLA documents; annual work plans; quarterly and annual reports; country assessments; trip reports; local partner assessments and training materials; thought leadership–related publications; and other materials. USAID Country and Regional Development Cooperation Strategies (CDCSs/RDCSs) were also reviewed. • KIIs with USAID, SCS Global staff and consultants, program beneficiaries, and other subject matter experts (in the United States and overseas) who work directly with local partner organizations; representatives of local partner organizations cooperating with SCS Global; and those who work in similar projects/programs. Annex D includes the names and affiliation of 99 individuals interviewed as part of this performance evaluation. Goals: Strengthen civil society and independent media organizations to advance DRG objectives Strengthen civil society and independent media organizations to advance other development goals (education, health, economic growth, environment, etc.) Leader Objective: Identify and develop evidence-based approaches to strengthen civil society and media in support of DRG and other development results in open and closing environments. Leader Objective 1.1: Develop and promote effective approaches to the strengthening of civil society and independent media actors. Leader Objective 1.2: Effective Management of SCS Global LWA Associate Awards Objective: Provide technical assistance and support to a wide range of civil society and media actors and organizations to enhance participation, inclusion, transparency, and accountability across sectors. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │8 • Case Studies. NORC’s evaluation team developed three illustrative case studies per the guidance provided in USAID’s Tasking Request: one 2-page case study product related to co￾creation findings (Kenya) and two 5-page case studies (Zambia and Djibouti). These case studies bring the project’s work to life and help the reader understand the context and operating environment in which the projects take place. The case studies were carried out through field work in February 2020 and seek to offer an operationalized understanding of how the LWA has benefited CSM programming. Case Studies are included in Annex A. LIMITATIONS • Generalizability is a challenge. While the evaluation team endeavored to interview as many people as possible, time and resource allocations prevented NORC from extending and widening the interview period and pool of respondents beyond what was reasonable in the time allowed. Moreover, partner interviews were only focused on three countries in a global program—Djibouti, Kenya, and Zambia. None of the other 15 AAs were profiled or assessed as part of the evaluation. • Lack of survey data was a challenge on this evaluation. The evaluation team relied largely on feedback from KIIs and group interviews. However, a survey would have enabled NORC to canvas a much broader and more representative pool of respondents and to get a wider range of opinions about the overall performance of SCS Global. Moreover, many of the evaluation questions would have been better answered with a survey questionnaire.6 • Data availability and data quality. While the implementer and evaluation team have tried to collect and generate primary data, some data is difficult to obtain and/or is of questionable quality. Some respondents targeted for interviews were not available or willing to participate due to issues and/or changes in their involvement in the SCS Global activity and/or lack of availability in terms of scheduling. Many interviewees who participated in the program took part in a “one-off” manner and only attended one workshop or training. • Recall bias. Since a number of questions raised during the interviews addressed issues or activities from the past, recall bias may have affected respondents’ answers. 4. FINDINGS, CONCLUSIONS & RECOMMENDATIONS The principal objective of SCS Global was to jumpstart or ignite a new approach that offers USAID and its DRG partners an evidence-based way of supporting and creating robust, context-appropriate, and forward-thinking CSM programming. The SCS Global LWA is a five-year program that has supported AAs in 18 countries. A separate, but parallel, effort is the Leader Award, which is managed by FHI 360 and meant to provide thought-leadership around CSM programming. These leadership was intended to support and improve the AAs that are part of this LWA and to have a knock-on effect in supporting other CSM programs as part of USAID’s wider effort to promote democracy and good governance. The evaluation team’s overarching finding is that FHI 360 has effectively managed the AA Objective, which comprises the majority of the overall SCS Global LWA mechanism’s funding. This aligns with feedback offered by USAID KIIs done for this evaluation. Where FHI 360 has fallen short, however, is in the execution of the Leader Objectives as they pertain to identifying and developing evidence- 6 Note: NORC suggested using a survey as part of the evaluation’s research design, but due to budgetary reasons, a survey was not feasible. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │9 based effective approaches to strengthen civil society and media in support of DRG and other development results in open and closing environments (Figure 5). The general feedback from KIIs with USAID staff suggest that USAID is still seeking best practices and lessons that can directly inform its CSM programming. USAID looks to the LWA—in particular the Leader—to provide the technical leadership needed to provide a critical, yet helpful voice in shaping DRG thought leadership around CSM programs. One of the central issues that needs attention is the program’s theory of change.7 There’s a need to fine tune or adjust how the program delivery could be improved to emphasize the two Leader Objectives of the LWA as outlined below. The evaluation team found that there’s a noticeable disconnect between the program’s rationale and theory of change and what FHI 360 has delivered: namely in that it’s not clear that this current iteration of SCS Global has helped to strengthen or advance DRG objectives. EVALUATION QUESTION 1 EQ1—HOW HAS THE SCS GLOBAL LWA BEEN EFFECTIVE IN ADVANCING USAID PRIORITIES IN CSM PROGRAMMING? The primary way SCS has advanced USAID priorities in CSM programming is through the effective management of 18 Associate Awards listed in Figure 6 below. Figure 6: List of SCS’s Associate Awards COUNTRY AWARD NAME AMOUNT OF AWARD REGION AWARD LENGTH (YEARS) Jamaica Local Partner Development $15,000,000 LAC 5 Guinea Citizens’ Involvement in Health Governance (CIHG) $12,170,000 AFR 3 Tanzania Boresha Habari/Media and Civil Society Strengthening Activity (TMCS) $9,700,000 AFR 5 7 The theory of change is: if evidence-based effective approaches to strengthen civil society and media in support of DRG and other development results in opening environments, closing environments are identified, effective approaches to the strengthening of civil society and independent media actors are developed, and the SCS Global Leader with Associate Award is effectively managed, then civil society and independent media organizations will be strengthened to advance DRG objectives, and they will also advance other development goals (education, health, economic growth, environment, etc.). Figure 5: USAID DRG Development Objectives DO 1: Promote participatory, representative, and inclusive political processes and government institutions. DO 2: Foster greater accountability of institutions and leaders to citizens and to the law. DO 3: Protect and promote recognized human rights. DO 4: Improve development outcomes through the integration of DRG principles and practices across USAID’s development portfolio. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │10 COUNTRY AWARD NAME AMOUNT OF AWARD REGION AWARD LENGTH (YEARS) Bosnia and Herzogovenia Balkan Media Assistance Project (BMAP) $8,625,069 EE 4 Kyrgyzstan Media K Cultivating Media Independence Initiative Activity $6,500,000 EE 4 Global— DCHA/DRG The Civil Society Organization Sustainability Index (CSOSI) $9,500,000 Global 5 Djibouti Civil Society Organizations’(CSO) Strengthening Program $2,900,000 AFR 2 Honduras Civil Society and Media Activity $4,760,112 LAC 4 Kenya Safeguarding Democratic Space in Kenya (SADES-K) $6,350,000 AFR 3 Azerbaijan ECSOFT $2,499,519 EE 5 Burma Advancing Transition to Peaceful Democracy— Participation of Civil Society & Media (CSM￾II) $25,000,000 ASIA 5 Zambia Youth Lead $2,818,743 AFR 3 Botswana Accelerating Progress in Communities 2.0 (APC 2.0) $25,000,000 AFR 5 Indonesia MADANI—Civil Society Support Initiative $19,800,000 ASIA 5 Democratic Republic of the Congo Media Sector Development Activity $15,000,000 AFR 5 Regional Development Mission for Asia (RDMA) Asia Networks for Peace $10,000,000 ASIA 5 Cambodia Innovations for Accountability $15,000,000 ASIA 5 Chad Civil Society and Media Strengthening Award $8,514,730 AFR 4 The level of interest from USAID Missions has been high: “Almost every day we get a new request for this mechanism—it’s a symptom of a couple of factors, not just the award.” (KII, USAID) As noted on the SCS Global website, this mechanism “is USAID’s way to quickly initiate, design and manage programs that enhance civil society, promote independent media, and integrate participatory approaches, social inclusion, transparency and accountability in all sectors.” SCS Global, however, is not intended to only manage the AA, though that should be recognized as a critical achievement unto itself. The unique quality of SCS Global is the features of the Leader CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │11 objectives of the award: “It’s a traditional LWA, but we wanted to push as much as possible the learning piece. How to embed the learning. Leader and thinker on CSM. We wanted creative approaches for learning.” (KII, USAID) Another USAID KII noted, “There are things that we are satisfied with and others that I don't think they have the most creative ideas for how to deliver.” (KII, USAID) The LWA has not hit the mark in providing the thought leadership and creativity8 that USAID was expecting from Leader Objective 1.1. FHI efforts to advance USAID’s priorities for this LWA (e.g., a combination of learning, leading by example of identifying, and applying best practices and approaches to CSM programming, and thought leadership about civil society and media; see Figure 7 below) has not met USAID’s expectations. KIIs emphasized that more could be done to leverage learning and share information and insights more broadly both to USAID and the broader DRG community. KIIs also said that FHI could be more creative, innovative, and entrepreneurial. This section further explores the strengths and weaknesses of the current approach with regards to Leader Objective 1.1. Figure 7: Iterative Learning between Leader and Associate Awards 1.a What are the most significant achievements, both intended and unintended? KIIs with USAID staff were consistent in their appreciation for FHI 360’s effective administration and management of the AAs. As shared by one USAID KII, “As partners they (FHI 360) have been phenomenal, making sure the LWA is responsive to DRG. They were good at managing and negotiating the space. They were very agreeable in working with the USAID team. They are very competent people.” (USAID, KII) Nearly all KIIs with USAID staff noted that SCS Global has been an important mechanism for USAID missions. As a partner, FHI 360 has been able to make diverse, contextually appropriate CSM 8 Creativity is an adjective used in the feedback from many key informants during interviews when they discussed Objective 1.1. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │12 programming available with respect to the level of development in each country. Examples of such programming include: • There were a diverse range of funded AAs. (See Figure 6 above.) • The Delphi panel approach had some success.9 While it had some mixed reviews, implementation of Delphi was a significant achievement because it produced and collected a wide range of opinions from CSM thought leaders and crafted a learning agenda on which the community felt it was important to focus. As noted by a USAID KII, “The Delphi Process was about learning and identifying gaps. The Delphi was a good effort in helping to identify what are the key issues in the field. It was not as successful in moving forward the process of advancing research, partly because they had to deal with CSM/USAID Division. Delphi was a good effort, but it fizzled out.” So, while this was an achievement, KIIs were critical of its utility to bring about results and drive the Leader Objectives. • The Local Philanthropy webinar, in particular, was frequently mentioned by USAID staff. For instance, as one USAID official shared, FHI 360 “has been technically strong and forward thinking about guidance to local philanthropy. They hired someone who was from Aga Khan and are truly responsive to us (USAID) on this.” (KII, USAID) “This toolkit started as an event, a thought leadership forum, last spring 2019. It was done in collaboration with Aga Khan Foundation.” (FHI 360 KII) • The Financial Action Task Force Toolkit (FAFT) was also highlighted: “ICNL (International Center for Non-Profit Law) did it. It is one of the strongest things that they have done. They’ve rolled it out in different ways.” (KII, USAID) FATF offers a toolkit for Defending Civic Space from Excessive Legal Regulation. It is designed to help CSOs map key laws related to money laundering and terrorism financing and evaluate how well they comply with FATF standards. • Some of the co-creation approaches were lauded. From the evaluation team’s review, one of the most interesting was FHI 360’s Whole Systems in the Room (WSR) methodology for co￾creation, which brought to life a multi-stakeholder approach to DRG programming. This approach adds value for the DRG community because it goes beyond the traditional organizational capacity building tools and self-assessments and is more partnership driven and context rich and creates a two-way dialogue. The resulting effort is not designed by experts coming into a local environment with only program delivery expertise. • FHI 360’s Monitoring and Evaluation (M&E) team was frequently cited as being a good asset for the DRG, hosting community-based weekly or biweekly meetings with field teams, biweekly meetings with MERL backstops, regular MERL roundtables, and institutionalized biannual reflection meetings. “These are well-attended—practical challenges and issues—they are facing similar issues. The way the message is delivered creates a buzz and hype around it.” (KII, FHI 360) FHI 360’s M&E team and approaches to MERL are regarded as significant achievements for working comprehensively with local partners to meaningfully gather useful MERL from local environments and make it available to achieve better program results. • The building of ties between government and civil society was notable in all three case studies—each with differing, but important, examples of where linkages and relationships were 9 The Delphi Process or Delphi technique is a structured communication technique or method, originally developed as a systematic, interactive forecasting method that relies on a panel of experts. It typically includes at least two rounds of experts answering questions and giving justification for their answers, providing the opportunity between rounds for changes and revisions. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │13 established/strengthened between civil society and government that will advance DRG programming because of SCS Global/FHI 360’s leadership. The AAs’ impacts go even further by resulting in strengthening capacities of some government agencies to deliver important services. The Zambia Youth Lead AA advanced delivery of key government services in cash-strapped ministries. The Djibouti AA resulted in linkages with civil society and FHI 360 opening up new lines of work for civil associations and creating opportunities for the government ministries to fulfill their mandates and be responsive to their respective missions, most notably with the Djiboutian Ministry of Health. 1.b What are key weaknesses of the program? What are the major challenges? • Connectivity and Communication. While there are some exceptions, as noted in the examples above, feedback from the KIIs revealed that dissemination and distribution of tools, thought leadership, and learning are core challenges and weaknesses. FHI 360 has good tools and information, but they are not always proactively and usefully shared. “The technical content is strong, but the rollout and the dissemination is weak. They (FHI 360) do not keep enough budget for marketing and dissemination,” (KII, USAID) In the same vein, one USAID KII also felt that FHI 360 could have been more innovative, but the challenge with LWAs across the board is how to integrate learning into the AAs. “This is everyone’s fault. With Missions coming into the game wanting to move money, it’s hard to allow time for an innovative approach.” (KII, USAID) • NGOConnect. NORC’s evaluation team found NGOConnect to be a useful resource hub, but KIIs noted they seldom used it, and many people interviewed didn’t know it existed. Within the NGO community there are a handful of important and useful destination websites for evaluation, research, and development practitioners to learn from, share knowledge on, and forge a community of practice for research and learning. SCS Global, through NGOConnect, has an opportunity to carve out a space for DRG, especially in terms of learning and improving civil society and media programs. • Data overload. FHI 360 has a tremendous asset in this wealth of data, but it is not always clear how the data are being used and integrated to adapt, refine, and improve programs, let alone inspire cross-sectoral DRG learning. Moreover, SCS Global has now produced and collected so much data that it has become a challenge to cope with all the research and insights that SCS Global collects. This creates somewhat of a new challenge for the LWA, but also one that other DRG programs are facing: how can all these collected data and insights be meaningfully channeled back into management, program design and DRG thinking? • Dissemination. FHI 360’s approach was seen by some KIIs as an impediment to the dissemination of major thought leadership project outcomes. FHI and the LWA in general provide effective assistance and management of the AAs, but this did not optimally translate into ways that pushed forward the next generation of thought leadership and useful approaches to DRG support. Notably, as detailed in the graphic above, the NGOConnect.net website was not well designed or utilized in an optimal way, and very few KIIs (from both USAID and other DRG stakeholders) said they used it. • Creativity and Presentation Style. USAID and local partner KIIs expressed some concern that FHI 360 has lacked creativity in program design and delivery and that they frequently have long time lags in sharing creative products. KIIs with USAID staff mentioned the latter frequently CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │14 • in regard to the delivery of reports and in terms of giving USAID DRG team members timely, swift, and meaningful feedback that the thought leadership piece of this LWA was meant to engender and should uniformly feature. As shared by a USAID representative when citing an example of their disappointment with one of the FHI tools, “I don’t want this to look like a boring old document. It’s fine, but it is not creative. It’s the same with the Youth Programming Assessment Tool. It’s not interesting to look at. How can they make it look better?” (KII, USAID) • Solutions, Not Problematizing, Needed. The Delphi process was a good effort, but it was not successful in advancing civil society initiatives. The process did not appear to have had much if any follow-up, and the impact of its benefits rapidly faded due to lack of follow-up/application. Moreover, as shared by a USAID representative, “A lot of the efforts identify the problems, but the LWA should be the response, i.e. the interventions and approaches. We were looking for something that was proactive, not reactive. We spend a lot of time talking about how bad things are and not enough time coming up with solutions.” (KII, USAID) • Underutilization of the LWA’s consortium. One of the core weaknesses of the program was that FHI 360 did not fully involve the LWA’s consortium or utilize their expertise to the extent that was hoped for by USAID. This sentiment was shared by USAID DRG KIIs in Washington, D.C., and at the Mission level. Because of the heavy degree to which FHI allocated the work to itself, there have been Figure 8: KII Feedback and Analysis of the Google Analytics of NGOConnect.net 1. The site has no “call to action.” It is not clear what a user should be doing on the site. The current site design has no “guided journey,” and the site is designed in a way that that one would need to know what to look for and how to access the information on the site in order to use it. The site design requires too much on the user side and should be redesigned to be more user-friendly. The purpose and the goals of the website are not clear. 2. It’s not clear, based on FHI 360 documentation, what the marketing/journey would look like for the NGOConnect site. How does FHI 360 and the SCS Global Consortium define success? One potential way would be to define the key pieces that FHI 360 and USAID want people to download and to have more of a dissemination strategy based on audience research and goal/target tracking of its online platforms for information dissemination. 3. The current organization of material and information on the site appears like a warehouse of information. It all seems important and valuable, but it is presented in a way that has no consideration about how to get from Point A to Point B and so on. 4. The website does little to build community or improve communication flows. It is hard to review the website and know how to become part of the NGOConnect community and how to stay in touch. 5. There is no social media platform or engagement on the site. Many (most) of the potential users of this site that live and work in the “global South” may access the information through social media rather than just a dedicated web-based platform. 6. Most of the site appears to be in English, which may be a limitation of budget and bandwidth of the LWA, but given the target audience and intended users, more languages should be added. 7. Google Analytics reveal that tools and thought leadership are not high up on the information being accessed. Analysis and further exploration of these analytics could help the team better pinpoint what information is being accessed so it can then develop a better strategy for leveraging online and social media platforms to reach target audiences. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │15 missed opportunities to leverage the considerable expertise of the wider LWA consortium on areas and expertise like elections, supporting civil society advocacy and legitimacy, marginalized populations, and handicapped/disabled peoples. • Media Programs Component. Several KIIs expressed concern or confusion about the essential absence of a meaningful media programs strategy (the support for independent media). USAID KIIs also questioned whether SCS Global is just reaching the usual suspects or if it is getting beyond that. Likewise, USAID KIIs wanted to know if the programs as a whole reach diverse coalitions of civil society and include marginalized groups, like women. 1.c To what extent has SCS Global influenced thought leadership within the broader development and DRG community on civil society and media-related issues? How effective has SCS Global been at ensuring integration of best practices, tools, or program methods beyond their immediate sphere of influence (i.e., within other SCS Global or FHI awards)? What more could be done to have an impact? What factors facilitated or impeded this impact? Figure 9: Leader Objective—Key Deliverables and Tools It was a bit of a challenge to ascertain how SCS Global influenced the broader development and DRG community, even anecdotally. As Figure 9 above illustrates, SCS Global, under Objective 1.1, has a number of activities designed to support thought leadership and improve development practices related to supporting and strengthening CSM programs. However, KIIs shared that they did not feel that the LWA has had a significant influence on thought leadership on CSM issues and approaches. That said, the evaluation team observed common themes in the three case studies (such as, a growing percentage of youth in the population, high rates of unemployment, unequal and discriminatory gender dynamics, and the heavy hand of government with a tendency to overstep power). These types of themes and use of the Leader Objective platform should be distilled in order to probe deeper on these issues and to facilitate conversations and debates about how DRG can better address these challenges. This work would be useful and within the parameter and realm of possibility for the current make up and arrangement of the LWA. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │16 What factors facilitated or impeded this impact? • Outside of USAID in Washington, the various tools (including the NGOConnect.net website, the data dashboard for SCS Global, and the webinars) were not well known or used by KIIs both at the USAID Mission level and with local implementing partners. • Marketing, dissemination, and outreach about tools were insufficient. KIIs (mainly Partners, but also USAID staff, especially at the Mission level) demonstrated that either these LWA participants were unaware of, or had little awareness about, the range of tools available on NGOConnect.net (the main website for SCS Global) or elsewhere. USAID Washington-based DRG staff had the most familiarity with the tools and thought leadership elements. USAID Missions, of the three interviewed, had little awareness or knowledge of the various tools, NGOConnect, or specific practices/methods that the LWA offered. • KIIs with local partners, CSOs, and media indicated they look to both USAID and the local AA leader for guidance, outreach, and communication about the LWA. KIIs with local partners stated they do not differentiate the SCS Global LWA program from other USAID programming—it is all USAID to them since USAID is the funder and seen as ultimately responsible for the program. They refer often to local USAID Facebook pages or other platforms to find out information. For instance, in Zambia, Youth Lead-related posts are the most popular on the USAID/Zambia Facebook page. While this is great in many respects, it has created an impediment for FHI 360 to control the message and to disseminate information about Youth Lead. USAID/Zambia did note, however, that all messages posted on USAID platforms about Youth Lead are drafted by FHI 360 working with the USAID Development Outreach And Communication team so they actually have control on the message; in fact, they have used USAID platforms multiple times to disseminate information about Youth Lead.10 The FHI 360 local partner also has a dedicated website used to disseminate Youth Lead information. What more could be done to have an impact? • How SCS Global could better influence thought leadership within the broader development and DRG community on civil society and media-related issues poses a central question for FHI 360, the wider network of the consortium, and USAID to think about collectively and answer. There’s a need to do some serious soul searching about what democracy and governance support can do and in what form it can most effectively be delivered. This is where the use of co-creation workshops, conferences, and forums could be particularly effective. There is also a need to bring in new, creative, and fresh voices and ideas and to allow room for experimentation and even failure. − FHI 360 could easily introduce again results and findings from the Delphi study it conducted at the start of the program and use it to stimulate conversation within the consortium, USAID, and the broader DRG community. KIIs suggest this could be done via webinars, online seminars, or in-person events. − Thought leadership can take on many forms, but one of the most crucial ways can sometimes be face-to-face and in-person networking and conferencing opportunities, including finding a way to link up more broadly with USAID events and knowledge centers 10 Youth Lead—as explained in more detail in the Zambia case study in Annex A—is the Associate Award program that FHI 360 leads and manages in Zambia. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │17 that may attract significant, serious, and broad attention, analysis, and dialogue from multiple perspectives. • In addition to strengthening civil society and media support for democracy global trends that have seen continued threats to democracy and backsliding, there is a need for SCS Global to look at how it communicates with the broader DRG sector and the larger community of practitioners and academics. FHI 360 and the SCS Global LWA mechanism can tap more into academic networks to better leverage thought leaders and specialists conducting cutting-edge work related to CSM programming. • SCS Global could do more to integrate digital and web-based communication and delivery of knowledge products, outreach, and overall thought leadership. The key website for the program, NGOConnect.net, has a great deal of material, but it is difficult to understand FHI 360’s target audience. The site has the look and feel of a warehouse of information, but there is little to orient the user on how best to use it. 1.d Have USAID missions’ and DRG priorities and objectives in country-level programming been consistently and accurately reflected in the awarded programs? If not, what could be done to strengthen alignment? DRG priorities and objectives have been accurately reflected to a great extent in the programming and the shaping of the AAs. The biggest improvement could be made in enhanced use and uptake of the broader consortium of the LWA and to lessen the domination of technical leadership and related LWA level of effort by FHI 360. Essentially the design, implementation, and delivery of the AAs have been closely controlled by FHI 360. Due to the evaluation design, the specific information on this question is largely based on the three African AA case studies selected for this evaluation. To get a more accurate picture and comprehensive assessment, additional input is needed from all USAID Missions where AAs have been awarded as part of SCS Global. The input and opinion of DRG Center staff, however, does feature prominently in the evaluation. It is important to note that KIIs generally agreed that the AAs are going well and delivering on what they were intended and designed to do; KIIs with USAID Mission staff said they are largely satisfied with AA progress. Zambia: The Youth Lead AA reflects both Mission and DRG Center priorities. It is a clear example of how the broad nature of SCS Global allows USAID Missions to find a home for programs that may not otherwise easily fit within USAID programming. As USAID/Zambia shared with the evaluation team, “The crosscutting nature of youth programming is something that all USAID mission technical offices must contribute to according to the mission CDCS (Country Development Cooperation Strategy). SCS provided an opportunity for the DRG team to use an existing mechanism and fund a youth program that would otherwise have been a cross-sectoral approach.” The program is a comparatively modest financial investment for the Mission but has potential to scale up and replicate within Zambia and other countries as an innovative way of moving beyond the typical capacity-building and training models frequently used by CSO and media programs. Kenya: According to feedback from KIIs, FHI 360 and the SADES-K program could have done more to better involve some of the key members of the SGS Global Consortium, especially ICNL, Internews, and National Democratic Institute (NDI)—especially as it relates to leveraging their expertise and experience in programs similar to the Building Bridges Initiative (BBI). KIIs also suggested that this would have improved program outcomes as well as improved the process of program design to ensure that it was Kenyan owned, managed, and led. The SADES-K program, perhaps more than the other programs profiled in this report, shows that the main problem of SCS CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │18 Global is that it has not optimally used what the Consortium partners can offer effectively: “FHI 360, in retrospect, could have been more than grants managers. They needed to hone in on a skill and nurture local organizations. They haven’t quite mastered this.” (KII, USAID) Djibouti: The USAID Mission felt the Programme de Renforcement des Capacités des Associations de Djibouti (PRECAD, Djibouti Associations Capacity Building Program) AA implementation was achieving progress on its planned results, especially with the use of the WSR approach. Through the use of that approach, a co-creation tool developed and used by FHI 360, PRECAD was able to harness the robust and holistic nature of this tool’s methodology to bring together a wide range of civic associations, government representatives, USAID, and other key actors. 1.e How do the case study results speak to similarities and differences between country programming in terms of meeting objectives? What are the notable innovations, including co￾creation and engaging local partners, examined through the case studies? Is there evidence of CLA within country programming? Is there evidence of learning, for example sharing of lessons and resources, across Mission programming? Each of the case studies points to the importance of strong technical and managerial Chief of Party (CoP) leadership. They also illustrate that, with the right team and/or supportive network of partners in place, each USAID Mission can achieve results through its allocation of funding and support to CSM programs. All three cases shared collaborative learning characteristics based on M&E and adaptations or modifications of the program in light of political realities and the feasibility of implementing the type of program that USAID had originally envisioned. Co-creation appeared to have the strongest impact in Djibouti, covering program design, influence, and buy-in of local actors and opening up pathways and bridges to support civic engagement. We do not have much evidence supporting the sharing of lessons learned across Mission programming, but there is evidence of sharing of some of the tools that were developed through SCS Global; notably, the M&E tools, the Youth Tool, and the Philanthropy Tool were frequently mentioned. Co-creation had a mixed reception in terms of its overall value and contribution in Kenya according to KIIs. Co-creation participants, who received funding from SADES-K expressed appreciation for the process. Those who did not receive funding were highly critical of the process and pointed out flaws in how it was organized, leveraged, and used. Insofar as it was used in Youth Lead/Zambia, KIIs did not report that co-creation was a major factor in the program’s success, attributing the program’s success rather to the outgoing CoP’s leadership and his determination, perseverance, and outgoing nature in relationship building, mentoring, and working collaboratively with a wide range of institutions. Additional similarities between the programs were that they were all FHI 360 led and managed. Unfortunately, according to KIIs, each program did not effectively use or leverage the SCS Global Consortium in an optimal way. USAID KIIs interviewed cited the lack of participation of a wider range of LWA consortium partners as an unfortunate missed opportunity. The USAID Kenya Mission had anticipated that at least three other organizations in the LWA Consortium would have been the primary interlocutors on the ground, working with and through civil society and media in￾country. They had pursued use of SCS Global because it was more feasible, given that they were under pressure to use obligated funds. When they saw them listed as part of the LWA resources, they were encouraged to buy into it. In the end, KII’s said FHI 360 served as the lead on almost all the activities, and they were disappointed that local organizations did not have an opportunity to serve as the lead on the program more directly. Specific findings below cover the case studies: CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │19 Djibouti: The WSR methodology and workshop was the real standout accomplishment of the program to date. It enabled trust-building between the government, partners, and key players; supported a participatory process and buy-in; and created an enabling environment to implement the program that has produced goods that could pay dividends in the future. All parties—government, CSOs, USAID, and FHI 360—reported learning from, and an appreciation for, the WSR method. Kenya: Kenya’s SADES-K AA is an example of one form of co-creation and how different approaches may work better or less effectively at different times and for different purposes. Its insights and lessons about the co-creation process provide considerations for donors and implementers of DRG programming on which type of co-creation process to pursue and how to sequence it. Zambia: The Zambia program Youth Lead did not use an overt form of co-creation,11 but it did work closely with the local USAID Mission and with the National Youth Development Council (part of the Ministry of Youth) to refine and improve the program. The nature of the Youth Lead program design and the management style of FHI 360’s oversight of the program overall were complementary with the general principles of co-creation since they involved local CSOs, the private sector, other traditional USAID implementing partners, local experts, and Zambia’s government. The overarching design of Youth Lead was a clear example of Collaboration, Learning, and Adaptation (CLA). One of the chief reasons for its success was the management of the program, principally in the leadership of FHI 360’s CoP and his overall approach and work ethic. The Youth Lead case study is important because it shows the importance of management to the co-creation process, and importance of interpersonal communication, leadership skills, diplomacy, and the ability of the program implementer to establish and pursue direct and ongoing feedback with beneficiaries and partners. 1.f How has the Leader award (as opposed to AAs under the Leader) been used to achieve related program objectives and results (i.e., Leader Objective 1.1 and 1.2)? KIIs were less enthusiastic about the Leader award aspect of SCS Global in terms of Leader Objectives 1.1. The main challenge is how a major LWA with nearly all the funding going to the AAs and management of these awards should meaningfully bridge and make use of the knowledge management and thought leadership portion of the LWA’s Leader Award. This is not to say that FHI 360’s handling of the LWA did not contribute to or enrich the CSM space with the tools, methods, and approaches that it offered, but that, for many reasons, the products and services that were delivered as part of SCS Global seemed to fall short or miss the mark in terms of what USAID had in mind when it developed and designed the SCS Global activity. NORC’s evaluation team discovered through KIIs that the need for overall high-level thinking identified in SCS Global activity design continues—it has only been partially met. SCS Global’s design needs updating and refinement, followed by a reorganization of activities, services, and management. There also potentially is a need to improve the leadership, networking, and communication so as to bring it to life in the way that it was imagined. 11 Note: during Phase 1 of the Zambia Youth Lead program, FHI 360 reported that Co-Creation and Partner Selection occurred during months 1–4 (October 2018–January 2019), during which time Youth Lead activities were co-created with USAID and a local Advisory board. Compared to other types of co-creation that SCS Global supported, this was not considered by the evaluation team as “strictly speaking” co-creation. See USAID Discussion Note on Co-Creation for the parameters on co-creation that the NORC team used throughout the evaluation process: https://usaidlearninglab.org/sites/default/files/resource/files/co_creation_discussion_note_august_13_2017_final .pdf CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │20 1.g Are there programming needs in the area of CSM that are not effectively addressed through the SCS Global LWA? Based on feedback from the KIIs, media programming seems to have been given short shrift. Moreover, the absence or underutilization of the wide range of skills and expertise of the varied set of skills from the Consortium needs to be addressed as each distinctive partner’s potential contributions need to be better assessed and evaluated—and utilized in the most appropriate manner during the remainder of the life of the LWA. This was particularly noticeable in terms of the types of skills and expertise of the civil society and human rights members of the consortium as well as the media and information-related representatives. In this regard, USAID Missions and local actors or direct beneficiaries/partners of the award could be better served if they had a bigger say in how the Consortium is to be utilized in the time remaining. EVALUATION QUESTION 2 EQ2—WERE THE DESIGN AND RESOURCE ALLOCATION WITHIN SCS GLOBAL LWA CONDUCIVE TO USAID’S NEEDS IN CMS PROGRAMMING? Based on feedback received from USAID staff, the evaluation team found the design and resource allocation under the Leader Objective to be not optimal, considering USAID’s needs. Resource allocation from FHI 360 fell short of expectations to produce the desired CSM learning. According to KIIs, FHI 360 has failed to get out of its comfort zone, i.e. it is a very effective grants and program manager, but less inclined to develop forward thinking, new knowledge and thought leadership products. It is worth noting that some experimentation was done but it could have gone further. SCS Global has been generally well-managed in organizing and implementing AAs, yet it did not seem to take the risks in pursuit of thought leadership with associated funding that would have resulted in USAID’s desired innovation in CSM programming. 2.a Has the focus on thought/technical leadership and capacity building been consistent with USAID’s original vision? How have each of these sub-focus programs been implemented to date? The SCS Global activity has produced mixed results in this area. As the leader, FHI 360 has partially demonstrated the kind of thought and technical leadership and capacity that was consonant with the original vision. However, outcomes were inconsistent. KIIs with USAID revealed several ways in which the outcomes felt uninspired, even though FHI 360 experimented with a number of different approaches starting with the Delphi process and continuing on through hosting CoP roundtables as well as cross-mechanism M&E discussions and other events. After the Delphi process (which had mixed reviews but is a good method that merits further analysis and understanding), five sub-focus areas were identified as priorities: 1) responding to propaganda and press repression; 2) operating in environments where governments are hostile to civil society; 3) technology, social media, citizen journalism, and media literacy; 4) informal organizations and online activism; and 5) community resilience to violent extremism. If the point was to bring in the ideas of the countries where USAID works, they should have brought in more people to the Delphi process from those countries. USAID KIIs were critical of how the program dealt with the Delphi Panel outcomes, for example, which provided many insights in setting the foundation for the program design—program design around thought leadership, DRG tool development, and so on. But, as noted by USAID, “The Delphi was a good effort in helping to identify what are the key issues in the field, but it was not successful in moving forward the process of advancing research.” (USAID, KII) CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │21 There was a marked difference in perspectives between USAID Washington and Mission field staff around the extent to which the outcomes were consistent. Many field staff compared the outcomes to Mission objectives sometimes subject to rapidly changing country situations and local needs. For instance, the Kenya program was intended to support the BBI, which is subject to a variety of ongoing local issues and political contexts that the program must be prepared to work around. USAID Mission staff wanted to evaluate the outcomes and the impact of SCS Global’s SADES-K relative to how FHI 360 supported the overall objectives of BBI and how it strengthened local civil society. In contrast, KIIs from USAID Washington DRG staff were looking for a higher level of thought leadership from the SCS Global LWA as a whole and had anticipated that there would be more of a feedback loop emanating from the AAs to create new knowledge and understanding in the wider DRG sector. 2.b Was the allocation of these resources and prioritization of these resources appropriate? Feedback from KIIs suggests that allocation and prioritization of resources was not wholly appropriate to the needs identified in the SCS Global LWA design. One USAID KII said of this, “We would expect more.” It is hard to verify this finding, however, since the evaluation only included three country cases of AAs, and the evaluation team did not have the full program budget for SCS Global or undertake a specific cost-benefit analysis that would be useful in answering this question. Most of the funding allocated for the LWA was used for the AAs, and $500,000 per year went to FHI 360 for the Leader Objectives related to learning, knowledge management, and thought leadership. Given the size and scope of FHI 360’s agreement and budget, this amount was, perhaps, not very significant for them, but it would have been significant for a smaller organization. 2.c How has the SCS LWA dealt with emerging needs and other priorities not anticipated during the program design? Providing a mechanism for identifying emerging needs and priorities did not seem to have factored much into the program’s overall approach. KIIs did not really address or comment on whether the SCS Global LWA dealt with emerging needs or priorities, perhaps because so much emphasis focuses on the management and implementation of the AAs. For instance, the LWA’s 21-member consortium is not really leveraged in a way that can tap into their considerable expertise to solicit feedback and suggestions related to emerging needs. The flagging of emerging needs and the ability of the Leader and the Associates to communicate emerging needs as they arise in the CSM space may be a useful area to follow up on in the next phase of this project, especially as political and social events of the day may have an impact on the focus of the program. That said, as demonstrated by the three case studies, there are always unanticipated emerging needs and priorities. The evaluation team observed that FHI 360 found ways to respond to these and work with the Missions to develop the programs accordingly. This dynamic was notable in both Djibouti and Zambia, where the local AA leadership had to reorient the programs based on what was originally conceived and made use of principles of Collaborating, Learning, and Adapting (CLA) practices. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │22 2.d What are the strengths and weaknesses of the organization and management structures of the current LWA mechanism? Overall strengths of FHI 360, noted in many KIIs, indicated that it provides good guidance and management to the LWA Mechanism. KIIs stressed its strengths as: great at managing associate awards, responsive to USAID Missions and the scope of the AAs, wonderful partner to work with, and good internal M&E. The main weaknesses expressed during KIIswere the lack of turning some of the lessons and priorities from the program into forms useful for other CSOs and USAID to learn from; program partners could have been more actively engaged to help tailor learning products drawing from the significant data produced by FHI 360’s M&E team. Other weaknesses included: • Poor strategic communications in reporting: “It’s just a lot of fluff and spin.” (USAID KII) The KII went on to say they were looking for more substantive, clear, and easy-to-follow data and insights useful to DRG programming. The reports and information shared by FHI 360 are often overwhelming and hard to distill and therefore are not helpful to the DRG Center in advocating for democracy and governance programming. It is hard for outsiders to FHI 360 to “see the forest through the trees.” The larger issues and key outcomes and achievements that FHI 360 and the other partners are making gets lost in reports and communiques that are laden with details and a reporting style that overwhelms the reader with lists of outputs and achievements. • Another key weakness of the organization and management structure is that the overall communication and representation of the program are all channeled through FHI 360 and represented by FHI 360, meaning that the wider consortium may not have a voice or get to communicate about the LWA and their role in it, or opinions about it, to USAID. KIIs from USAID noted that they did not find SCS Global newsletters helpful. • The current organization and management is very Washington, D.C., based, meaning that most of the key decision-makers are infused with a Washington, D.C., or “inside the Beltway” mentality; this was flagged by USAID Missions and local offices as a barrier that needs to be addressed and overcome. 2.e Are the consortium structure and arrangements providing maximum benefit and value for USAID? The consortium structure and current arrangements do not provide maximum benefit to USAID. KIIs with USAID staff, while clear about the program’s relevance, indicate the program is not producing maximum benefit in providing lessons that are helpful in shaping effective DRG programs across the world. While FHI 360 assembled a strong group of partners, one USAID representative noted, “We were excited about the partners they assembled, but they never tapped into them, and we are not sure why. It just didn’t happen.” (KII USAID) This view was also shared by one USAID mission staff who commented, “I haven’t seen or felt Internews. Where is Internews in this process?” (KII USAID) Another USAID representative questioned, “To what extent did different partners engage? What was their take? I have a feeling a smaller group was involved. I’ve heard some rumblings; they didn’t get a chance to engage much. That could have been done better.… It wasn’t spread out in the Consortium—not sure why not.” (KII USAID) It is worth noting that some consortium partners were not used at all. In Kenya, Djibouti, and Zambia, the evaluation team noted—and KIIs confirmed—that there were plenty of opportunities where they could have engaged more of the consortium partners to work CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │23 with local groups. In Djibouti, for example, a good entry point into civil society are groups working on disabled people’s issues, and one of the consortium partners could have been an asset to the program. Similarly, the expertise of ICNL may have been useful during the WSR workshop. In Kenya, ICNL, Internews, and NDI could have all been tapped for larger roles and involvement in the design and implementation of the program. 2.f Are key consortium partners, and sub-partners, including regional partners, used effectively and their relative expertise made available to USAID? KIIs indicate that partners have not been used effectively nor has their relative expertise been readily available to USAID through this initiative. Part of this has to do with the nature of the LWA system. As one USAID KII said referring to the program communications with USAID, “It is window dressing.” (KII USAID) Another USAID KII commented, “We were excited about the partners, the regional partners, and the networks. But FHI360 did not tap into them, and we are not sure why not. It just didn’t happen.” (USAID KII) As the leader of the SCS Global Consortium, FHI 360 is missing this opportunity to be a significant connector and facilitator. As shared by a USAID KII, “The Leader said they would make it all connected and reinforcing. They are the force for the connectivity of all the efforts. The leader as the leader should be doing this. At the moment, their efforts are a little stove-piped.” (USAID KII) 2.g Is communication between USAID and the consortium facilitated by the management structure—both within the consortium and within DRG? Communication is a core area for improvement. Communication is on two levels: (1) interpersonal interaction between USAID and the consortium; and (2) reporting. KIIs noted strong and effective interpersonal interaction with USAID Washington, D.C., staff, but highlighted that reporting is an issue. While M&E is clearly an FHI 360 strength, USAID KIIs said the length and substance of FHI 360’s project reports overwhelms them and misses an opportunity to effectively report and share the types of lessons they need. 2.h Has the consortium’s management of the current LWA, including division of programming, communications with USAID and external audiences, and reporting and other obligations proven effective—why or why not? The FHI 360 Consortium does not seem to be optimally leveraged or used for programming (most of the programs are led, managed, and designed by FHI 360), and communications with USAID and external audiences could improve. This question is hard to answer based on the research design for this evaluation, in which there was limited fieldwork in only three of the 18 program countries. By and large, KIIs with USAID were universally appreciated and satisfied with FHI 360’s management of the LWAs. KII’s with LWA Associate organizations suggest that communication could be more proactive, and the consortium could be tighter in terms of sharing and collaborating on CSM initiatives: “I am not always clear on what the Consortium as a whole is doing. I can imagine based on our own experience what people do not get. They do not know the achievements and the like. I get random emails.” (KII, USAID) “My experience with this consortium is that it is reactive. FHI 360 comes to us, but also the Mission goes to FHI 360. The way the Consortium works is that they (FHI 360 and USAID) have a definite idea of what they want to do.” (KII, ICNL) CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │24 USAID KIIs shared a similar concern about the division of programming, especially in terms of the use and uptake of the consortium. KIIs with USAID staff expressed a concern that there have been few events and other engagements with external audiences, although the limited number that FHI facilitated were appreciated. Another issue expressed in some KIIs is that a lot of program communication is heavily directed at Washington, D.C.-based audiences, and is not strategically directed to intended program beneficiaries in the Global South. 2.i Is USAID DRG responsive to the management needs of the SCS LWA, including substantial involvement? Based on feedback the evaluation team received from FHI 360, it seems that they feel DRG is responsive: “On all this thought leadership and tool development, it is a very iterative process. There’s a difference between the AA and the Leader Award. On the AA we get explicit direction.… On the Leader Award side, decisions on thought leadership, these are made with USAID and DRG colleagues. The cadre of DRG folks, they have become an informal Steering Committee. The AOR brings these people together when they want to do something. In the second year of the program, when FHI 360 put together the proposals for the Research and Learning Agenda, DRG came together to provide feedback on the proposals. Also, on the Youth Programming Assessment Tool, when they set up a youth-focused working group at USAID; the AOR brought them together. (Group interview, FHI 360) USAID DRG staff were more critical of their own shortcomings, citing that there had been several changes in leadership and management of the LWA and that due to their own bandwidth issues they may have not always been as responsive as the LWA needed. On the other hand, several USAID KIIs felt they had overstepped their role in managing the LWA, and frequently had to get overly involved and micromanage different processes. Additionally, both Washington, D.C., and Mission￾based USAID staff cited that frequent delays and time lags obstructed the desired outcomes of SCS Global. 2.j How effective has SCS Global been at integrating technical leadership developed under the Leader into the design and implementation of Associate Awards? The SCS Global program has not been very effective at integrating technical leadership from the LWA into the design and implementation of AAs. USAID was looking for SCS Global to provide more technical leadership and to stimulate key lessons and directions for USAID CSM programming itself, but does not feel this has been accomplished, except for a few areas (Local Philanthropy and the YPAT). One USAID KII noted, “Individuals have been great to work with. They are attentive and have good ideas. But we are looking for more technical leadership.” The desire for technical leadership that can strengthen USAID’s DRG programming is echoed in many of the USAID KIIs, “A lot of the program identifies the problem but not the response…. We don’t know what we don’t know.” The need for this technical leadership is also important to designing and understanding CSM trends across mission countries: “The priority of USAID missions is the Associate Awards. We have a responsibility to the broader thinking and the processing of information and then synthesizing it.” Anecdotal evidence from FHI 360 as presented and shared with the evaluation team indicates that SCS Global invests a lot of time, money, and effort into integrating technical leadership into the AAs. Based on the Evaluation Team’s site visits to the three local offices where SCS Global CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │25 supports the AAs, there is reason to believe that the technical leadership has informed local (FHI 360-led) thinking and approaches to project-level M&E and learning. EVALUATION QUESTION 3 EQ3—HOW HAS THE SCS GLOBAL LWA UTILIZED CO-CREATION DURING THE DEVELOPMENT AND IMPLEMENTATION OF ASSOCIATE AWARDS? Results of the co-creation workshops were shared by FHI 360 in interviews and presentations for this performance evaluation. According to FHI 360, all programs have had co-creation or CLA as part of the program design, implementation phase, or learning component of the program. Anecdotally, according to FHI 360, the efforts to co-create have been beneficial, but the evidence to support this is not always clear or captured in a way that would be useful for program monitoring or evaluation purposes. Moreover, during the Evaluation Team’s KIIs with USAID staff, it was not clear that USAID or other donors have developed a set of criteria for how to evaluate the success of a co-creation workshop or similar type of process. As one USAID representative shared, “It’s not necessarily the case that co-creation was intended to be an activity that would have a monitoring component.” Nonetheless, as much of this LWA and other DRG sector work is now expected to utilize co-creation as a process, it would be useful to have evaluation criteria to better make assessments about the benefit or lack thereof of the co-creation process. SCS Global LWA utilized co-creation in three different ways: 1) during pre-award development of award design and content; 2) during post-award development of collaborative objectives and work plans; and, 3) during the implementation phase development of program activities. In evaluating co￾creation overall, KIIs gave a mixed response about whether or not the mechanism was useful. KIIs were happy that the process was being carried outside of USAID, as it gave space to co-create with sub-awardees, but some felt co-creation represented a large cost in terms of time and institutional energy that was difficult for their organizations to bear. Figure 10: Useful Parameters for Co-Creation According to one USAID representative, while there are currently no indicators or evaluative framework to assess co-creation, there are some useful parameters to keep in mind: 1) Co-creation doesn’t have to be led by USAID. It’s helpful if it can be led by the Prime Contractor and done with the subs and local partners. The whole value of co-creation coheres when you have stakeholders coming together to create a solution to a problem. 2) Collaboration is a joint endeavor. All partners should work together to come up with solutions to problems and consider what stakeholders are doing to address the problem. 3) Co-creation that involves stakeholders who may not be part of the awards package are some of the most important people to involve in a co-creation workshop. 4) Co-creation is not an approach to procurement. Co-creation is best when it is part of the intervention footprint in the technical thinking of the overall program. It cannot and should not just be part of a procurement process, i.e., part of the grants and contracts. 5) The real value of co-creation is treating it to engage different stakeholders—not about figuring out how, who, and what to fund programs. The framing of co-creation events matters. It needs to be tied to the work, linkages, and connections and disparate groups. Interesting results were found at many points in the evaluation process: 1) co-creation in Kenya led to new coalitions being formed for advocacy work, such as by the Kenyan Media Council; 2) co￾creation as used in Djibouti brought together government, civic associations, and FHI 360 in such a CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │26 way that would not have been possible—the whole program could have been jeopardized without the WSR workshop; and 3) co-creation as used in Zambia enabled the CoP to figure out how to manage Youth Lead and run a successful program. 3.a What are the strengths and weaknesses of using a co-creation process with Associate Awards? KIIs from all groups (USAID, Associates, and partners) commented that the co-creation process included significant strengths: • Co-creation is an important tool in the toolbox and is “just good development practice.” • Co-creation provides a space for programs to be designed outside of the halls of the donors and shows success in shifting some resources to the implementing organizations. • Co-creation works well with USAID’s Journey to Self-Reliance (JTSR) initiative. • Co-creation inspires technical thinking as it relates to in-country work and priorities. • Co-creation requires gathering insight from new stakeholders and not the usual suspects. • The value of co-creation coheres when stakeholders come together to create solutions to a problem. • With co-creation the need to manage expectations was frequently cited as key: − “Expectations and processes by FHI 360/USAID should have been clearly laid out.” (KII, Kenya) − “With co-creation, there are a lot of conversations going on, which sometimes can lead to frustrations about expectations.” (KII DRG Implementer/other LWA) − “They knew what they wanted. They should have stepped back. What’s co-creation, and what does it mean in our context here? The problem was with expectations. The rollout should have gone better.” (KII, Kenya) • Co-creation was not productive as a way of raising program funds. CSOs and media in the countries where USAID works need funding and often look to events and workshops, like the co-creation activities, to help improve their business development plans and bring in grants or other revenue. For many KIIs, if the pre-award form of co-creation does not lead to funding, it is not the best use of their time. In this sense, a post-award co-creation is a preferable model, as their guarantee of funding is there, and it is also easier for them to share ideas and contribute to problem solving. • Co-creation can lead to feelings of unease in terms of the donor power dynamic. KIIs from local partners indicated that the local organizations view FHI 360 as a donor, not a partner, and that this dynamic made it difficult to participate in an unguarded and open manner. • USAID procurement procedures have not integrated co-creation workshops well. Some felt the co-creation process was directed by the procurement staff or viewed as a developmental intervention. They felt the process cannot just be directed by the people in charge of grants and CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │27 contracts. The value, they said, is realized when the process is well integrated into the intervention footprint, not just in the technical thinking and project design. • KIIs felt FHI 360 was new to co-creation and did not have the relations with them it needed before the workshops. In Kenya, for example, FHI 360 did not have in-country staff that were viewed as part of the work these organizations were doing: “The challenge for FHI 360 is that they don’t understand the co-creation process. They should have given out materials about co-creation process beforehand. There needed to be more preparation. There were no preparation meetings, there was no understanding of the co-creation methodology. We needed to spend time talking about joint planning and really think it through.” (KII, Kenya) • KIIs said it led to some exacerbated tensions where stakeholders felt left out or did not see follow-up from their efforts in the co-creation process. They said, in practice, co-creation appears to make some people feel cut out of the process or directed by these one-time events. • Other KIIs said that the workshops seemed to be a more extensive consultation than a creative design process. Depending on how it is framed, KIIs said the co-creation process must be better tied to the work, resulting in strengthened linkages and connections with disparate groups. They see a real value from co-creation in engaging different stakeholders and not about figuring out how the design will be funded. Managing all of this input, however, is also a complex task. • KIIs cited the costs in terms of time and organizational resources and questioned FHI 360’s approach. KIIs pointed to an insufficient understanding of co-creation as a concept among many stakeholders as well: “There was a real attempt in working with partners to learn what they wanted the project to look like. Broadly speaking, I do not think that people understood the concept. For a lot of partners, it was the traditional way of looking at development. The people who came to the workshop had the sentiment, ‘so they were putting into the project what they wanted all along’” (KII, Kenya). • There is a need to be sensitive of the time commitment and level of effort you ask of co￾creation participants. In Kenya, for example, the co-creation workshops lasted three days each. This was a huge investment of time that has a trade-off cost for all the participants. 3.b What are the effects of different forms of co-creation (pre-award or post-award), and do different forms have related or distinct strengths and weaknesses? The different forms of pre-award and post-award both have different merits and effects. In either case, it is all about managing expectations, having a strong facilitator (preferably an impartial outsider), and making sure participants are well informed and clear on the concepts and the point of the co-creation experience; their buy-in is essential. There is a time and place for both forms. It is not just about the positive and negative aspects of co-creation, but timing and the form used can determine when it is a productive brainstorming and design mechanism. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │28 Figure 11: Strengths and Weaknesses of Pre-Award and Post-Award Co￾Creation STRENGTHS OF PRE-AWARD CO￾CREATION WEAKNESSES OF PRE-AWARD CO￾CREATION • Brings in a wider range of stakeholders, including partners that will get no financial gain from the program (such as members of the government or donor partners) • Outcomes and learning are community driven • Participants expanded partnerships beyond the usual suspects • Difficult to manage expectations for the participants who seek to get financing out of the workshop • Level of investment of participants has to be commensurate with the benefits • Dependent on having the right people in the room to build trust and share the right expertise STRENGTHS OF POST-AWARD CO￾CREATION WEAKNESSES OF POST-AWARD CO￾CREATION • Already has commitment of many participants who know what the financing will be and so have the right people at the table • Clear follow-up can be assigned, oversight possible • Might be stuck with partners or people that you do not really want • Driven by the project not the overall goal to solve the problem Some KIIs said that whether co-creation is done before or after the awards is not important. More significant factors, they felt, include having the strong facilitation, clear overall goals, and the right timing. They said more communication, follow-up, and continuity before and after the workshop are essential elements to co-creation efforts. As one KII said, “The FHI 360 workshop was organized as a one-off event with no follow-up afterwards with participants. However, co-creation is actually part of a process in which many of the participants are already engaged together to resolve an issue, and it should be designed with this in mind; the outcomes are not one project but one of many better and more aligned initiatives. As a one-off event, the program benefits to participants are minimized.” (KII, Local Partner) Some specific issues raised during KIIs are in Figure 11, highlighting strengths and weaknesses for both the pre- and post-awards approaches. 3.c How does the current SCS Global consortium structure affect or relate to co-creation processes? The evaluation team did not find any evidence that the consortium was engaged significantly or utilized in any meaningful way in the co-creation processes in the three case study countries visited. General observations are limited by this scope. According to one USAID KII, “FHI 360 has done eight co-creation workshops including some notable ones in Cambodia, Djibouti, Kenya, Jamaica, and Thailand, and there may be more lessons about the consortium structure to be gleaned from these experiences that were outside the scope of this evaluation.” EVALUATION QUESTION 4 EQ4—HOW UNIQUE IS SCS GLOBAL FROM SIMILAR DRG MECHANISMS? SCS Global is unique because it gives USAID missions a way to easily and quickly program money in support of approaches Missions have identified as having the potential to strengthen civil society and CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │29 media in support of DRG and other development results in open and politically restrictive environments—programming that may not be overly related to elections support or strictly and overtly related to human rights promotion and advocacy. There are few other mechanisms within DRG devoted specifically to civil society and media that USAID programs can readily tap into and that allow them the kind of mechanism that SCS Global offers. 4.a To what extent are the scope and capabilities of SCS Global significantly different from those available under similar DRG LWAs, such as the Human Rights Support Mechanism (HRSM)? The scope and capabilities of the SCS Global LWA program fill a significant niche because it is wider and does not have to have such a specific focus on human rights or a specific development sector. As compared to HSRM programming, for example, which focuses exclusively on human rights, SCS Global can reach a broader range of actors from multiple sectors (education, health, etc.) with a more neutral tone—i.e., SCS Global doesn’t overtly claim to be about human rights promotion or electoral reform, which for some countries and programs can be problematic or a non-starter. It can work, for example, in Djibouti where a human rights program such as HSRM would be more restricted. This program is wide enough that most missions can find a way to work with it in terms of DRG programming. LWAs focused on human rights or elections are much narrower in scope. This can help to engage missions more directly in DRG work and make the link to other mission priorities like WASH, education, or health. “USAID has had a choice of how political they want to get. They do not have to take a political approach. It was broad enough to take on the apolitical human rights work and services. It is not inherently political. HRSM ratchets up the political nature by virtue of being human rights. There has been some reticence of working with it because it is so in-your-face political. It is useful for USAID to have both awards, as there has to be some discussion in terms of what approach is better to take and more appropriate.” (KII, DRG program implementer/LWA member) “We needed a new programming mechanism at USAID—we built it rather broadly: 1) about environment building—normative frameworks, capacity of human rights defenders, and rights at the country level; and 2) how do we respond—when rights are on-going—are there things we can do to mitigate impact?” (KII, USAID) Each of the LWA mechanisms has a specific purpose and function, and each is based on a specific set of strengths of the consortia and programmatic needs of USAID given the time, place, and context of DRG-related programming. The strength of SCS Global is that it is broad enough to allow for interpretation and has a very flexible design to be used for a variety of DRG programming needs, whereas the elections LWA is specific to elections and the human rights LWA is specific to human rights. “In places where governments are allergic to human rights, this one (SCS Global) might be more palatable than overt human rights and easier to sell. Working with CSOs is easier than human rights.” (KII, DRG program implementer/LWA member) 4.b Do there appear to be unique capabilities or areas of focus under SCS Global that are not available under other DRG LWAs, such as HRSM or the Global Elections and Political Transitions (GEPT) LWA? CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │30 Yes, there are unique capabilities that others preclude (see above answers as part of EQ4). SCS Global is an important mechanism for USAID Missions to buy into when they want to have programming that is not overtly political, human rights centered but is also building stronger cross￾sectoral linkages. In thinking about the LWAs comparatively, they have talked about this with USAID and technical directors of other LWAs: “This mechanism—Freedom House overall, is one of the oldest democracy non-governmental organizations (NGOs) in the U.S. We basically have a mission to support democracy around the world— to fight and push back against authoritarianism: 1) research—freedom of the world and net (based out of NY), 2) advocacy—working with Hill and Executive Agencies—and 3) on the ground programs.” (KII DRG program implementer/LWA) CONCLUSIONS How has the SCS Global LWA been effective in advancing USAID priorities in programming? 1. The SCS Global LWA has been effective in advancing USAID priorities in CSM programming chiefly because it has provided a useful mechanism to manage the AAs and enables both the DRG Center and local USAID Missions to support and bring the CSM programming to life. As the Prime, FHI 360 is most effective with its implementation of Objective 1.2 and the management of the AAs. Through SCS Global, USAID￾supported programs that might otherwise not have happened due to procurement rules and issues and contractual regulations restricting what a USAID Mission can support. SCS Global provides a simple mechanism for USAID Missions to provide democracy and governance support to CSOs and to media outlets without the often expensive and labor￾intensive process that an openly competitively bid cooperative agreement or contract might involve. In short, it presents a useful and straightforward way for USAID Missions to program their funding in support of CSM objectives. Moreover, the LWA has grown in reputation within USAID and has become a known mechanism that USAID Missions tap into if they are interested in supporting CSM and DRG-related programming. In this regard, the SCS Global LWA is an efficient mechanism for USAID programs to champion CSM programming within the priorities and assessments of local USAID Missions. 2. Under FHI 360’s leadership, the SCS Global LWA has been less effective with the fulfilment of the activities outlined under Objective 1.1. The Evaluation Team noted that a top-down management structure and heavy-handed Washington-D.C. based operations impeded the vision that USAID had for optimal delivery of Objective 1.1. 3. The program has built a solid foundation upon which FHI 360, USAID, and the wider SCS Global consortium can build. Demand for innovative ways to impart knowledge and best practices and how LWAs can help advance learning and evidenced-based programming around civil society and media are paramount to today’s DRG sector and international development programming. As we live in digitally driven world and development practitioners have come to expect sharp and useful websites and mobile applications for their everyday work, websites like NGOConnect.net and the online learning and other digital outreach potential of this LWA have a lot of promise but need to be tactically designed and marketed in order to appeal to intended audiences and to achieve their desired effect. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │31 Were the design and resource allocation within the SCS Global LWA conducive to USAID’s needs in CSM programming? 1. The design and resource allocation of SCS Global, as envisioned and developed by USAID, offered a fresh approach to civil society and media support. It encouraged and funded co￾creation, collaborative learning, and adaptation, as well as robust monitoring, evaluation, research, and learning that are all essential to the overall process of CSM program design and delivery. Many aspects of SCS Global appear to be performing well and, based on the three countries visited for the case study fieldwork, they are meeting the needs and expectations of the USAID Missions. 2. Leader Objective 1.1, which includes thought leadership through the development and dissemination of new toolkits, approaches and methodologies to support CSM programming, fell short of expectations to provide significant innovation and learning. Part of this is due to the current design and overall management of the LWA, with FHI 360 taking the lead on most aspects of the program and with little opportunity for consortium members to participate. Other observations on Objective 1.1 are: • Thought leadership products have not been used significantly. It is not clear, for all the effort that FHI 360 has put into Objective 1.1., that the deliverables and outputs were useful to the wider CSM/DRG community or that they are improving the delivery of USAID/DRG programming. • There is a disconnect between the program implementation and its Theory of Change. The design of the program does not yet deliver effective strengthening of civil society globally, at least not in terms of Objective 1.1. • Furthermore, the current MERL is not an effective support mechanism for the program’s theory of change. Program MERL could be an effective tool to engage civil society and media more broadly through innovative and catchy modalities, such as papers, web, social media, video, and conferences. A chief shortcoming in the SCS Global LWA was a perceived imbalance of funds managed and held by FHI 360 as compared to other partners in the consortium, as well as resources available for local partners and the intended beneficiaries of the funding. The end result is that the overall program design and resource allocation, according to some KIIs, resembled an old-fashioned, top-down model with FHI 360 taking the lead on most deliverables. While the program brought together a competent staff with relevant skills and expertise, the LWA has not created a dynamic and inspiring model for 21st century CSM and DRG support. There is a need for a user-centered design approach to better allocate resources within the LWA consortium and to make use of the incredible array of local, regional, and international partners in the CSM space. Notably, in terms of resource allocation, additional thinking is needed to better assess whether the current business model of the LWA handicaps innovation, thereby crowding out innovation from below. While the co-creation approach is meant to facilitate a wider range of stakeholders and to move past the “usual suspects” approach to donor assistance of CSM and DRG programming, this has not happened. (To repeat, this evaluation only examined three of the 18 AAs and a full review could provide further evidence of what is being missed and what could be gained by a shift in LWA management paradigm.) 1. Over-focus on deliverables coming from the project Lead in Washington appeared to have contributed stymied more active participation across the consortium and network. SCS Global is not reaching optimum impact by delivering DRG programs in a truly collaborative, networked, and multidimensional way. The chief challenge CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │32 standing in the way of more collaborative programming is the funding model that the LWA offers. While it has boosted DRG programming on the whole and while USAID Missions are very interested in the overall mechanism, it has not been used effectively to enable the significant collaboration in line with the expectations and needs of USAID’s DRG Center. Such collaboration would provide improved thought leadership, learning opportunities from MEL, and support for development and dissemination of useful and timely tools and platforms. To help the broader DRG sector improve how it embraces and makes use of the evidence-based lessons and rich data collection that FHI 360’s M&E team and approach provides, more participation from the Associates and USAID staff was needed. How has the SCS Global LWA utilized co-creation during the development and implementation of Associate Awards? 2. SCS Global has used co-creation, notably both pre-award and post-award, and in program implementation to create a robust effort to support locally led development. FHI 360 has endeavored to support the co-creation process as a hallmark of SCS Global. While it has not always been perfect, the goals and intentions of supporting a locally led process have been sincere, and in each of the three countries visited, all the feedback from KIIs was indicative of the care and attention that FHI 360 has given to using co-creation. By all accounts, and in different ways, the efforts to utilize the co-creation approach were positive and have yielded a significant number of new insights, opportunities, and interesting challenges for living up to the hope (and hype) that the co-creation process can bring about. This has resulted in a need for FHI 360 and its partners to manage expectations and to come up with appropriate responses to the overall build-up or rising expectations that result from the co-creation process. 3. FHI 360 has incorporated and championed co-creation tools with inconsistent results. Results have been inconsistent across the region we researched and are not well documented or reflected upon, resulting in a lost opportunity to gather useful insights for the DRG sector and to help shape and move forward an important tool and process for creativity and design of CSM programs. It would be useful for USAID and others in the DRG space to learn more about the FHI 360 and SCS Global experience on co-creation and to use the platform and leadership role that the LWA provides to go deeper on what works and what does not and why. As an older report from the Center for International Media Assistance titled Good, But How Good: Monitoring and Evaluation of Media Assistance Projects,12 once quipped, “Nothing succeeds like success.” As the report nicely articulates, NGOs and program implementers seldom report their shortcomings, failures, or lessons learned in ways that are honest or offer a meaningful way to learn from them. This is, perhaps, one of the shortcomings of this LWA. As an example of the shortcoming, USAID asked for lessons learned and feedback about SCS Global’s use of the co-creation process. “We asked them to do this this year—we asked them to do a summary of their co-creation process—so we can have a conversation with our internal office.” (USAID, KII) USAID really only wanted one page on this in order to facilitate a conversation, as co-creation is a priority area for USAID improvements. It never received the one-pager. 12 See: https://www.cima.ned.org/wp-content/uploads/2015/01/CIMA-Monitoring_and_Evaluation-Report.pdf — “Can a multimillion-dollar industry undertake hundreds of projects aimed at tackling the intricate and seemingly endless problems facing media around the world for two decades without experiencing a single flat-out failure? Apparently so— at least in the case of the United States’ biggest media assistance organizations. In interviews, representatives of the International Center for Journalists (ICFJ), the International Research and Exchanges Board (IREX), Internews, and Search for Common Ground all acknowledge that while specific aspects of some projects had not worked out, they could not recall a project that had been evaluated as a failure.” CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │33 4. The LWA has encouraged local collaboration, learning, and adaptation of each AA. In the three case studies there was: (1) a learning curve that was needed to figure out how to bring to life the initial vision of the AA, and (2) required time and space to learn and reflect from implementation efforts or program designs that either did not work or needed adjustments. In this way, the LWA has allowed space for the AAs to grow, learn, and find their footing to be responsive to the needs of the local contexts and environments that they work in. This is the type of learning and reflection that needs to be shared, discussed, and debated as part of the LWA and the broader DRG community. Furthermore, the assessment of effectiveness of co-creation revealed some key lessons learned: • Co-creation raises CSO expectations for program development funding and support for collaborative initiatives. The mixed response that KIIs gave on whether co-creation was useful or not points to the need for organizations who choose to use co-creation to think about the timing of co-creation, managing expectations, the need for clear communication strategies with co-creation workshop attendees about what is and is not being offered, and a good plan of action for post co-creation follow-up. • Co-creation introduces and must contend with power dynamics among sectors and types of organizations that is an important consideration for USAID to recognize timing, participation, and facilitation are important factors to address these power dynamics. • There were differences of opinion between USAID’s view of co-creation, particularly when and how to use it, as compared to program implementers like FHI 360 or Internews as well as local CSOs and media. There was a mismatch between how USAID and the implementing partners saw co-creation, especially in terms of whether to do co-creation pre-award or post-award. • The time investment in terms of those who are taking part in the co-creation process was a major consideration for participants. • A number of KIIs felt that what is being labeled as co-creation is just “good development.” In other words, they felt that co-creation was just another buzzword they needed to use or incorporate with the end result being the co-creation as it was viewed by USAID, and the potential it could have for SCS Global was not fully actualized. • Future LWAs must figure out how to use members of the broader Consortium in the co￾creation process—a place where there expertise and opinions on a subject matter, program design, or “blue sky thinking” would be particularly valuable. Unfortunately, the evaluation team observed little evidence that the SCS Global Consortium was adequately involved in the co-creation events for the AAs. How unique is SCS Global from similar DRG mechanisms 1. SCS Global is a unique mechanism that allows greater flexibility to do cross sectoral programming than some other mechanisms. This enables USAID Missions to work within a broad and expansive universe of DRG and CSM programming and gives Missions a way to pilot new programs and try out lines of support that they may continue to support through an LWA type of arrangement or decide to competitively solicit through other assistance or acquisition instruments. While it is structurally similar to other DRG mechanisms, KIIs note that its flexibility and openness to a range of CSM programming and other development goals CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │34 (education, health, economic development, environment, etc.) make it a unique mechanism. KIIs perceived that—while the SCS Global Mechanism encompasses human rights—it is not overtly branded as a “human rights” program. With its more neutral tone, SCS Global can reach a broader range of actors from multiple sectors (education, health, etc.). KIIs thus felt the program was easier for some countries to buy into. RECOMMENDATIONS Our overall recommendation is for USAID to continue this global civil society strengthening initiative to build on and advance the networks, results, and products produced in the first phase. The program, despite faults in the thought leadership objective. 1.1, has built a strong foundation for the future. Some specific recommendations to enhance the opportunities for thought-leadership and collaborative programming are: 1. Plan a pause-and-reflect process with SCS Global and USAID using the same principles of co￾creation that are espoused at the country level. It is important to use the remaining program time to spearhead and ignite the type of critical thinking and decision-making needed to take SCS Global to the next level. A workshop or facilitated learning event to critically reflect on the program’s strengths and weaknesses could help with this, bringing the program lessons to bear on revising Objective 1.1. in terms of thought leadership, toolkits, and the LWA’s underlying theory of change. The use of an outside, impartial, but informed facilitator is important. The facilitator should be someone who understands DRG and USAID and can craft a meaningful learning agenda to articulate the vision, creativity, and bold ideas that USAID is looking for. FHI 360 and all members of the consortium should be invited to attend, and the facilitated learning event should use the same principles of co-creation that are espoused in the start-up of the AA in the remaining program time. This is an important time to figure out the ways forward to take what FHI 360 has developed and created and improve upon it. A learning focus might include: − A follow-up, global survey of program stakeholders on the sustainability of activities and program outcomes and/or a follow-up to the Delphi process. − Reflection on SCS Global’s thought leadership and knowledge management role. Since this is a relatively new approach to knowledge management and sharing of data and learning, it would be useful to hold a well-designed and facilitated learning workshop to advance the thought leadership component. In such a discussion, the full SCS Global consortium of partners led by FHI 360 could have a dialogue with other leaders and key influencers from related DRG LWAs, thought leaders from democracy and governance studies, and MEL specialists. The workshop could examine the tools and services on offer through SCS Global and help co-create or design a next iteration of the Leader part of the Award. − A stock taking of SCS Global experience with co-creation, in which FHI 360 can identify a set of strategic best practices drawn from its program experience that will inform future iterations of the initiative. USAID Missions are keen to get feedback and advice on co￾creation as a tool and process, and SCS Global could provide a real value added on this. − Collaborative learning mechanisms for USAID and FHI 360—along with the rest of the SCS Global Consortium—to work together in a partnership around learning. The current M&E network that has been established through this LWA provides a unique and exciting foundation to create such a mechanism. If the M&E leaders from the other DRG LWAs could participate, it would result in stronger learning products. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │35 − Discussion about best practices and desired channels and formats for information dissemination related to SCS Global. Use this pause-and-reflect meeting to do user-centered design approaches to getting feedback and strategic direction on the NGOConnect.net and how to better communicate with and deliver thought leadership and knowledge products to the DRG community. Dissemination strategies for USAID may differ from local partners and the wider DRG community, and this should be taken into consideration. 2. Deepen the work around thought leadership (Objective 1 of SCS Global) with a focus on improved outputs in the final year of the program and making it a greater priority in future iterations of the program. This recommendation stems from the pronounced difference between the effectiveness and perceived impact of Objective 1 and Objective 2 and questions the evaluation team had about how to best improve the delivery and execution of the Leader Objectives. The program has stimulated significant learning that can serve civil society and media organizations but needs to be communicated and discussed both within and outside of the program. In this way, there are short-term and long-term improvements that can be made: − In the short term, over the final year FHI 360 should focus on communicating and disseminating the thought leadership results achieved in the program. Following on Recommendation 1 outlined above, FHI 360 as the Leader should assess what it can realistically do in the last year of the SCS Global program to frame thought leadership results to reach national, regional and global civil society and media organizations in ways that can stimulate discussion, innovation and improvements in civil society activities. To jumpstart the creative process of forging a final year workplan for deepening the thought leadership components of Objective 1, FHI 360 could convene a meeting of the consortium partners together with key experts involved in the Delphi Panel process and to showcase the results gleaned from that initial research and to produce a follow up short report of how consortium partners and key CSM experts see thought leadership in 2020 as compared to when SCS Global started and solicit feedback on future trends. − Over the longer term, USAID should consider how it can achieve deeper results in thought leadership. A menu of possibilities to consider might be: • Look for a more visionary and innovative CSM thought leadership process and/or leader to help bolster the work related to Objective 1 of SCS Global. This process/person should be able to take the program in a fresh direction and bring significant local expertise and on-the-ground experiences into the knowledge products, toolkits, and activities. Such a process should incorporate CSM expertise but also take risks to increase innovation, thought-leadership, and tech-aware approaches for civil society and media. • Consider program staffing proposals and activities that can take the program in a fresh direction and bring significant local expertise on the ground. Perhaps staffing or country programs could bring in more SCS Global—USAID Thought Leadership Fellows to play central roles in future thought leadership activities. • Enhance the Associates Award and partner activities to bridge research and practice and increase dialogue among the think tank, CSO, and academic networks of interest to the CSM community. This should consider how to support more and better spaces for collaboration and learning among the global partners as well as country program fora (workshops, co-creation processes, etc.) that bring in more global partners as resources. To support the increased uptake of collaboration, the Leader may want to consider CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │36 including a Thought Leadership Advisory Council that would meet twice a year to help review and comment on the knowledge products and outputs generated by SCS Global or its future iterations • Seek a better balance between program management and implementation, research and learning, and higher-level type of thinking that needs to be drawn out of the M&E data and other learning that results from the AAs. This balance can help support more open discussions that would help keep in motion the flow of learning that FHI 360 envisaged in its proposal to USAID for the program. 3. Revise the design and resource allocation under the Leader Award. USAID and FHI 360 should sit together to re-think this and come up with a plan to better meet the needs of USAID's DRG programming. They should consider whether a potential new hire/leadership practice manager would help in this effort, someone who is distinct from the AA programs and who could produce and manage thought papers, toolkits, etc. (See Recommendation 1.) What is needed to improve the Leader Objective deliverables is a strategic review to how funding is used and the development of a strategic plan to make this happen. While FHI 360 has done an effective job on the management of the AAs, KIIs agreed that FHI 360 can improve cost efficiency in the future and focus its role into global coordination and learning. The program workplans should prioritize a more equitable/shared distribution of the funding and better leverage and use the expert consortium partners. In the future, FHI 360 or another Leader should consider ways to better leverage and use the expertise of the wider consortium of the LWA partners. If cost is a barrier for having much of the leadership and management for the Leader award be based in D.C., FHI 360 may want to consider locating these positions in Nairobi or another key location where there are many local and regional experts who could help manage and lead on knowledge products. 4. Facilitate knowledge management and learning systems for the 21st century DRG practice. Lack of effective knowledge management is a sector-wide problem and one that many NGOs are dealing with individually. After assessing SCS Global, we find a continuing demand in the DRG sector for better sector-wide knowledge management databases and to integrate an approach to research and learning that is built from common architecture and approaches to data management. In this regard, FHI 360’s current knowledge management system—which uses SharePoint, Excel, and other Windows-related platforms—is seemingly user-friendly and well organized; perhaps it could serve as a launching pad for a wider DRG sector discussion about this long-standing challenge CSM programming has faced with regards to how to improve the uptake of evidence in program design and implementation. 5. Improve program reporting to target the needs of country stakeholders and USAID. The program can look at other reporting approaches—i.e., shorter reports that distill the most important information for the reader (the Freedom House-led LWA consortium has evolved a five-page reporting format that might be a good model for the future). 6. Stress opportunities for interpersonal interaction in the program delivery. Currently much of the interaction among Associates and with USAID is facilitated only through FHI 360, even while many of the consortium partners are USAID partners in other programs, providing more opportunities for direct feedback. At the very least, the formal feedback and discussions would be helped with learning events or other fora in which FHI 360 is not the only conduit for learning and discussion. One easy solution would be to have an annual conference for SCS Global Consortium members. In short, there needs to be a better and more inclusive approach to interpersonal communication and the exchange of ideas. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │37 7. Improve functionality of NGOConnect.net to make it more accessible and user friendly for civil society and media organizations around the world. We live in a digital world, and websites, apps, and email communication play a vital role in connecting us all and helping us to learn and improve our knowledge and practice related to DRG programming. The website is a tremendous asset for SCS Global. It needs to do an overhaul of the site and improve its design and functionality. It has the potential to become a go-to source for the DRG community. SCS Global could make better use of its consortium media partners and communication experts to help improve the overall functionality of the site. It could put forward approaches to improving user experience and implementing user-centered design approaches to developing the site and adding in more popular functionalities, such as apps or social media. SCS Global should consider developing a robust marketing and communications strategy so that more people in the DRG sector community know about the site and what information could be found there and how to access and use it. In this regard, the strategy should also coordinate with other LWAs and their knowledge products as well as other USAID-related knowledge management, thought leadership, and MERL-related guidance. SCS Global could be a hub for this kind of coordination if it follows this approach. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │38 ANNEX A: CASE STUDIES CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │39 ANNEX A: CASE STUDIES Figure A1: Mural in Zambia of African leaders Nelson Mandela, Kofi Annan, and Kwame Nkrumah CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │40 DJIBOUTI CASE STUDY Abstract The Djibouti case study was chosen as a case of strengthening civil society in a closed environment. The program began in April 2019 to foster accountable governance and improve service delivery in education and health through cross￾sectoral dialogue and collaboration. It created a viable strategy and proved working with civil society in Djibouti was possible. It supported 12 civic associations with capacity￾building support to deliver training and be better communicators. Challenges were getting buy-in and trust from many stakeholders; defining key topics collaboratively between government ministries and civil society organizations (CSOs); expanding to civil society more broadly; and reaching newer and more grassroots groups. Opportunities to build on this investment are to promote locally led research, raise awareness and increase visibility of CSOs, provide an annual report on CSO development for donors, expand to work with other ministries, add a media component, and look for regional collaboration opportunities. Methodology The case study researcher carried out nine interviews and site visits in-country, including with the USAID/Djibouti technical offices, the Program Office, Government of Djibouti officials, implementing partners, local organizations, and other donors. Fieldwork occurred on February 27, 2020. Interview data were complemented by background documentation from USAID/Djibouti and other stakeholder sources. Country Context In Djibouti, civil society is in very nascent stages. Civicus rates it as a “closed society” with few protections for freedom of expression, and Freedom House rates the country as “Not Free” in its annual Freedom in the World report, describing the country thusly: Djibouti is a republic ruled by a powerful president, Ismail Omar Guelleh, who has been in office since 1999 and is not subject to term limits. While Djibouti technically has a multiparty political system, the ruling Union for a Presidential Majority (UMP) uses authoritarian means to maintain its dominant position. The opposition’s ability to operate is severely constrained, and journalists and activists who air criticism of Guelleh or the UMP are regularly harassed or arrested.13 13 Figure A2: Youth playing soccer in downtown Djibouti https://freedomhouse.org/country/djibouti CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │41 Elections in April 2016 reportedly intensified repression of independent voices and civil society (for example, soldiers killed more than 19 people at a religious ceremony in 2015). CSOs and activities must be sanctioned by the State in order to operate legally. It is worth remarking that the most influential CSO, based on feedback shared with the NORC evaluation team, is led by the President's wife. Mission Factors According to the U.S. State Department Integrated Country Strategy for Djibouti,14 “Our bilateral collaboration to strengthen governance focuses on refugees and anti-trafficking-in-persons. The Government of Djibouti’s challenges in attaining economic security for its citizens, many of whom live in poverty, depend on developing a strong civil society to assist with services that the government cannot meet alone. We anticipate that USAID’s nascent civil-society strengthening project, conducted in collaboration with sector-specific ministries, will assist in this area.”15 Additionally, the SCS Global’s program in Djibouti contributes directly to one of the U.S. Mission Goals in Djibouti: • Mission Goal No. 3: Improve Djiboutian Government delivery of key services. − Mission Objective 3.1: Improve delivery of education, electricity, justice, and health services by furthering collaboration among citizens, civil society, and government, and advocating reforms, in coordination with relevant ministries and other development partners. IMPLEMENTATION OF THE PRECAD, AN ASSOCIATE AWARD UNDER SCS GLOBAL: The objective of the Civil Society Organization Strengthening Project (or Programme de Renforcement des Capacités des Associations de Djibouti (PRECAD) is to strengthen CSOs to foster accountable governance and improve service delivery. Led by FHI 360, PRECAD supports Djiboutians to engage in constructive dialogue with government officials to improve service delivery in education and health and to advocate for gender equality, equitable service delivery, and the protection of vulnerable populations, while improving the enabling environment for civil society. 14 https://www.state.gov/wp-content/uploads/2019/01/ICS-Djibouti_UNCLASS_508.pdf 15 Note: Integrated Country Strategy is a State Department document and provides the overall strategic framework within which USAID’s program fits. This document dates from August 2018 and therefore was completed prior to the implementation of the SCS Global activity, but the information contained in it remains relevant to the work of SCS Global in Djibouti. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │42 Figure A3: Photo of Djibouti Mural PRECAD’s technical strategy aims to achieve the following three results: • Outcome 1: Accountable governance better integrated into education and health sectors • Outcome 2: CSOs more effectively advocating for gender equality, equitable service delivery, and protection of vulnerable populations • Outcome 3: Improved enabling environment for CSOs in Djibouti There was a year delay in the launch of PRECAD; its start date was moved from April 2018 until April 2019. Three result areas were defined: strengthening CSO capacity; improving government capacity to work with CSOs; and working with the Ministry of Women and Family Affairs to improve CSO/government work on gender. For its capacity-building work, PRECAD must work closely with CSOs involved in health and education; it is required to get buy-in from the health and education Ministries before it can proceed with activities. As an outcome, PRECAD seeks improvements in service delivery in education and health in which: More women and young people will be involved in civic engagement, CSOs’ expertise in gender and social inclusion will be deeper, and civic activism will be expanded around gender, social inclusion, and the protection of vulnerable populations. Additionally, Djiboutian civil society, government, and international stakeholders will better collaborate to strengthen the enabling environment for civil society. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │43 Figure A4: Outside Wall of the Ministry of Health in Djibouti The highlight of PRECAD was a health workshop in November 2019, which was guided by a co￾creation process called “Whole Systems in the Room”16 (WSR). Key informant interviews (KIIs) reported that the workshop invited all possible stakeholders from CSOs, including government and USAID, which allowed for buy in, trust, and cross-sectoral dialogue. (Note: while media could be considered part of civil society, a decision was made to have no media-related programs involved in PRECAD. Also, for Djibouti, the term “civil society” was not used; the preferred term “civic association” was used instead.) − “The Whole Systems in the Room workshop addressed the needs of civil society. I thought it was useful. The mainstream organizations who came were very impressed by the training. Before the training workshop, the projects were on their own—they never had long-term vision—it was very important to have the support. One of the key outcomes— to have all the civil society in one room, to talk to each other, to have collaboration, and to create a network of civil society in this field. The Ministry has been doing a lot of work to talk with the parents association—without their involvement, parents’ involvement — they might not reach the milestones. The workshop was useful in this way.” (KII, Djibouti, 16 From FHI 360’s FY 2020 Quarter 1 Report: In cooperation with USAID, PRECAD partnered with the Ministry of Health to organize a two-day multi-stakeholder consultation workshop on November 19–20 at the Kempinski Hotel to bring together key players in the health sector to discuss and collectively identify “the roles and contributions of associations in improving community health services.” A total of 108 participants (46% women and 29% youth) from partner associations, government ministries, public institutions (Ministry of Decentralization, Ministry of Women and Family, Ministry of Interior), locally elected officials, Ministry of Health partner associations, and PRECAD Technical and Financial Partner took part in the workshop. USAID’s Mission Director from Addis Ababa attended the workshop, accompanied by USAID’s country representative in Djibouti. The Ministry of Health Senior Advisor delivered the workshop’s opening address. The workshop was facilitated using the WSR methodology, which is designed to identify and bring together a representative sample of actors operating in a sector. It was facilitated by PRECAD’s Chief of Party, who is a specialist in the methodology. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │44 Government Ministry) “The Whole Systems in the Room Training was useful; it was very useful, especially for the youth organizations who were active in the training sessions.” (KII, Djibouti, CSO) − “The Kempinski meeting brought visibility to CSOs. During that meeting, they came to realize the importance of civil society, [which was helpful for us] because every ministry has an action plan, and we got a sense of what civil society can do. This is so important for awareness, inclusivity.” (KII, Djibouti, Government Ministry) − “During the training at Kempinski (Hotel)—building capacity and giving them the tools, how to communicate, how to provide training. It built our own confidence. We already have all those skills, but we didn’t know how to apply them on the ground.” (KII, Djibouti, CSO) − “The biggest gain for me as a director of a CSO is that we could break down the wall of trust—to change the perspectives of CSOs and to change the image of civil society with the Government.” (KII, Djibouti, CSO) Achievements • Created a viable strategy and proved working with civil society was possible in Djibouti. • Even with its delayed start, the project demonstrated that working with civil society in a country where there are few known local and international CSOs is possible. • Supported 12 civic associations, including supporting the development of strategies and action plans and offering them capacity-building support to deliver training and be better communicators. • Through PRECAD support, USAID furthers its work in the areas of: − Supporting education, environment, health, gender, poverty, sustainable development, women and children (education, health, nutrition) − Building government/civil society trust and interaction17 − Trust building and government ministry buy-in to the project, which was enabled by an innovative pre-award co-creation process 17 Government also participated in CSO training. The director of the Ministry of Education has been to the trainings. Staff say this approach provides continuity and builds trust among the different stakeholders. It is “Like when you braid hair. There’s a knot in which they come together.” As a result of some of this work, FHI 360 was invited to present its work to an international donors group that meets once a month at the Ministry of Education, and there is some evidence that the World Bank, GIZ, JICA, and UN agencies in the country have committed to invite CSOs to relevant meetings. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │45 PRECAD’s 12 primary local partners representing entry points in Djibouti to further develop democratic governance. The selected Djiboutian Civic Associations / NGOs included four “organizations of public utility.” 1. Union Nationale des Femmes Djiboutiennes (UNFD), national focus 2. Bender Djedid, national scope 3. Ecologie de Village Association (EVA), national focus 4. Association pour le Développement Intégré de Mabla (ADIM), focus on Mabla 5. Association Toha, focus on Djibouti and Tadjourah 6. Paix et Laix, national focus 7. Association Mouhbani, focus on Tadjourah 8. Association Solidarité Féminine, focus on Djibouti 9. Association pour l’Avenir et le Développement d’Obock (AADO), focus on Obock 10. Association Al Sounna Al Haya, focus on Djibouti 11. Union pour le Développement Culturel (UDC), national focus 12. Association AL FOWZA, focus on Djibouti and Tadjourah Most Significant Change All interviewees were asked to describe what they thought the most significant change was as a result of the first phase of PRECAD. These are some of the highlights: • “The biggest impact has been as a Djiboutian to see civil society organizations take part in meetings with government institutions and to impact the work of the government institutions. I’ve never seen this before. The ministries were actually encouraging the members of civil society to contribute more. They were actually asking more from civil society.” (KII, FHI, Djibouti) • “The most significant change for us was in the area of capacity-building because we have acquired more knowledge. FHI has helped us to get closer to different stakeholders in society—state and non-state actors and institutions. The acquisition of new knowledge and improving our intervention in society, it brought us closer to the Djiboutian society overall.” (KII, CSO, Djibouti) • “One of FHI’s key areas of support for us has been on institution building. They came at the right time. FHI has been collaborative. It didn’t impose—it was collaborative. By discussing and having equal opportunity to express our ideas—to openly discuss and debate—we could highlight the things that need to be improved. Before FHI, for example, it was like people coming and saying let’s work on this, without any guidance or strategic direction: FHI brought this to our organization.” (KII, CSO, Djibouti) Challenges The challenges included: 1) the need to get buy-in and trust from many stakeholders; 2) the need to define key topics collaboratively between government ministries and CSOs; 3) expanding to and reaching civil society more broadly; and 4) a related challenge of reaching newer and more grassroots groups. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │46 With the caveat that the evaluation team had very limited time in-country, from what they could discern, the program has made some significant gains and steps that appear to be valuable to both USAID and to Djibouti Ministries in Education and Health, as well as to civic associations. The researchers recommend further critical thinking and analysis of what the Whole Systems in the Room workshop led to and where they are at now, given the goals and strategies that were developed as a consequence of the workshop. If this first phase was about “baby steps” and just getting a civil society strengthening program in place and functioning, a second step might start with a scoping exercise to see what should happen next and toward what ends. The focus on “governance,” service delivery, and strengthening of institutions seems key. While there is no possibility at this stage of adding an independent media-strengthening program as part of the existing PRECAD program, there are ways to augment the service delivery and improve the institutions through Djibouti’s social media/alternative media space. There also could be ways to use and harness the power of radio; this would need to be scoped. A “do no harm” approach was taken first and foremost, so as to not compromise the other work that USAID is involved in. The impact of the continuing linkages and interworking with other FHI programs must also be a considered. For example, FHI’s reputation and brand as a health NGO could be useful, though interviewees also thought there would be space for other players. That said, FHI’s health brand may have provided a boost to the PRECAD program and the gains made (baby steps achieved). These linkages have been established, and 12 organizations have been awarded some funding to do work, though the evaluation team was not able to discern if there has been any overarching change or major outcome of the work done by these organizations. The fact that FHI was able to set the program up and begin this work was considered important and an accomplishment, especially in light of the extent to which Djibouti is considered a closed society. Lessons learned (1) Good facilitation and meeting management is key. According to KIIs, the first challenge “is to convince the Ministry to actually allow the participation of the CSOs identified by FHI and not themselves. This is an obstacle.” (in-country KII) Then the project must prepare CSOs to be able to contribute to improve education or health strategies or delivery. (2) Clarity of the scope and definition of topics is also important. The program has worked to define four topics it is working on: • With the Ministry of Education, how to enhance the role and contribution of CSOs to the education system • With the Ministry of Health, how to develop and support community health-based activities • With the Ministry of Women Affairs, how to be more gender inclusive in activities • Utilizing the “Whole Systems in the Room” co-creation process as a tool for buy-in and ideation (3) Working with CSOs to promote gender inclusivity will potentially pay off. To work on gender inclusivity, an FHI KII said, “We can decentralize the work. A lot of the CSOs are doing gender work, including gender-based violence, gender discrimination.” (4) It takes time to design the right strategy and to build sufficient trust in a country with largely closed civil society. As one in-country KII said, “As a Djiboutian, the biggest impact has been to see civil society organizations take part in meetings with government institutions and to impact the work of the government institutions. I’ve never seen this before.” CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │47 (5) Invest more time and effort into ensuring local staff are well trained and up to speed on USAID policies and procedures, especially around monitoring and evaluation. Feedback from USAID suggested that the local PRECAD team would benefit from deeper understanding and improved skills in carrying out research, report writing, and better communicating the results of the program with an evidence-based approach. Implications The late start of the program provides an opportunity to review the program strategies. As one KII said, “Strategy has to be reviewed in light of the one-year-late start. This affected capacity building, and … capacity building takes time to plan in order to get it right. It requires a strategy.” (FHI KII) Having built a foundation with health and education CSOs, the program plans to widen the scope of intervention gradually beyond these issues. KIIs with FHI stress the need to provide continuity in the initiative to demonstrate to the government that this is not a one-off effort and that they will see improvements in the program delivery of CSOs. The program is an institution-building strategy that needs to create a sentiment of trust and mechanisms to transfer strength to new and grassroots CSOs. While the program should be seen as a very initial effort to strengthen civil society in Djibouti, the initial results show that engagement and trust between government and CSOs is possible. However, it may take time before this can have ripple effects to alter policy or conditions needed to encourage a robust sector of CSOs. A continued, patient effort can begin to expand the small base. The evaluation team suggests that these opportunities for increased investment should be pursued: • Promote locally led research. According to KIIs, there is significant demand for research in which CSOs can participate, particularly in education and health. • Raise awareness and increase visibility of CSOs. One idea raised in KIIs is to hold an annual CSO conference where the 12 civil society organizations can present and publicize their work. The yearly approach would showcase the work of the CSOs and demonstrating the work that has been implemented can show donors how the organizations have been on the ground and implementing activities. • Provide an annual report on CSO development for donors. FHI plans to do a baseline presentation in 2020 for the year ahead. • Expand to work with other ministries. As the initiative begins to show some positive results with the education, health and women’s ministries, it can act as a bridge and conduit more broadly. For example, it could work with the Ministry of Home Affairs that serves as a focal point for civil society on the ground or the Decentralization Ministry to ensure reach beyond the capitol. • Add a media component. The initial strategy of the initiative was low profile; however, as it builds CSO capacity and seeks some visibility, it should consider including media outlets and a media-strengthening component. According to KIIs, all activities carried out so far have been covered by the media; the initiative could build on these nascent links. FHI can help CSOs communicate their work and strategically publicize their work on radio, TV, or online. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │48 • Look for regional collaboration opportunities. Given the connections between Djibouti and neighboring Somalia and Somaliland, there may be opportunities to explore how to link and develop collaborative programming between civil society and even media. Somaliland, like Djibouti, does not have a free media space; however, there are very important and active media associations, research centers, think tanks, and emerging self-regulatory bodies. Somalia has a vibrant local media scene, startup culture, and gender empowerment movement. There are some intriguing differences of these three places, yet the similarities are also notable, and it might be interesting to do a regional project on CSOs and media—using the co-creation process—to see what could happen and whether regional/peer to peer connections and networks would help boost civil society and media in organic ways. ZAMBIA CASE STUDY Abstract The Zambia case study was chosen as a representative of a youth-focused civil society program. Youth Lead began in October 2018 with a goal to support, network, and elevate outstanding young leaders ages 18 to 35 from across Zambia to amplify the impact of their work, inspire civic innovation, and promote a culture of sustained leadership. Youth Lead provided an experience nobody else is offering in Zambia that gave youth access to employment and skills and experience to participate in governance and development of civil society activities within their own communities. Challenges are to extend reach outside of the capital, Lusaka, and a small in-country budget. According to beneficiaries, the program is highly successful. Some suggestions for improvement are to expand the program beyond Lusaka, strengthen communications both coming from FHI 360 and between the cohorts, monitor the goals of the interns, scale and build a more robust program, create more outdoor activities, and continue support for the National Youth Development Council. Methodology The case study researcher carried out interviews with more than 36 individuals from a variety of stakeholders of the Youth Lead program, including FHI 360, students, young professionals, civil society organizations, Government of Zambia officials, implementing partners, and USAID/Zambia DRG and M&E staff. Fieldwork occurred from February 17–20, 2020. Interview data were complemented by background documentation from USAID/Zambia, FHI 360 program records and other stakeholder sources. Country Context Zambia has a relatively robust civil society with a strong sector of church-related, development and human rights CSOs. Civicus rates its civil society as obstructed, noting that it has limited capacity Figure A5: Cohort of Youth Lead interns outside of FHI 360 office CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │49 and networks and a dependence on external funding. Despite political polarization, a lack of judicial independence and prevalence of state human rights abuses, CSOs have worked successfully to uphold democratic practice and peaceful power transitions, and they negotiated with the state after a highly restrictive NGO act in 2009. Zambia currently has the largest youth population in its history—nearly 37 percent of the population is between the ages of 15 and 35.18 Mission Factors USAID’s development solutions in Zambia, as captured in the Country Development Cooperation Strategy (CDCS), focus on integrated results from various sectors. The CDCS focuses on three key development objectives: • Development Objective 1: Enabling Governance Environment Improved • Development Objective 2: Rural Poverty Reduced in Targeted Areas • Development Objective 3: Human Capital Improved Given the significant size and needs of the youth population, the Mission has a Youth Focus in which youth development is a cross-cutting theme. The mission views youth as a potential catalyst for accelerated economic and social development or, if politically marginalized and overrepresented among the poor and unemployed, a major source of social and political instability that could derail Zambia’s journey to self-reliance. Among many other activities the mission advances, DO1 activities advance civic engagement through citizen-led structures, including focusing on strengthening the role of women and youth through leadership development and networking, which builds social capital for Zambian self-reliance versus dependence on an outside power. In addition, they protect human rights and provide assistance to human rights defenders and organizations, thereby strengthening the foundation for freedom and advocacy to express the will of the Zambian people. SCS Global’s project in Zambia, Youth Lead, helps address the Mission’s priorities to support long￾term change for youth and development programs. It identifies and selects a pool of promising youth to take part in leadership development alongside paid internship opportunities that come with mentorship and other life skills opportunities. As one USAID KII noted, the goal of Youth Lead is “to support, network and elevate outstanding young leaders from across Zambia to amplify the impact of their work, inspire civic innovation and promote a culture of sustained, citizen-responsive leadership.” In terms of workforce development, the logic of the program is based on the notion that if you can provide opportunities for young people at an early stage and give them the mentoring and leadership training needed to support their early career ambitions, they will be more likely to secure long-term employment and establish a pathway to a secure future. The Zambia program is built around USAID support for the government’s efforts to support inclusion and leadership for youth and works through a Government Ministry of Youth and its agency, the National Youth Development Council (NYDC), FHI’s core partner, which is headquartered in the same building in Lusaka. NYDC is a statutory body established by an Act of Parliament to register youth organizations, coordinate and regulate youth activities, and mobilize resources for youth development in the country. It works with youth organizations, programs, clubs, and enterprises. 18 http://www.daily-mail.co.zm/development-revolution-demands-stronger-youth-engagement/ CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │50 Implementation of the Youth Lead Program, an Associate Award under SCS Global Youth Lead is a three-year program launched in October of 2018. It is implemented by FHI 360 in partnership with NYDC. The goal of Youth Lead is to support, network, and elevate outstanding young leaders ages 18–35 from across Zambia to amplify the impact of their work, inspire civic innovation, and promote a culture of sustained leadership. Youth Lead will fund a six-month internship for 350 youth over the life of the program (for a total of seven to eight cohorts in all). Applicants selected for internships must successfully complete a four￾day, pre-internship training. Upon completion, participants will be placed with a host organization— CSO, government entity, or private business—and will be eligible for a monthly stipend of $135.00 USD for a period of up to six months. The monthly stipend is intended to defray costs associated with transportation to and from the workplace and participation in required leadership training and activities. Interns are required to work four days per week and must attend a leadership, skills and civic training activity one day per week. The internship consists of matching an intern applicant with a local host organization (so far they are all in Lusaka). The Friday session is meant to be focused time for leadership training, life and job skills, and capstone projects. The interns are grouped into teams to develop and administer a capstone project that relates to women, health or governance issues. According to KIIs—both hosts and interns—Youth Lead internships are the first of their kind in Zambia. There have been others that send Zambian leaders out of the country or that focus on the health sector, but a fellowship program that seeds and supports youth professional skills and leadership training and that keeps Zambian youth connected so directly to a multitude of civil society and government institutions is rare. Calls for applications are posted online through multiple channels, including the USAID/Zambia Facebook page. Youth Lead will now have its own Facebook page and a website, youthleadzambia.org. The program is very popular; for instance, when it first recruited applicants, it had roughly 4,000 applicants that were winnowed to 800 and then narrowed down to a few hundred semi-finalists out of which about 50 youth are selected for each cohort. Achievements Youth Lead offers an experience nobody else is offering in Zambia, which has the potential to train and support a new generation of professionals and equip them with leadership, management, practical administrative, and other office skills needed for today’s Zambian work environment. The program, according to KIIs, gives youth access to employment. Youth gain the necessary skills and experience to be active participants in governance and development of civil society activities and within their own communities. The program contributes significantly to USAID’s commitment to the Journey to Self-Reliance. As noted by some of the KIIs: • Hosts are enthusiastic to get Youth Lead interns. “When the opportunity to have an intern came up, we just jumped at it. I came from the UN: we had a lot of interns. It’s important to give these Figure A6: Youth Leader Fellow at the World Bank in Lusaka with her Mentor CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │51 youth a chance. It’s not easy to go into the workplace.” (KII, Zambia, host) Another sentiment heard from many hosts is summed up nicely: “Youth Lead is a great project—I hope to see it continue in the future. It will make a great impact: it’s one of the only intern programs in Zambia. No one else is really doing this.” (KII, Zambia, host) • Interns are grateful for the program. “What FHI is doing, it is great—especially given our high unemployment. I wish they could expand it. An entry-level job is hard to come by. The Youth program gives you skills.” (KII, Zambia, intern) • The internship includes a capstone experience. For interns, many cited the capstone as one of the key experiences of their internship. Through internships, many youth had access to high-level meetings, opportunities to represent their host organizations at government ministerial meetings and get experience with budgeting, report writing, and other practical job experience that they do not receive at university. A key component of the program is direct youth engagement with civil society and other partners. Interns design and implement community projects and establish longer-term partnerships beyond Youth Lead. Interns say they received valuable experience that expanded their professional skills and gave them practical experience. • Youth Lead helps with job placement. For many of the interns, the experience also led to a full-time job. For instance, one of the interns works with the World Bank, which is trying to create a full-time position or her (she is an engineer by background). She will be offered a World Bank consultancy following the internship through Youth Lead. See her picture at the right. • Youth Lead reaches a broad section of inspiring youth. Other internship or scholarship support goes primarily to college-educated professionals or recent graduates, whereas the Youth Lead internship reaches many youth leaders who could not otherwise access these opportunities. The project is also unique in that it lasts for an extended period of time (six months) and provides sustained training and oversight through placement at top institutions: Government ministries, Clinton Foundation, International Development Enterprises, World Bank, Pact, Democracy Works, and more. Challenges 1. Reaching outside of Lusaka. To extend program reach outside of the capital will require more money and time, which is feasible. 2. A small in-country budget. The program is run on a very small budget, and most of this money covers FHI 360 staff in Lusaka and Washington, D.C. There should be a budget review to look at how the program could be better budgeted for, staffed, and scaled (or at least run at current levels), given how impactful the program seems to be based on the NORC team’s limited but intense period of qualitative research. Lessons Learned 1. Running and managing an internship program such as Youth Lead is human resource intensive. FHI 360 dedicates a great deal of staff time to managing the internship program. For example, the Chief of Party invests significant time and energy working with NYDC to help them develop curriculum and manage the overall Youth Lead program. He also works with host institutions on enabling them to provide a fruitful internship experience. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │52 2. It takes time to find the right partner. There were two early partners, NYDC and a local CSO. The CSO did not work out because they were not capable of carrying out the vision and workplan that was submitted. 3. A financial incentive for Youth Lead interns is important. The interns get a stipend that pays for travel. Some interns use this to supplement their or their family’s income and would need to look for other income in lieu of the internship. Some interns get the FHI stipend as well as stipends offered by their place of work, e.g., National Democratic Institute (NDI) or Democracy Works, and some do not. This causes dissatisfaction. Getting the level of the financial incentive right is a key factor in program success. 4. The value-for-money proposition is significant. This program transforms lives and gives hope and opportunities to youth in an unparalleled way. The initiative could be replicated in many countries to support a new generation of civil society and media-related programs. 5. The model keeps the assistance in the country and directly impacts and benefits the local community. A number of KIIs commented that most fellowship programs take Zambians out of the country. This program localized the intern experience and led to local job creation. Interns got practical experience they can apply in their current and future jobs. Youth Lead is supportive of USAID’s strategy to support the Journey to Self-Reliance and also beneficial to the Agency’s focus on positive youth development. 6. Need for flexibility. One of the first partners of FHI to run the program was a locally based organization called Avencion. It turned out Avencion did not have the right contacts, staffing, or program management expertise to work with FHI and USAID to run the program. With FHI 360’s flexible approach and ability to leverage other partnerships, like with the NYDC, it quickly pivoted, and the program has been run successfully with just one partner instead of two. In addition, USAID has been able to bolster and support government capacity to run youth programming. FHI 360’s leadership and oversight of the program contributed to its successful efforts to employ USAID Collaborative Learning and Adaptation principles. Implications The Youth Lead project, according to interns and hosts, is highly successful, and all interviewed would like it to continue in Lusaka and expand throughout Zambia. Overall, Youth Lead is a strong example of the Associate Award objective for the SCS Global LWA.19 KIIs did, however, offer some suggestions for how the program could improve: 1. Strengthen communications both coming from FHI and between the cohorts. Host KIIs said they got conflicting communication. As an intern/host commented in a joint interview they had with NORC, “When we first started the program, we expected communication. We had to ask the previous cohorts. We had a lot of confusion about the Capstone project. We were not sure what was expected of us. There were also some limitations on the capstones and where we could do them. They (FHI) should share a calendar of activities and deadlines with the hosts.” (KII, Zambia, Intern Host) 2. Monitor the goals of the interns. As one of the interns noted: “We shared our goals last year. I never got feedback last year. Maybe during one of the Friday sessions we could review 19 AA Objective: Provide technical assistance and support to a wide range of civil society and media actors and organizations to enhance participation, inclusion, transparency, and accountability across sectors. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │53 these goals. Cut down on some topics: give one-on-one with leaders. Not everyone feels comfortable with the group setting format. Professional development classes would be useful.” (KII, Zambia, intern) 3. Scale and build a more robust program, such as including more guest speakers. “Guest speakers would also be great. We’ve had a few; they are related to the projects.” (KII, Zambia, intern) Guest speakers could be an area of collaboration. While Youth Lead was considered unique, there were some similar programs like Global Health Corps that interns and hosts felt could be leveraged. For instance, Global Health Corps could offer guest speakers to the Youth Lead Friday sessions: they have over 60 alumni in the country. Peer Learning and Peer Mentoring could be enhanced. Zambia has many Young African Leadership Fellows and other USG State Dept/ USAID-funded Fellows and a rich alumni network of experts to draw on. A guest speaker series and a kind of “careers in action” would be a boost to the Youth Lead experience. 4. Create more interesting and advanced intern activities and classes—feedback from interns (and hosts) suggest a need for more creativity and diversity of activities. One intern noted, “I would like to get to know others and have mixers with different cohorts. There is little mixing between cohorts now. The Friday sessions need some improvement. Some topics are repetitive. Don’t repeat classes that they had in high school and college.” (KII, Zambia, Intern) 5. Continue support for NYDC. The National Youth Development Council is revered as a strong partner for the program, and with the right support and collaborative partnership put into place, they could readily take over the program and run it as part of the work they support with Zambia’s Ministry of Youth. Ending the program now would jeopardize the significant accomplishments made and the reputation of this initiative. Given the lack of opportunities for youth in Zambia and the success of getting young people the skills and connections they need to find gainful employment, FHI and USAID should look into ways to further support this effort. KENYA CASE STUDY Abstract The Kenya case study was chosen as a deep look at how SCS Global utilized the co-creation process to support a broad and diverse set of actors within CSM programming. The program began in 2018 with a goal to enhance Kenya’s ability to hold a national conversation on reforms and national cohesion and to safeguard democratic gains, including protecting civic space, respect for human rights and observance of rule of law. The co-creation processes in tandem with project design and the overall sequencing of deliverables brought people together, enabled networking and conversation, and created significant opportunities to bring in new players. A major challenge was managing expectations raised in co-creation. Frustration with FHI was created through unclear expectations and follow-up. Recommendations for the program included improving follow-up communication and holding events and workshops related to the initial co-creation process. Additionally, KIIs noted that co-creation was not the only option and that sometimes it is better to just have forums such as a bidder’s conference. Given the importance of communication, media/journalistic reporting, and the need for the public to have access to fact-based reporting, KIIs noted that for a program like Safeguarding Democratic Space in Kenya (SADES-K), media programming needs a bigger focus. Local CSM stakeholders and those involved in the BBI process noted that SADES-K program leadership requires governance and that any grants process that is used to support local CSM organizations CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │54 needs to have an efficient and timely grants management system and a robust communications and outreach strategy. Methodology The case study researcher carried out more than 38 interviews and did multiple site visits in￾country, including with the USAID/Kenya technical offices, the Program Office, Government of Kenya officials, implementing partners, local organizations, and other donors. Fieldwork occurred from February 21–25, 2020. Interview data were complemented by background documentations from USAID/Kenya and other stakeholder sources. Country Context Kenya has been moving toward a more open society for more than a decade, but its efforts to democratize have been punctuated by political and election time violence. It has a robust and vociferous civil society that is playing a significant role across a large number of issues but feels under threat with the current government and is aware of the potential for closing space. The Building Bridges Initiative (BBI) is an important continuation of the Government of Kenya’s efforts to support democratic elections and encourage a healthy civil society. Kenya also has a vibrant and multi-faceted media sector, but it is often prone to hate speech, rumor, misinformation, and running biased or misleading stories that slant in the direction of the owners. The youth population in Kenya is also big: as of 2018, 25 percent of the population was between the ages of 18 and 35. LGBTQ issues are also emerging, but largely not accepted in society. Kenyan CSOs are strongly invested in work around gender and women's empowerment. According to the USAID CDCS, although Kenyans esteem democratic norms in theory, ethnic and tribal affiliations supersede national identity in practice. Political parties are often expressions of regional and ethnic interests, and it is consistently easier to mobilize voters around ethnicity than around issues or national problems. However, ethnic groups share legitimate grievances over corruption, poor service provision, lack of jobs and opportunities for meaningful participation and influence, and the lack of accountability by politicians to constituents. These led to broad-based support for the new constitution, and high expectations that it will remedy long-standing injustices and improve service delivery. Nowhere are expectations higher than in public anticipation of devolution. Mission Factors One concern of the USAID Mission, shared with government and civil society, is to avoid the election-related violence that is destabilizing and has emerged during all of the elections in the last five years. For the Mission, “devolution” is the strategic entry point. Devolution targets effective and accountable government systems, primarily at the county level, while building essential linkages at the national level and with civil society, thus fostering an enabling environment for the success of devolution and informed and engaged citizens.20 With its focus on strengthening and building bridges and understanding focused on civil society, the Kenyan Associate Award is not directly related to the CDCS since in 2013–2014 when the Mission developed its strategy. At that time, devolution and decentralization were seen as the key socio-political constraint on Kenya's future development as the then-new 2010 constitution and the elections subsequent to that had made it such a big issue. With the violence of elections and the fission of society along ethnic and political/personality lines 20 This is taken from the USAID CDCS for Kenya. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │55 that led to the violent 2017 election, the Mission pivoted to a focus on civil society and peace￾building, as seen in the SCS Global AA examined here. (Note that at the time of the interviews for SADES-K in February 2020, the USAID Mission was in the process of writing its new CDCS.) Implementation of SCS Program The overall goal of SADES-K, a three-year project (2018–2021) implemented by FHI 360, is to enhance Kenya’s ability to hold a national conversation on reforms and national cohesion and to safeguard democratic gains, including protecting civic space, respect for human rights, and observance of rule of law. SADES-K is grounded in a locally driven co-creation process with Kenyan stakeholders to further the national dialogue under the Building Bridges to Unity Initiative (BBUI), promote social and ethnic cohesion, protect civic and democratic space, and monitor the national conversation’s implementation. The project seeks to provide a much-needed bridge for CSO and media engagement between electoral periods in Kenya, mainly by supporting locally led initiatives and advocacy through technical assistance and grants. A key mechanism employed by the project is the pre-award co-creation process. Co-creation is a process or approach that USAID and other donors encourage to support locally led, dynamic, bottom up and partnership/collaborative development practices. The co-creation process was Kenyan-led and Kenyan-designed and focused on governance reform, social and ethnic cohesion, and protecting civic space. The co-creation workshops were conducted within 60 days of a financial award from the project on October 15–19, 2018. Specifically, the workshops objectives were to: (1) ensure SADES-K’s engagements and programming are informed by a wide range of Kenyan stakeholders (2) draft project frameworks that outline locally led initiatives that seek to protect, support, and promote civic voices, independent media, and implementation of the PBO Act, 201321 (3) project frameworks that would guide future grant applications submitted to SADES-K. In the co-creation sessions, participants worked together to recommend priority issues for SADES￾K to address, including challenges and opportunities. Participation in these sessions was not a guarantee of funding. A portion of each co-creation session was devoted to arriving at collaborative recommendations for project design. These recommendations processes were then linked to potential funding mechanisms, primarily by informing the design of the SADES-K grants program. 21 The PBO Act 2013 is a law seeking to provide a more enabling environment for NGOs, with clear criteria regarding NGOs’ registration, enhanced accountability, and set timelines for processing applications, as well as tax incentives and benefits for organisations conducting “public benefit activities.” It was passed by the Parliament in 2012 and signed into law by then-President Mwai Kibaki on January 14, 2013, but has not been operationalized yet under the Kenyatta administration, while several attempts have been made to introduce new restrictive amendments prior to its official commencement. For further information see: https://www.fidh.org/en/issues/human-rights-defenders/kenya-last-warning-from-the-court-to￾implement-the-pbo-act-2013 CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │56 Figure A7: Coalition led by the Kenyan Media Council (includes other key Kenyan voices in media self-regulation) Achievements Different approaches may work at different times and for different purposes. SADES-K provides a number of insights and lessons learned about the co-creation process and the various considerations that donors and implementers of DRG programming might want to consider before deciding on which type of co-creation process to pursue and how to sequence it. The CSOs and media interviewed had varied experiences with co-creation, and many were prepared to and wanted to talk about it. Some had never done co-creation before. The co-creation process used for SADES-K was a pre-award style of co-creation that helped FHI 360 better understand and program their work to take into consideration all the views and expectations raised in the co-creation workshops that were held before any grants or support were directly given in Kenya. The FHI team held four regional co￾creation workshops, and many of the participants ended up forming coalitions and partnerships with other attendees to apply together for funding. The evaluation noted that the process of using co-creation in tandem with project design and the overall sequencing of deliverables has a lot of pros but also some cons. Its pros include that it brought people together, enabled networking and conversation, and created unique opportunities to bring in new players. Cons include a number of frustrations voiced by participants about the management of the co-creation process by FHI, especially around consistent and meaningful communication, co-creation preparation, and managing expectations. Challenges The feedback on managing expectations was heard loudly and clearly. The facilitators of the co￾creation workshops themselves also expressed a lot of frustration with FHI: one of them even refused to be interviewed. FHI should do more due diligence to evaluate the co-creation experience and use this program to get better insight about what did and did not work well. There has been too little follow-up with those who took part in the co-creation workshops—especially for those who attended the workshops but, for whatever reason, did not get or apply for funding. FHI could have better conveyed that this was a pre-award co-creation experience and, as such, its purpose was to inform program design and the overall needs and interests of CSOs and media in Kenya. Many CSOs and media expected the idea that the co-creation was tied to procurement, but it was a little hard for them to appreciate or jump on board with the idea that after spending three CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │57 days doing co-creation with FHI and USAID, they would not get anything out of it. One way forward could be to shorten the co-creation workshop and focus it on pre-award program development. FHI, for its part, did endeavor to make it clear that this was pre-award co-creation and that the RFP for funding would come later and as part of a separate process. Still, based on the interviews and feedback with CSOs and media, there was a lot of confusion and a desire by the CSO community to engage in dialogue with FHI about what worked well and what could be improved. FHI should focus on the creation of more opportunities for engagement. CSOs and media with whom the evaluators spoke said that they received too little benefit, in terms of financing, engagement, or collaboration from their participation. They pointed out that the handling of communication between FHI and the wider CSO and media stakeholder community lacked follow￾up and clarity on FHI’s investment into the process. CSOs indicated that the FHI team should have had more direct leadership and expertise on governance, on the BBI process and a vested stake and sense of involvement in the overarching effort. There was also feedback from some KIIs, especially those persons who have been long￾standing USAID partners for 10 or more years, that FHI 360 needs to do more to show that it has the expertise and credibility in the democracy and governance sector (this comment was based on current views of the program and local leadership of BBI). The brand of FHI is very much tied to health, and KIIs brought this up as an issue that may have affected some aspects of the co-creation process, how it fed into program design, and the overall management of the grants and awards that came after the co-creation workshops. FHI for its part has endeavored to support co-creation as an active part of the program cycle in Kenya and to put forward experiences, methods, and approaches that will live up to DRG goals and interests and at the same time be responsive to the CSO and media community in Kenya. This should be an ongoing process: there is much to learn from the SADES-K experience—both successes and weaknesses—that could be of interest to the broader DRG community. As this process and tool of co-creation is used by more and more DRG programs, SADES-K offers an important glimpse into what it takes to utilize co-creation from a project design and management point of view as well as how to sequence the co-creation process into the overall program delivery. Lessons Learned (4) There are many models of co-creation and no one-size-fits-all approach. There are some clear markers that indicate when co-creation is really achieved. What the markers are, however, are not well-defined, and USAID and implementers such as FHI 360 should discuss the pros and cons of the different types of co-creation, perhaps together with partners, to select the best approach for a given context or program. While this lesson came from evaluation interviews in Kenya, it could be applied as a general comment to the whole LWA—and one that would be useful for the DRG sector and CSM programs in particular to think about. KIIs expressed a difference of opinion about whether a pre-award or post-award co-creation process would be better. USAID KIIs preferred pre-award co-creation processes because they felt this strengthens the overall program design and encourages collaboration between organizations. On the other hand, implementer KIIs preferred post-award co-creation processes as part of an already selected and confirmed set of grantees or partners. This difference appears to be related to valuing the time and trade-offs that must be made by CSOs that have very few resources and must make strategic decisions on how to employ them and when. Co-creation is not the only tool or process that USAID partners can use—and it may not always be the best option; alternatives should be considered. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │58 (5) Some useful do’s and don’ts emerged from the co-creation process work that SCS Global supported. It is important to manage expectations. Implementers need to provide clear guidance, open communication, and simple, bold text explanations that tell attendees of co￾creation workshops and events, what the co-creation workshop is for (purpose), and who should attend, why they should attend, and what they will get out of it (even if they will receive no material or financial support). Implications • Communicate regarding co-creation workshop outcomes. Many KIIs stressed that co￾creation is not a one-off process. It needs to be supported by significant follow-up and setting clear benchmarks and milestones. Some suggested that webinars or half-day learning events are effective ways to reinforce and build from the co-creation process. KIIs with co-creation workshop attendees indicate that they want to be kept informed, even if they did not get selected for grants or other forms of support. Having contributed to a co-creation workshop on the order/magnitude of the Kenya BBI-related co-creation workshops, which required a three￾day commitment, they want to stay apprised of what comes out of the process. CSO KIIs stress that they are “all in this together,” and they want to know what is working and what is not. • Media programming needs a bigger focus. Media stakeholders shared the perception that this program was a missed opportunity to work on media influence and BBI. Media stakeholders and CSOs noted that there’s been a lot of misinformation and inaccurate reporting related to the BBI process. Given the current state of affairs, CSM stakeholders felt there could be more of a focus on media. • Governance expertise expected from local stakeholders. The program needs to have a leadership with a Chief of Party and other senior-level staff who know the players in the governance sector and can engage and network with them directly. As noted by one KII (media), “You need to be meeting with the players, you have to have the contacts and know the issues and know how to read the room.” • Put in a grants process with an efficient and timely grants management system and a robust communications and outreach strategy. The process that FHI 360 has in place to review grants for local organizations is perceived as time consuming, laborious, overly managed, and one that creates unnecessary delays. It was also considered a kind of obstruction to the program’s goals because sometimes the approval process took so long that the window of opportunity for a media or CSO partner to seize upon the moment would pass once the approval was secured. KIIs asked for a more time-sensitive grants process as well as a rapid response grants mechanism. • Look at alternatives other than co-creation, like a bidder’s conference. The co￾creation workshops received a mixed response, but interviewees were in agreement that the co-creation process could use some re-thinking. It is not clear that co-creation was that useful, and some KIIs mentioned that a bidder’s conference would have been better: “The co-creation process didn’t have value.” (KII, Kenya, CSO representative) CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │59 ANNEX B: CONCEPT NOTE CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │60 TASKING N030 Performance Evaluation of the Strengthening Civil Society Globally (SCS Global) Leader with Associates Award (LWA) December 2019 Prepared under Contract No.: GS-10F-0033M/7200AA18M00016 Submitted to: Brandy Witthoft Submitted by: NORC at the University of Chicago Attention: Renee Hendley, Program Manager Bethesda, MD 20814 Tel: 301- 634-9489; E-mail: Hendley-Renee@norc.org DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │61 CONTENTS Acronyms 62 Overview and Objectives 63 Evaluation Objectives and Research Questions 66 Research Methodology 67 Research Limitations 71 Research Timeline and Deliverables 71 Proposed Staffing 72 Annex B1: Evaluation Design Matrix 74 CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │62 ACRONYMS AOR Assistance Officer’s Representative AA Associate Awards AAC Associate Awards Committee CDCS Country Development Cooperation Strategy CLA Collaborating, Learning, and Adapting CSM Civil Society and Media CSO Civil Society Organization DAI Development Alternatives International DEC Development Experience Clearinghouse DRG-LER Democracy, Human Rights, and Governance Learning, Evaluation, and Research Activity FGD Focus Group Discussions FHI360 Family Health International 360 ICT Information and communication technology KII Key Informant Interview LAC Latin America and the Caribbean LWA Leader with Associate Award MEL Monitoring, Evaluation, and Learning NGO Non-governmental Organization NORC National Opinion Research Center PAD Project Appraisal Document PD Program Description PMP Performance Management Plan RDCS Regional Development Cooperation Strategy RDRN Regional Democracy and Rights Networks RIA Research Into Action SI Social Impact SOW Statement of Work STTA Short Term Technical Assistance TA Technical Assistance USAID United States Agency for International Development USG United States Government CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │63 OVERVIEW AND OBJECTIVES Background As part of the DRG Learning, Evaluation, and Research II (DRG-LER II) Activity, USAID requested NORC to design and budget for a performance evaluation of USAID’s Strengthening Civil Society Globally (SCS Global) Leader with Associates Award (LWA), which began in May 2016 and will be completed in May 2021. SCS Global total funding is currently $2,500,000, the estimated Associate Awards (AAs) amount was originally $200,000,00022, and, based on mission demand to date, this ceiling may be increased for the remainder of the award. To date, SCS Global has 18 associate awards, with AA funding in the amount of $199,138, 173 SCS Global is designed to identify and develop effective approaches to strengthen civil society and media in support of democracy, human rights, and governance (DRG) and other development results in open and politically restrictive environments. The LWA focuses primarily on design and learning as well as fostering technical leadership and brokering connections between specific AA interventions to enhance outcomes and achieve economies of scale. A set of specific activities are generating practical tools and/or knowledge useful to most AAs as well as other related efforts. The award is currently held by FHI 360, with a consortium of partners including: Internews, Social Impact (SI), and Development Alternatives International (DAI); 13 other partners; and 4 network partners. Evaluation Overview The purpose of this tasking is to conduct, in line with USAID’s Collaborating, Learning, and Adapting (CLA) framework2, a performance evaluation of the SCS Global LWA to inform its principal audience (the Civil Society and Media (CSM) team at the DRG Center and Mission DRG offices implementing AAs) about LWA results achievement and challenges to date, with the goals of improving future SCS Global implementation through LWA completion and informing future DRG programming. The secondary audience for this evaluation includes USAID Mission and DRG Center and regional bureau staff who may be considering implementing or supporting civil society and media programming. To attain those goals, the evaluation will focus on how effective SCS Global has been, primarily under the LWA, at fostering technical leadership, including identifying key DRG trends and challenges in the civil society and media sectors and demonstrating thought leadership to address those challenges across the broader sector. The evaluation will assess the extent to which the SCS Global LWA meets the needs of the Agency in the Civil Society and Media (CSM) sector. The evaluation will also assess how effectively the LWA has integrated technical leadership and products into AAs, how SCS Global has served Missions, and how the mechanism was capable – or not capable – of adapting to Mission and operating unit needs. 22 This figure is an estimated amount. There is no guarantee regarding the magnitude of AAs in dollars or number of awards. The actual total of AAs funding originally was set to not exceed this amount though USAID has indicated that it may increase the amount. 2 See https://usaidlearninglab.org/lab-notes/exploring-cla￾framework CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │64 Project Objectives and Approach To address the state of the CSM sector, the FHI 360 Team is capitalizing on the momentum that the USG and other donors have already built: the burgeoning Civil Society Innovation Hubs that connect CSOs regionally for peer-to-peer learning and mutual support; the Open Government Partnership (OGP) that provides domestic reformers with an international platform to advance transparency and accountability; and the Community of Democracies that encourages democracies around the world to share best practices and improve political freedom. All of these initiatives allow civil society, media, and other system-wide actors in open and closed spaces to collaborate, coordinate, and act in new and more effective ways. The results of the LWA activities are informing approaches under AAs and in international development more widely, potentially influencing the entire field of civil society and media promotion. This will enable more consistency in USAID support to civil society and media, demonstrate value-added strength to CSOs and media, and broaden development approaches aiming to achieve both DRG and other development outcomes. The program objectives below are being implemented under the LWA are as follows: • Leader Objective 1.1: Develop and promote effective approaches to the strengthening of civil society and independent media actors. • Leader Objective 1.2: Effective management of SCS Global LWA. • AA Objective: Provide technical assistance and support to a wide range of civil society and media actors and organizations to enhance participation, inclusion, transparency and accountability across sectors. The following are SCS-Global programming principles as stated in the award: Coordination SCS Global Leader activities will be coordinated with all USAID DRG Center mechanisms that relate to civil society/media strengthening or civil society/media as a vector for participation, inclusion, transparency and/or accountability, including Making All Voices Count, the Legal Enabling Environment Program, the Global Labor Program, the Global Alliance for Community Philanthropy, and the Global Human Rights Support Mechanism, among others. This cross-sharing and coordination will ensure that learning takes place among the different DRG divisions that support field programming. Learning from this effort will also feed into the Agency’s Local Solutions working group for correspondence with Agency-wide efforts of similar nature, and to the Agency’s participation in the multi-stakeholder Task Team on CSO Effectiveness, a forum for discussion from multiple donors. Sustainability Sustainability Analyses are conducted during the program design phase by USAID Missions as they design and fund projects in the field. Leader activities for program design and learning will include data collection, evaluation and reporting, and will be built into the AA soliciting documents and resulting agreements. Program Descriptions (PDs) for discrete technical leadership activities will factor in sustainability analysis in design, implementation, monitoring and evaluation criteria and work plans. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │65 A key element of the SCS Global project is facilitating long-term, sustainable results and impacts in the area of civil society and media. On the Mission level, activity designs will incorporate measures to strengthen local institutional capacity of CSOs and independent media through ensuring sound financial management of participating organizations, facilitating —where feasible— the use of host country systems and CSO-government cooperation, promoting local ownership while improving the skills of local stakeholders, and analyzing and addressing the social and cultural dimensions of ensuring the sustainability of results in the country they work in, including determining the societal forces supporting and inhibiting sustainable change based on the assessment of the underlying political dynamics of society and the philanthropy environment. Gender Youth and Other Inclusive Development Considerations Addressing gender inequalities is only possible when the patterns that drive them are understood. FHI 360 intends to use, and train local partners to use, its qualitative tool to conduct gender assessments. This tool examines gender dynamics and inequalities according to five domains: (1) access to and control over assets and resources; (2) cultural norms and beliefs; (3) laws, policies, regulations, and institutional practices; (4) gender roles and responsibilities, and time used; and (5) patterns of power and decision-making. FHI 360 intends to use the resulting data to craft an overarching gender strategy for the Leader Award that will recommend how to integrate gender into program activities, and indicators to include in a baseline. FHI 360 intends to use the same approach in AAs, building the capacity of CSOs and independent media to undertake their own investigations into gender dynamics and inequalities, both institutionally and externally, that might affect their work. Additional support from Women for Women International will advance gender equality by promoting financial inclusion, advancing women’s entrepreneurship, increasing access to health services, and engaging men as allies. Monitoring, Evaluation and Learning FHI 360’s Team approach intends to ensure processes are in place to foster learning and that learning is central. Leader activities include, as a part of broader technical assistance, recommendations for monitoring and/or evaluation approaches relevant to strengthening civil society and independent media organizations in both open and restrictive political environments, which can be leveraged for AAs as well as in other programming. FHI 360 builds its learning upon individual AAs, as the bulk of anticipated activity under this LWA is through those AAs. SCS Global will ensure appropriate commonalities in the monitoring, evaluation and learning plans of AAs. All AAs will incorporate an Activity MEL Plan that uses applicable common metrics and learning approaches. FHI 360 intends to place an emphasis on the measurement of the activity’s theory of change, and to enable rigorous analysis of the contribution of activities to outcomes, and of outcomes within civil society and independent media to higher-order sustainable results. The theory of change is: if evidence-based effective approaches to strengthen civil society and media in support of DRG and other development results in open and closing environments are identified and developed, effective approaches to the strengthening of civil society and independent media actors, and the SCS Global Leader with Associate Award is effectively managed, then civil society and independent media organizations will be strengthened to advance DRG objectives and they will also advance other development goals (education, health, economic growth, environment, etc.). All such data will be disaggregated by gender and by other factors as appropriate (i.e. age, development sector, type and size of organization, etc.). In collaboration with DRG Center stakeholders and mechanisms during SCS Global implementation, the Leader Team developed a Learning Agenda that defined pertinent topics for exploration. This CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │66 Learning Agenda was intended to be revisited on at least an annual basis. To establish the Learning Agenda, the Team asked questions at an intermediate level of analysis: broader than examining the impact of a single intervention, but narrower than proposing a unified theory of development. In particular, the Learning Agenda addresses issues of broad concern that also have concrete programming implications. EVALUATION OBJECTIVES AND RESEARCH QUESTIONS Evaluation Objectives The evaluation questions will examine how effective SCS Global has been, primarily under the Leader award, at fostering technical leadership, including identifying key DRG trends and challenges in the civil society and media sectors and demonstrating thought leadership to address those challenges across the broader sector. Further, the evaluation questions will assess the extent to which the SCS Global LWA meets the needs of the Agency in the Civil Society and Media (CSM) sector. Finally, the evaluation will also assess how effectively the Leader award has integrated technical leadership and products into AAs (18 AAs to date), how SCS Global has served Missions, and how the mechanism was capable – or not capable – of adapting to Mission and operating unit needs. Annex I provides the Evaluation Matrix with greater detail on the questions and related issues. Research Questions and Sub-questions (1) How has the SCS Global LWA been effective in advancing USAID priorities in CSM programing? (a) What are the most significant achievements, both intended and unintended? (b) What are key weaknesses of the program? What are the major challenges? (c) To what extent has SCS Global influenced thought leadership within the broader development and DRG community on civil society and media-related issues? How effective has SCS Global been at ensuring integration of best practices, tools, or program methods beyond their immediate sphere of influence (i.e., within other SCS Global or FHI awards)? What more could be done to have an impact? What factors facilitated or impeded this impact? (d) Have USAID missions’ and DRG priorities and objectives in country-level programming been consistently and accurately reflected in the awarded programs? If not, what could be done to strengthen alignment? (e) How do the case study results speak to similarities and differences between country programming in terms of meeting objectives? What are the notable innovations, including co-creation and engaging local partners, examined through the case studies? Is there evidence of CLA within country programming? Is there evidence of learning, for example sharing of lessons and resources, across Mission programming? (f) How has the Leader award (as opposed to Associate awards under the Leader) been used to achieve related program objectives and results (i.e., Leader Objective 1.1 and 1.2)? (g) Are there programming needs in the area of CSM that are not effectively addressed through the SCS Global LWA? (2) Were the design and resource allocation within the SCS Global LWA conducive to USAID’s needs in CSM programming? CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │67 (a) Has the focus on thought/technical leadership and capacity building been consistent with USAID’s original vision? How have each of these subfocus programs been implemented to date? (b) Was the allocation of these resources and prioritization of these resources appropriate? (c) How has the SCS LWA dealt with emerging needs and other priorities not anticipated during the program design? (d) What are the strengths and weaknesses of the organization and management structures of the current LWA mechanism? (e) Are the consortium structure and arrangements providing maximum benefit and value for USAID? (f) Are key consortium partners, and sub-partners, including regional partners, used effectively and their relative expertise made available to (g) USAID? (h) Is communication between USAID and the consortium facilitated by the management structure – both within the consortium and within DRG? (i) Has the consortium’s management of the current LWA, including division of programming, communications with USAID and external audiences, and reporting and other obligations proven effective – why or why not? (j) Is USAID DRG responsive to the management needs of the SCS LWA, including substantial involvement? (k) How effective has SCS Global been at integrating technical leadership developed under the Leader into the design and implementation of Associate Awards? (3) How has the SCS Global LWA utilized co-creation during the development and implementation of Associate Awards? (a) What are the strengths and weaknesses of using a co-creation process with Associate Awards? (b) What are the effects of different forms of co-creation (pre-award or post-award), and do different forms have related or distinct strengths and weaknesses? (c) How does the current SCS Global consortium structure affect or relate to co-creation processes? (4) How unique is SCS Global from similar DRG mechanisms? (a) To what extent are the scope and capabilities of SCS Global significantly different from those available under similar DRG LWAs, such as the Human Rights Support Mechanism (HRSM)? (b) Do there appear to be unique capabilities or areas of focus under SCS Global that are not available under other DRG LWAs, such as HRSM or the Global Elections and Political Transitions (GEPT) LWA? RESEARCH METHODOLOGY NORC’s approach to the SCS Global performance evaluation entails a mix of mutually reinforcing qualitative methods that reflect the project logic and research questions being addressed. Given the important role of the DRG Center’s CSM approaches for the Leader award and the local contexts in CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │68 shaping the AA activities, the qualitative analysis will be focused on the DRG Center information and on the country case studies (carried out in depth through field visits) to provide concrete examples that illustrate in greater detail the process and results of the project. NORC’s approach to selecting the appropriate methodology is based on the USAID Evaluation Policy, our experience conducting evaluations in the field, and funding availability. NORC will conduct the SCS Global performance evaluation in a participatory manner, finalizing the evaluation design in consultation with USAID and the SCS Global Leader Implementing Partner and implementing the evaluation in close coordination and/or consultation with USAID, the implementing partner, and other stakeholders, as needed, through various phases of the evaluation. This includes working collaboratively to: (1) identify participants for key informant interviews (KIIs); (2) develop questions and protocols for KIIs; (3) review existing data available from the SCS Global; and (4) reach consensus on conclusions and recommendations based on the evaluation’s findings. Included in Annex A is the Evaluation Design Matrix. Qualitative Data Collection NORC’s evaluation of SCS Global will rely on qualitative data collection and analysis. Qualitative approaches can address aspects of research questions that are not well suited to quantitative analysis, and there are limits on the depth of information that can be collected cost-effectively through surveys given the dispersed nature of the respondents and the context specific nature of SCS Global activities. Qualitative approaches are well suited to investigating complex processes, and in many cases qualitative research can reveal unanticipated project effects. For example, interview guides are flexible and can be modified up until the beginning of the actual interview (whereas survey instrument instruments must be developed and finalized well in advance). Finally, qualitative methods can support quantitative methods: they can explore quantitative findings with weak statistical significance and they can provide context (or counter￾examples) for surprising quantitative results. Qualitative data collection and analysis will include the following: • Desk-top review of materials related to SCS Global, such as the Leader Award Agreement, select AAs, Co-Creation information, and the following documents primarily related to the LWA and select AAs: Performance Management Plan (PMP), MEL Plans, CLA documents, annual work plans, quarterly and annual reports23, country assessments, trip reports, local partner assessments, and training materials, “thought leadership” related publications, and other materials. USAID CDCSs or, as relevant, Regional Development Cooperation Strategies will also be reviewed. • KIIs with USAID, SCS Global staff and consultants and other subject matter experts (in the U.S. and based overseas) who work directly with local partner organizations, representatives of local partner organizations cooperating with SCS Global, and those who work in similar projects/programs. NORC will develop semi-structured, open-ended interview guides for the KIIs that can be modified as additional information is acquired in the field. Respondents will be identified by SCS Global based on the countries and local partner organizations selected for the case studies. The methodology used in conducting the KIIs is outlined in Exhibit 1, below. 23 To the extent possible, the evaluation will review and assess progress on how the Leader Award and the three case study AAs performed related to meeting/not meeting PMP indicator targets. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │69 Exhibit 1: Methodology for Conducting KIIs • KII guides for various classes of stakeholders will be prepared in advance of fieldwork, outlining the key questions and probes to address during the interview; the guides may be adapted before each individual interview. KII questions will be developed during the Desk Review period and USAID will be consulted to discuss those questions. • The interviewer will take written notes during the interview, and within a day of the interview transcribe those interview notes. • Respondents will be offered anonymity with the option of speaking “off the record,” and where such protection is requested, data will be coded and stored so that the respondent’s identity cannot be matched to the interview content, except by the interviewer. • The interview will start with more structured questions and pre-determined topics and will end with more open-ended questions that may reveal unanticipated topics and themes. • Analysis of the KIIs will include a summary of common themes that emerge in response to each topic in addition to “outliers” – highlighting the range of responses and experiences. This will be supported by quotes from the participants. Analysis of the data will include such approaches as tracing, for example, how SCS Global has helped to support citizen participation, improved government responsiveness to citizen demands, and access to information and quality independent media. NORC will undertake a comparison of achievements funded through SCS Global against reasonable normative expectations or standard best practices of democratic governance including indications of participation, inclusion, transparency, and accountability. The approach will be useful in developing case studies in the three countries identified for field visits (Djibouti, Kenya, and Zambia), given the complexity related to collecting such data with respect to considerations of budget and time allowed for this mid-term evaluation. With respect to the PMP and M&E systems, NORC’s analysis will focus on the relevance and effectiveness of monitoring data in managing SCS Global and responding to changing contexts and priorities. Careful consideration will be given to the overall utility of the MEL approaches, how FHI 360 and its lead partners make use of monitoring data collected to date, and how and whether it informs meaningful CLA of SCS Global’s goals. Case Studies NORC’s evaluation team will develop three illustrative case studies per the guidance provided in USAID’s Tasking Request (one 2-page case study product related to co-creation findings (Kenya), and two 5-page case studies (Zambia and Djibouti) to help to bring the project’s work “to life” and help the reader understand the context and operating environment in which the project takes place. The case studies will be carried out through field work in Kenya, Djibouti, and Zambia and will offer an operationalized understanding of how the LWA has benefited civil society and independent media programming. NORC’s evaluation team will travel to and work with local partners and beneficiaries to carry out interviews, conduct site visits, and observe and listen to local stakeholders. As indicated above, interview questions will be developed during the Desk Review and USAID will be consulted to review them. The focus of the Kenya case study will be on how the AA utilized the co-creation process and stimulated innovation. The Zambia and Djibouti case studies will more generally analyze multiple elements of the design and implementation of those AAs. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │70 The evaluation also will feature emerging themes and show how the project’s work demonstrates impact and achievement. These illustrative case studies are meant to be descriptive in character and intended to add realistic examples to the study. They will be narrative in style and feature different issues of interest to USAID and other evaluation readers. The case studies will examine lessons learned and demonstrate whether the activities in question did or did not meet the indicated outcomes of SCS Global’s design. They also will document key learning points as they pertain to project logic and implementation. The information needed to develop the case studies will be based on the document review, the KIIs, and direct observation conducted during site visits. From each case study country, NORC’s evaluation team will select a sample of local partner organizations to be interviewed for the case study. The sample will be chosen to achieve a balance of types of organizations and support received and duration of collaboration with SCS Global to reflect the overall population of participating local partner organizations. Given limited time for fieldwork, AA partners and beneficiaries will be requested to be available in the capital city for interviews with the evaluation team. Required Client Inputs To facilitate the evaluation, NORC will seek SCS Global’s assistance to obtain the following inputs:24 • Project documents including Leader and case study AAs: agreement program descriptions, annual work plans, and quarterly and annual reports as noted above beyond those already provided by SCS Global. Other Project documents required include project’s Co-Creation process documents, Performance Management Plan (PMP) and MEL Plans, and CLA-related documents. USAID CDCSs or, as relevant, Regional Development • Cooperation Strategies will also be required. If AAs are included in the results achievement frameworks of Mission Projects, Project Appraisal Documents (PADs) may also be reviewed. Detailed descriptions of any organizational capacity building and outcomes of that work as measured or assessed will be particularly important from those or other reports. NORC will develop a detailed list with the type, extent (intensity), dates of assistance to local partners and any project measures of local partner success/compliance with SCS-advocated approaches and tools based on these materials and USAID and the Leader may be requested to review the list for accuracy. • Lists of potential respondents with email and telephone contact information and assistance in establishing successful contact with local partner organizations selected for case studies. These respondents will include SCS Global staff and consultants in the U.S. and overseas and staff/representatives of local partner organizations and collaborating technology developers. • Assistance in sending email invitations and gaining cooperation with local partners invited to participate in the planned web-based survey. • All published, grey, and other literature (blogs and other web-based communications) that demonstrate SCS Global’s achievements in the advancement of thought leadership that the LWA and AA partners can provide. 24 NORC may need to add to this list as the evaluation is implemented. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │71 RESEARCH LIMITATIONS It is important to identify general limitations inherent to evaluation design: • Data availability and data quality. While the implementer and evaluation team will collect and generate primary data, some data may be difficult to obtain or be of questionable quality. It also is likely that not all respondents targeted for interviews will be available or be willing to participate. • Recall bias. Since a number of questions raised during the interviews will address issues or activities from the past, recall bias may affect their response. • Halo bias. There is a known tendency among respondents to under-report socially undesirable answers and alter their responses to approximate what they perceive as the social norm (halo bias). The extent to which respondents will be prepared to reveal their true opinions may also vary for some questions that call upon the respondents to assess the attitudes and perceptions of their colleagues or people on whom they depend upon for the provision of services. To mitigate this limitation, the evaluation team will provide appropriate confidentiality and anonymity assurances to all KIIs respondents who desire them and to all survey respondents through an informed consent process. • Case Study Participation Representativeness/Generalizability. Conduct of the case studies risks lack of representation of sufficient diversity in organizations and individuals who will be contacted and included. However, this is a normal risk considering the nature of the project. RESEARCH TIMELINE AND DELIVERABLES NORC’s evaluation team’s anticipated work schedule for the SCS Global evaluation is presented below and assumes COR’s approval of the Concept Note and commencement of the evaluation work on/about January 2, 2020. NORC’s evaluation team will meet weekly and more frequently as needed throughout the duration of the assignment, overseeing the process and ensuring smooth progress of the evaluation. Weeks 1-4: Planning, Desk Study, Preparation, and Washington KIIs and Meetings During the first four weeks, NORC’s evaluation team will complete planning, carry out a desk study of SCS Global documents, and prepare for Washington, D.C.based KIIs and meetings, as well as arrange logistics and travel approval requests for the three evaluation case studies. This will include reviewing documents; conducting interviews with U.S-based SCS Global staff; scheduling and arranging logistics for field visits and remote interviews; creating respondent contact lists and contacting protocols; and developing KII guides for Washington staff and those related to each case study. NORC also will develop a detailed work plan based on the agreed-upon schedule of field visits by evaluation team members. The evaluation team intends to schedule an in-brief with USAID and field representatives as well as key staff from SCS Global (as deemed appropriate) to clarify schedules, review the work plan and key monitoring and evaluation related plans and documents, and to better understand USAID’s strategic, objectives, and goals for funding SCS Global and for this performance evaluation. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │72 Weeks 5-7: Case Study Field Work Field work for each of the three country case studies will take place in February; each field visit is expected to include up to 5 days (one week trip) for each country (o/a February 3-21). It is expected that staff from AA implementing partners and other beneficiary organizations will be interviewed during the course of each case study. KIIs and discussions will advance qualitative data collection. Review and analysis of the LWA and AA PMP and MEL plans will also take place during this period. NORC’s evaluation team suggests meeting with the USAID Mission for each field visit to provide an in-brief and/or out-brief. These meetings will be proposed and confirmed with USAID missions as far in advance of field travel as possible. Weeks 8-11 Data Analysis and Reporting The evaluation team will complete the draft evaluation report within three weeks of completing field work. The draft final report summarizing findings and recommendation, will be presented March 20, 2020, and not exceed 25 pages, with annexes, and include a co-creation 2-page report on the Kenya AA and two 5-page case studies on the Djibouti and Zambia AAs. Weeks 12-15: USAID Review, Report Revision, and Presentation USAID will have two weeks to provide NORC with any edits and comments on the final report draft by April 10, 2020. NORC will submit a final document by April 17, 2020. A final presentation will be made to USAID on evaluation findings, conclusions, and recommendations o/a April 20, 2020. PROPOSED STAFFING NORC has selected a well-qualified team to conduct the SCS Global performance evaluation. The team will be led by Senior Level Evaluation Expert Susan Abbott who will provide subject area expertise and guide the substantive aspect of further developing the research questions and their adaptation into the evaluation protocols. Senior Level Evaluation Expert David Hess will be responsible for the analysis of the SCS Global PMP and M&E systems and oversee Ms. Abbott’s conduct of the evaluation and remote case studies. Susan Abbott (Senior Level Evaluation Expert – Performance Evaluation). Ms. Abbott has worked extensively with media, including projects with internet communications, human rights, civil society organizations, and media development. She has worked on multiple globally focused programs and consultancies with fieldwork in Central Europe, the Balkans, the Middle East, and Africa, both as a program manager and as an independent consultant leading and participating in technical assistance, training, and monitoring and evaluation around information management and information security issues for human rights and civil society organizations. Ms. Abbott spent 12 years in full time employment working on all these issues at Internews, the University of Pennsylvania, and the International Research and Exchanges Board prior to engaging in an impressive number of consultancies working with U.S. and other international development institutions. Many of her consultancies have involved conducting evaluations of international media strengthening and security improvement efforts. She has an array of highly relevant and impressive publications both as an author and editor. She has a deep understanding of what the needs and requirements are of vulnerable NGOs, journalists, and human rights defenders with regards to information management CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │73 and security. She has analyzed the impact of their projects and her analysis has illuminated important issues about project successes and challenges in reaching their objectives and how to improve their performance for a number of important donor organizations. David Hess (Senior Level Evaluation Expert – Performance Evaluation). Dr. Hess has over 39 years of experience monitoring and evaluating foreign assistance for USAID. He has conducted MEL assignments across a wide variety of sectors in nearly 20 countries, especially in Latin America, Asia, and Africa. Dr. Hess has spent more than 15 years working abroad, and has significant experience advising and training foreign assistance managers in designing and implementing MEL systems. Dr. Hess trained USAID staff in Liberia, Tanzania, Mozambique, Morocco, Bolivia, Paraguay, Vietnam, Cambodia, and Indonesia on MEL approaches within strategy and project design and evaluation. He has led design, implementation, and evaluation of USAID projects, including data collection, analysis and reporting, and training/supervision of field teams. Dr. Hess’s research has made extensive use of mixed methods, particularly qualitative methods, including in-depth interviews and focus groups. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │74 ANNEX B1: CONCEPT NOTE EVALUATION DESIGN MATRIX Evaluation Questions Evaluation Sub￾Questions Data Source Methodology Analysis 1. How has the SCS Global LWA been effective in advancing priorities in CSM programming? 1.a What are most significant achievements, intended and unintended? 1.b What are key weaknesses of the program? Major challenges? 1.c To what extent has SCS Global influenced thought leadership within the broader development and DRG Community on CS and media￾related issues? How effective has SCS Global been at ensuring integration of best practices, tools, or program methods beyond their immediate sphere of influence (i.e., within other SCS Global or FHI awards)? What more could be done to have an impact? What factors facilitated or impeded this impact? 1.d Have USAID missions and DRG priorities and objectives in country level programming been consistently and accurately reflected in awarded programs? If not, what could be done to strengthen alignment? 1.e How do case study results speak to similarities and differences 1.f How has the leader award (as opposed to the associate awards under the leader) been used to achieve related program objectives and results (Leader Obj’s 1.1 and Obj. 1.2? 1.g Are there programming needs in the area of CSM that are not effectively addressed through the SCS Global LWA? PMPs, Activity Reports, Quarterly and Annual reports, Project Work Plans, MEL Plans, CDCSs, RDCSs, DRG Center Strategy SCS staff International stakeholders / local partners staff Accessible stakeholder/ local partner reports Published, grey, and online literature and information Document review In-person & remote interviews Analyze and compare project targets and outcomes Correlate results from SCS Global and local partner reports Qualitative Review of KIIs CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │75 Evaluation Questions Evaluation Sub￾Questions Data Source Methodology Analysis 2. Was the design and resource allocation within the SCS Global LWA conducive to USAID’s needs in CSM programming? 2.a Has the focus on thought/technical leadership and capacity building been consistent with USAID’s original vision? How have each of these subfocus programs been implemented to date? 2.b Was the allocation of these resources and prioritization of these resources appropriate? 2.c How has the SCS LWA dealt with emerging needs and other priorities not anticipated during the program design? 2.d What are the strengths and weaknesses of the organization and management structures of the current LWA mechanism? 2.e Are the consortium structure and arrangements providing maximum benefit and value for USAID? 2.f Are key consortium partners, and sub-partners, including regional partners, used effectively and their relative expertise made available to USAID? 2.g Is communication between USAID and the consortium facilitated by the management structure – both within the consortium and within DRG? 2.h Has the consortium’s management of the current LWA, including division of programming, communications with USAID and external audiences, and reporting and other obligations proven effective – why or why not? 2.i Is USAID DRG responsive to the management needs of the SCS LWA, including substantial involvement? 2.j How effective has SCS Global been at integrating technical leadership developed under the Leader into the design and implementation of Associate Awards? Quarterly reports, regional and country reports DRG Center and SCS Global staff International stakeholders/ local partners Regional reports from other organizations/ funders Non-project organizations comparable to local partners (if possible) Third-party observers Site observations conducted by evaluation team Document review Web-based questionnaire to be filled out by (selected) local partners In-person and online interviews with (selected) local partners Local partner feedback / security audit forms Site observations Verify project reporting Qualitative analysis results KIIs Correlate analysis of local partners and comparison organizations Qualitative analysis of site observations Social Network Analysis CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │76 Evaluation Questions Evaluation Sub￾Questions Data Source Methodology Analysis 3. What are the strengths and weaknesses of the organization and management structures of the current LWA mechanism? 3.a What are the strengths and weaknesses of using a co￾creation process with Associate Awards? 3.b What are the effects of different forms of co-creation (pre-award or post-award), and do different forms have related or distinct strengths and weaknesses? 3.c How does the current SCS Global consortium structure affect or relate to co-creation processes? Quarterly reports, impact stories, regional and country reports SCS GLOBAL staff Beneficiaries' reports, interviews Third-party observers Document review Quarterly and regional reports Review of events related to beneficiaries and digital security In-depth interviews with local partners and third-party observers in the region Correlate SCS Global reports and assessment Compare and analyze required assistance against given assistance Assess benefits of project against increased risk 4. How has the SCS Global LWA utilize co-creation during development and implementation of Associate Awards? 4.a What are the strengths and weaknesses of using a co￾creation process with Associate awards? 4.b What are the effects of different forms of co-creation (pre- or post-award), and do different forms have related or distinct strengths or weaknesses? 4.c How does the current SCS-G consortium structure affect or relate to co-creation processes? Quarterly reports, impact stories, regional and country reports SCS GLOBAL staff Beneficiaries' reports, interviews Third-party observers Document review Quarterly and regional reports Reviewing events related to beneficiaries and co￾creation In-depth interviews with local partners and third-party observers in the region Correlate SCS Global reports and assessment with beneficiaries' responses and feedback Compare and analyze required assistance against given assistance Assess benefits of project co-creation process 5. How unique is SCS Global from similar DRG mechanisms? 5.a To what extent are the scope and capabilities of SCS Global significantly different from those available under similar DRG LWAs, such as the Human Rights Support Mechanism (HRSM)? 5.b Do there appear to be unique characteristics or focus areas under SCS-G that are not available under other DRG LWAs, such as HRSM or Global Elections and Political Transitions (GEPT) LWA? Quarterly reports, impact stories, regional and country reports DRG Center, SCS Global, HSRM, and GEPT staff reports, interviews Third-party observers Document review Quarterly and regional reports Review events related to beneficiaries and digital security Analyze KIIs about unique qualities of all mechanisms Analyze documents from all mechanisms CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │77 ANNEX C: EVALUATION QUESTIONS CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │78 ANNEX C: EVALUATION QUESTIONS EVALUATION QUESTIONS 5. How has the SCS Global LWA been effective in advancing USAID priorities in CSM programing? 1a. What are the most significant achievements, both intended and unintended? 1b. What are key weaknesses of the program? What are the major challenges? 1c. To what extent has SCS Global influenced thought leadership within the broader development and DRG community on civil society and media-related issues? How effective has SCS Global been at ensuring integration of best practices, tools, or program methods beyond their immediate sphere of influence (i.e., within other SCS Global or FHI awards)? What more could be done to have an impact? What factors facilitated or impeded this impact? 1d. Have USAID missions’ and DRG priorities and objectives in country-level programming been consistently and accurately reflected in the awarded programs? If not, what could be done to strengthen alignment? 1e. How do the case study results speak to similarities and differences between country programming in terms of meeting objectives? What are the notable innovations, including co-creation and engaging local partners, examined through the case studies? Is there evidence of CLA within country programming? Is there evidence of learning—for example, sharing of lessons and resources—across Mission programming? 1f. How has the Leader award (as opposed to AAs under the Leader) been used to achieve related program objectives and results (i.e., Leader Objective 1.1 and 1.2)? 1g. Are there programming needs in the area of CSM that are not effectively addressed through the SCS Global LWA? 6. Were the design and resource allocation within SCS Global LWA conducive to USAID’s needs in CMS programming? 2a. Has the focus on thought/technical leadership and capacity-building been consistent with USAID’s original vision? How have each of these sub-focus programs been implemented to date? 2b. Was the allocation of these resources and prioritization of these resources appropriate? 2c. How has the SCS LWA dealt with emerging needs and other priorities not anticipated during the program design? 2d. What are the strengths and weaknesses of the organization and management structures of the current LWA mechanism? 2e. Are the consortium structure and arrangements providing maximum benefit and value for USAID? 2f. Are key consortium partners, and sub-partners, including regional partners, used effectively and their relative expertise made available to USAID? 2g. Is communication between USAID and the consortium facilitated by the management structure— both within the consortium and within DRG? 2h. Has the consortium’s management of the current LWA, including division of programming, communications with USAID and external audiences, and reporting and other obligations proven effective—why or why not? 2i. Is USAID DRG responsive to the management needs of the SCS LWA, including substantial involvement? CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │79 EVALUATION QUESTIONS 2j. How effective has SCS Global been at integrating technical leadership developed under the Leader into the design and implementation of AAs? 7. How has the SCS Global LWA utilized co-creation during the development and implementation of AAs? 3a. What are the strengths and weaknesses of using a co-creation process with AAs? 3b. What are the effects of different forms of co-creation (pre-award or post-award), and do different forms have related or distinct strengths and weaknesses? 3c. How does the current SCS Global consortium structure affect or relate to co-creation processes? 8. How unique is SCS Global from similar DRG mechanisms? 4a. To what extent are the scope and capabilities of SCS Global significantly different from those available under similar DRG LWAs, such as the Human Rights Support Mechanism (HRSM)? 4b. Do there appear to be unique capabilities or areas of focus under SCS Global that are not available under other DRG LWAs, such as HRSM or the Global Elections and Political Transitions (GEPT) LWA? CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │80 ANNEX D: DATA COLLECTION INSTRUMENTS CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │81 ANNEX D: DATA COLLECTION INSTRUMENTS ASSOCIATE AWARDS – CIVIL SOCIETY AND MEDIA KII PROTOCOL Interview Guide: Associate Awards Informed Consent and Confidentiality: Hi, my name is Susan Abbott/ David Hess, and I work for NORC, which is an independent research company based in the Washington, DC area. We are collecting data about the SCS Global LWA that launched in May 2016 and runs through May 2021. We selected you and other respondents to interview because we understand that you may have perspective on the SCS Global LWA. We expect the duration of this interview to be one hour. We do not anticipate any risks to you as a result of your participation. Additionally, you will not receive any direct benefits by participating in the evaluation. However, your inputs may lead to recommendations that benefit actors engaged in DRG related work. All information that you share will be kept confidential. We will aggregate and present our findings to USAID in a way that cannot be attributed to any individual or organization. Therefore, please feel free to speak openly and candidly with us. Your participation is voluntary. Please feel free to ask to skip any question that you do not feel comfortable answering, end this interview at any point, or withdraw your responses after the interview. Do you confirm your consent to participate in this interview? ⃝Yes ⃝No Interview Date: Interviewer(s): Name(s): Sex: ⃝Female ⃝Male Affiliation: Location: General Information - Ownership and Type of Medium (1) Please tell us about your media outlet and the audience you intend to serve. Please briefly describe the history of this company, including when it was founded and any changes in focus since its founding. Probe: How would you describe your demographic market that your organization serves? EQ1) How has the SCS Global LWA been effective in advancing USAID priorities in CSM programing? (a) What are the most significant achievements, both intended and unintended? (b) What are key weaknesses of the program? What are the major challenges? (c) To what extent has SCS Global influenced thought leadership within the broader development and DRG community on civil society and media related issues? How effective has SCS Global been at ensuring integration of best practices, tools, or program methods beyond their immediate sphere of influence (i.e., within other SCS Global or FHI awards)? CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │82 What more could be done to have an impact? What factors facilitated or impeded this impact? (d) Have USAID missions’ and DRG priorities and objectives in country-level programming been consistently and accurately reflected in the awarded programs? If not, what could be done to strengthen alignment? (e) How do the case study results speak to similarities and differences between country programming in terms of meeting objectives? What are the notable innovations, including co-creation and engaging local partners, examined through the case studies? Is there evidence of CLA within country programming? Is there evidence of learning, for example sharing of lessons and resources, across Mission programming? (f) How has the Leader award (as opposed to Associate awards under the Leader) been used to achieve related program objectives and results (i.e., Leader Objective 1.1 and 1.2)? (g) Are there programming needs in the area of CSM that are not effectively addressed through the SCS Global LWA? EQ2) Were the design and resource allocation within the SCS Global LWA conducive to USAID’s needs in CSM programming? (a) Has the focus on thought/technical leadership and capacity building been consistent with USAID’s original vision? How have each of these subfocus programs been implemented to date? (b) Was the allocation of these resources and prioritization of these resources appropriate? (c) How has the SCS LWA dealt with emerging needs and other priorities not anticipated during the program design? (d) What are the strengths and weaknesses of the organization and management structures of the current LWA mechanism? (e) Are the consortium structure and arrangements providing maximum benefit and value for USAID? (f) Are key consortium partners, and sub-partners, including regional partners, used effectively and their relative expertise made available to (g) USAID? (h) Is communication between USAID and the consortium facilitated by the management structure – both within the consortium and within DRG? (i) Has the consortium’s management of the current LWA, including division of programming, communications with USAID and external audiences, and reporting and other obligations proven effective – why or why not? (j) Is USAID DRG responsive to the management needs of the SCS LWA, including substantial involvement? (k) How effective has SCS Global been at integrating technical leadership developed under the Leader into the design and implementation of Associate Awards? EQ3) How has the SCS Global LWA utilized co-creation during the development and implementation of Associate Awards? (a) What are the strengths and weaknesses of using a co-creation process with Associate Awards? CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │83 (b) What are the effects of different forms of co-creation (pre-award or post-award), and do different forms have related or distinct strengths and weaknesses? (c) How does the current SCS Global consortium structure affect or relate to co-creation processes? EQ4) How unique is SCS Global from similar DRG mechanisms? a) To what extent are the scope and capabilities of SCS Global significantly different from those available under similar DRG LWAs, such as the Human Rights Support Mechanism (HRSM)? b) Do there appear to be unique capabilities or areas of focus under SCS Global that are not available under other DRG LWAs, such as HRSM or the Global Elections and Political Transitions (GEPT) LWA? Cooperation Focus and Experience (2) Tell us about your cooperation with SCS Global. When did you begin working with the program? What have been the areas of cooperation? Probe: What type of cooperation have been useful/less useful? Why? Probe: Can you provide any examples? (3) To what extent has SCS Global been flexible and adaptive in working with you and responsive to your needs as a CSO or media organization? Can you give examples? FHI 360 KII PROTOCOL Interview Guide: FHI Informed Consent and Confidentiality: Hi, my name is Susan Abbott/ David Hess, and I work for NORC, which is an independent research company based in the Washington, DC area. We are collecting data about the USAID/FHI 360 SCS Global that began in May 2016 and runs through May 2021. The purpose of our interview is to get your feedback as part of a mid-term evaluation we are doing for USAID. We selected you and other respondents to interview because we understand that you may have perspective on the SCS Global LWA mechanism. We expect the duration of this interview to be one hour. We do not anticipate any risks to you as a result of your participation. Additionally, you will not receive any direct benefits by participating in the evaluation. However, your inputs may lead to recommendations that benefit actors engaged in administering the SCS Global LWA. All information that you share will be kept confidential. We will aggregate and present our findings to USAID in a way that cannot be attributed to any individual or organization. Therefore, please feel free to speak openly and candidly with us. Your participation is voluntary. Please feel free to ask to skip any question that you do not feel comfortable answering, end this interview at any point, or withdraw your responses after the interview. Do you confirm your consent to participate in this interview? ⃝Yes ⃝No Interview Date: Interviewer(s): Name(s): CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │84 Sex: ⃝Female ⃝Male Affiliation: Location: May I begin? Questions for FHI 360: Background 1) Describe your role in the design and implementation effort. When did you begin working with FHI 360? Which aspects of SCS Global are you involved in? When did you join the program? Questions for EQ1 1) To the extent that you can comment, what do you see as USAID’s priorities in CSM programming, and how does the SCS Global mechanism support this? 2) How has the Leader award delivered on its primary objectives (For reference – Leader Objective 1.1: Develop and promote effective approaches to the strengthening of civil society and independent media actors. Leader Objective 1.2: Effective management of SCS Global LWA) Go through Objective 1 and Objective 2 with the Performance Indicator Tracking Table – with the Chief of Party and the M&E officer. 3) How have FHI 360’s technical expertise and knowledge projects helped to develop and promote effective approaches to the strengthening of civil society and independent media actors? 4) How has the SCS Global mechanism contributed to USAID’s Learning Agenda around CSM programs and leadership? Can you share feedback from the process that FHI 360 has led on how shaping the Learning Agenda? What have you learned? 5) What has your experience been as the Leader in managing the AAs and coordinating with the DRG sector at-large, i.e. the SCS Associate Awards, consortium partners, USAID, and practitioners who work to strengthen civil society and independent media. 6) What are FHI 360’s approaches to developing and sharing knowledge products and learning events and what are your key delivery methods for getting this information out to the stakeholders who need them? a) How have you measured their use and uptake? Do you have any evidence to indicate that they are making a difference and being used? b) In terms of dissemination and deliver of the knowledge products or the hosting of the learning events, are there aspects that could have been done differently? (process effectiveness) c) Can you share any analytics from the NGOConnect site? How do you use Google Analytics to inform the site’s content and delivery? d) What has been the reaction from stakeholders related to the “use case” studies – clarify whether these have been produced or if they are under development? e) What types of innovations in the DRG space has the SCS Global LWA supported? Are you able to comment on any feedback or lessons learned related to how SCS Global can support processes and approaches like co-creation, collaborative learning and adaptation, etc.) 7) In your opinion, what are the most significant achievements of SCG Global thus far? CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │85 8) What are the relative strengths and weaknesses of the SCS Global Mechanism? 9) Let’s turn to the Associate Awards now. To the extent you can answer, from your perspective, have USAID missions’ and DRG priorities and objectives been reflected in the awarded programs? a) Can you comment further on how and why? b) If not, what could be done to change this? c) Which of the programs stands out to you as the most successful/ least successful? 10) To the extent that you are able to answer, are there programming needs in the area of CSM that are not effectively addressed through the SCS Global LWA? EQ2) Were the design and resource allocation within the SCS Global LWA conducive to USAID’s needs in CSM programming? 1) How well are the resources being used? Have you done a cost benefit analysis to determine whether the resources allocated to the SCS Global LWA are being used in the right way? 2) Please comment on FHI 360’s set-up and program management systems in terms of overarching purpose and goal of the SCS Global LWA? a) Do you feel that the consortium’s management of the current LWA, including division of programming, communications with USAID and external audiences, and reporting and other obligations proven effective – why or why not? b) What effect do the set-up and systems have on the FHI 360’s strategic approach? c) Is there anything you’d change? 3) How is the SCS Global LWA set-up in terms of the management and communication systems needed to support the needs of the DRG CSM sector – for USAID, the implementers and the local media and civil society they support? a) How does FHI 360 make the broader network of partners and the consortium available/ known to USAID? b) How has FHI 360 approached integrating technical leadership developed – as developed and led by FHI 360 – into the design and implementation of Associate Awards? c) How has SCS Global responded to emerging needs and other priorities not anticipated during the program design? Is there any flexibility for integrating new ideas? 4) How has FHI 360’s experience been with USAID DRG in terms of supporting the overall management of the SCS LWA? What can be improved? EQ3) How has the SCS Global LWA utilized co-creation during the development and implementation of Associate Awards? 1) What are the strengths and weaknesses of the co-creation process? Do you have any success stories? What difference does co-creation make in terms of the overall AA that gets implemented? 2) What are the forms of co-creation (pre-award or post-award), and do different forms have related or distinct strengths and weaknesses? 3) How does the current SCS Global consortium structure affect or relate to co-creation processes? CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │86 EQ4) How unique is SCS Global from similar DRG mechanisms? 1) To the extent that you can answer, how is SCS Global different from other DRG LWA? 2) What is unique about SCS Global USAID STAFF/ KII PROTOCOL Interview Guide USAID Staff Informed Consent and Confidentiality: Hi, my name is Susan Abbott/ David Hess, and I work for NORC, which is an independent research company based in the Washington, DC area. We are collecting data about the SCS Global LWA that launched in May 2016 and runs through May 2021. We selected you and other respondents to interview because we understand that you may have perspective on the SCS Global LWA. We expect the duration of this interview to be one hour. All information that you share will be kept confidential. We will aggregate and present our findings to USAID in a way that cannot be attributed to any individual or organization. Therefore, please feel free to speak openly and candidly with us. Your participation is voluntary. Please feel free to ask to skip any question that you do not feel comfortable answering, end this interview at any point, or withdraw your responses after the interview. Do you confirm your consent to participate in this interview? ⃝Yes ⃝No Interview Date: Interviewer(s): Name(s): Sex: ⃝Female ⃝Male Affiliation: Location: May I begin? Background 1) Please tell us about yourself and your role at USAID/ DRG related programs. 2) In your work at USAID, do you have a specific role vis-à-vis oversight and administration of the SCS Global LWA – if so, can you explain this? EQ1) How has the SCS Global LWA been effective in advancing USAID priorities in CSM programing? Questions for EQ1: 1) To the extent that you can comment, what are USAID’s priorities in CSM programming, and how does the SCS Global mechanism support this? 2) How has the Leader award delivered on its primary objectives (For reference – Leader Objective 1.1: Develop and promote effective approaches to the strengthening of civil society and independent media actors. Leader Objective 1.2: Effective management of SCS Global LWA – interviewer should have a list of the ) CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │87 a) Please comment on whether and how FHI 360’s technical expertise and knowledge projects have helped to develop and promote effective approaches to the strengthening of civil society and independent media actors. b) Has FHI 360 has effectively managed the SCA Global LWA in terms with the SCS Associate Awards, consortium partners, USAID, and practitioners who work to strengthen civil society and independent media. c) Is SCS Global delivering on outputs and outcomes (knowledge products and learning events) as planned, i.e. in the ways that you expected? (gets at the efficiency and effectiveness of the program.) 3) Has FHI 360’s approaches to developing and sharing knowledge products and learning events and their delivery methods been effective? Are there aspects that could have been done differently? (process effectiveness) 4) In your opinion, what are the most significant achievements of SCG Global thus far? 5) What are the relative strengths and weaknesses of the SCS Global Mechanism? 6) Are you familiar with the evidence, research, M&E related materials or other thought leadership generated by the SCS Global Mechanism (by FHI 360)? If yes, then? a) The NGOConnect website. Do you go to the site and access information related to your work there? b) Have you read and used the case studies? How have they informed your thinking about DRG programming? c) Have any of the tools, best practices, or other program methods that you have learned about through SGS Global influenced your work? Please explain. d) Are there any notable innovations in terms of methods, approaches or practices to DRG program implementation and monitoring, evaluation, research and learning that you have learned about from the SCG Global mechanism? Please share with me what these are. How have you used them? What difference did these tools and methods make to your work or the work you support? e) What about Collaborative Learning and Adapting – has the SCG Global mechanism and its range of support made a difference to your CLA practices? 7) Let’s turn to the Associate Awards now. To the extent you can answer, from your perspective, have USAID missions’ and DRG priorities and objectives been reflected in the awarded programs? a) Can you comment further on how and why? b) If not, what could be done to change this? c) Which of the programs stands out to you as the most successful/ least successful? 8) To the extent that you are able to answer, are there programming needs in the area of CSM that are not effectively addressed through the SCS Global LWA? EQ2) Were the design and resource allocation within the SCS Global LWA conducive to USAID’s needs in CSM programming? Questions for EQ2: CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │88 1) How well are the resources being used? Do you think the resources allocated to the SCS Global LWA being used in the right way? a) Was the allocation of these resources and prioritization of these resources appropriate? b) Do you think that the consortium structure and arrangements provide maximum benefit and value for USAID? (Ask how and why) c) Has the focus on thought/technical leadership and capacity building been consistent with USAID’s original vision? d) How have each of these sub-focus programs been implemented to date? 2) How effective and efficient is FHI 360’s set-up and program management systems in terms of overarching purpose and goal of the SCS Global LWA? a) Has the consortium’s management of the current LWA, including division of programming, communications with USAID and external audiences, and reporting and other obligations proven effective – why or why not? b) What effect do the set-up and systems have on the FHI 360’s strategic approach? 3) How suited is the SCS Global LWA set-up and systems to the needs of the DRG CSM sector – for USAID, the implementers and the local media and civil society they support? a) How would you rate the use and involvement of key consortium partners, and sub-partners? Do you think they are used effectively? b) Is the expertise of the broader network of partners and the consortium made available to USAID? c) How effective has SCS Global been at integrating technical leadership developed by FHI 360 into the design and implementation of Associate Awards? d) How has the SCS LWA dealt with emerging needs and other priorities not anticipated during the program design? 4) What has your experience been with the SCS Global LWA set-up in terms of the management and communication systems? a) Is communication between USAID and the consortium adequately facilitated by the management structure – both within the consortium and within DRG? b) Do you feel that you have the information you need about the broader network of partners and the consortium available? c) Do you think SCS Global has done an adequate job of integrating emerging needs and other priorities not anticipated during the program design? Is there any flexibility for integrating new ideas? d) How could FHI 360 strengthen and improve its organizational set-up and program management systems for SCS Global in the future? EQ3) How has the SCS Global LWA utilized co-creation during the development and implementation of Associate Awards? EQ4) How unique is SCS Global from similar DRG mechanisms? 1) To the extent that you can answer, how is SCS Global different from other DRG LWA? 2) What is unique about SCS Global? CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │89 ANNEX E: LIST OF PERSONS INTERVIEWED CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │90 ANNEX E: LIST OF PERSONS INTERVIEWED Washington, DC – January 20, 2020 Name Organization Area Title Barney Singer SCS Global Leadership Civil Society and Leadership and Capacity Development Programs Director, SCS Global LWA Lisa Peterson SCS Global Leadership Civil Society Strengthening Technical Director Michael Kott SCS Global Leadership Civil Society and Peacebuilding Department Director Zach Andersson SCS Global Support/Management Monitoring, Evaluation, and Learning Marie Bellon SCS Global Support/Management SCS Djibouti Kristin Brady SCS Global Support/Management SCS Global Product – Emerging Technologies and Digital Threats Workstream Lydia Cordes SCS Global Support/Management SCS Kenya Theo Dolan SCS Global Support/Management SCS Global Product – CVE Reference Guide; SCS Kenya Deb Ensor SCS Global Support/Management SCS Global Product – Emerging Technologies and Digital Threats Workstream Alex Levy SCS Global Support/Management SCS Global Product – Youth Programming Assessment Tool; SCS Zambia Anne Salinas SCS Global Support/Management SCS Global Product – CVE Reference Guide Meriam Sassi SCS Global Support/Management Monitoring, Evaluation, and Learning Megan Scanlon SCS Global Support/Management SCS Global Product – Local Philanthropy and Self Reliance Abbie Wilson SCS Global Support/Management SCS Djibouti Natalia Bourjaily ICNL Vice President￾Eurasia and Catherine Shea, Vice President - Programs International Center for Not-for-Profit Law Associate Awardee, Oversees development and publication of Civil Society Index Deborah Ensor Senior Vice President, Technical Leadership; Chris Wild, Business Development, and Myles Internews Associate Awardee, Oversees development and publication of media literacy tool and emerging technology tools CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │91 Name Organization Area Title Smith, Program Development Zambia, February 17-19, 2020 Name Organization Area Title Travis Gartner FHI 360 FHI 360 Technical Director Alice Bwalya FHI 360 FHI 360 Chief of Party Charlene Bangwe USAID Zambia USAID AOR Emily Fitzsimmons USAID Zambia USAID Governance Advisor Davis Musonda National Youth Development Council, Government of Zambia (Local Partner) National Youth Development Council, Government of Zambia (Local Partner) Council Secretary Brian Chipasha National Youth Development Council, Government of Zambia (Local Partner) National Youth Development Council, Government of Zambia (Local Partner) Senior Trainer and Internship Coordinator Margaret Chambeshi National Youth Development Council, Government of Zambia (Local Partner) National Youth Development Council, Government of Zambia (Local Partner) Monitoring and Evaluation Specialist Kathy Short Friedrich Eert Stiftung Zambia Advisory Board Members Program Manager Emmanuel Chisalu Auditor General's Office, Government of Zambia Advisory Board Members Auditor Towela Nyirenda FHI 360 Youth Lead Beneficiaries (Cohort 1) Intern Esther Sinda World Wildlife Fund Youth Lead Beneficiaries (Cohort 1) Intern Bryson Musonda Volunteer Youth Lead Beneficiaries (Cohort 1) Intern Nchimunya Chadukwa Counterpart International Youth Lead Beneficiaries (Cohort 1) Consultant Gerald Sikazwe Bongohive Business Incubator Youth Lead Beneficiaries (Cohort 1) Consultant Yvonne Sikombe Medical Store Zambia Youth Lead Beneficiaries (Cohort 1) Sales and Marketing Officer Chinyama Luse Shamambo Volunteer Youth Lead Beneficiaries (Cohort 1) Volunteer Mwila Bwanga Volunteer Youth Lead Beneficiaries (Cohort 2) Intern Verronica Namukonda Zambia Center for Communications Change Youth Lead Beneficiaries (Cohort 2) Accountant Mpundu Mwenya Plan International Youth Lead Beneficiaries (Cohort 2) Program Officer Ian Banda Apters Insurance Youth Lead Beneficiaries (Cohort 2) Program Officer CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │92 Name Organization Area Title Annie Mwamba Crown Agents Youth Lead Beneficiaries (Cohort 2) Communications Officer Naadirah Latife Society for Family Health Youth Lead Beneficiaries (Cohort 3) Program Officer Peggy Nyambe Global Health Corps Youth Lead Beneficiaries (Cohort 3) Communications Officer Hastings Chafwakale Ministry of Local Government Youth Lead Beneficiaries (Cohort 3) Program Officer Tamara Moyo Center for Infectious Disease Research Youth Lead Host Organizations HR Manager Chikwe Antoinette Mbweeda Children International Youth Lead Host Organizations Country Director Carol Milambo Mufana Clinton Health Access Initiative Youth Lead Host Organizations Program Manager Mwangala Chibbabbuka Expanded Church Response Youth Lead Host Organizations Chief of Party Humphrey Chishala Lusaka Chamber of Commerce and Industry Youth Lead Host Organizations Director Sylvester Kalonge iDE Zambia Youth Lead Host Organizations Country Director Anna Cestari World Bank Youth Lead Host Organizations Water Resources Specialist Barnabas Chiboboka Zambia Center for Communications Change Youth Lead Host Organizations Program Manager Miselo Bwalya Zambia Institute Policy Analysis & Research Youth Lead Host Organizations Associate Researcher Misozi Siwela Pact Youth Lead Host Organizations Senior DREAMS Manager Shakerrie Almond Virtual Doctors Youth Lead Host Organizations Director Fannie Nthakomwa Democracy Works Youth Lead Host Organizations Country Director Kenya, February 21-25, 2020 Name Organization Area Title Zephania Aura USAID Kenya USAID AOR Faye Haselkorn USAID Kenya USAID Alternate AOR/Deputy Chief Stephen Brager USAID Kenya USAID Chief Phyllis Ombonyo FHI 360 SADES-K SADES-K Staff SADES-K CoP Caroline Watheka FHI 360 SADES-K SADES-K Staff SADES-K Grants Manager Maureen Omondi FHI 360 SADES-K SADES-K Staff SADES-K M&E Officer Shitemi Khamadi Internews SADES-K Staff Media Specialist Leah Kimathi Independent Consultant Co-Creation Consultant Consultant Amb. Stephen Karau Building Bridges to Unity (BBI) Advisory Taskforce/Secretariat BBI Taskforce BBI Taskforce Chief Administrator CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │93 Name Organization Area Title Frankline Mukwanja Centre of Multi Party - Democracy SADES-K Grantees (Result Area 1 - Governance Reform) Executive Director Abshiro Halake Centre of Multi Party - Democracy SADES-K Grantees (Result Area 1 - Governance Reform) CMD Chairperson Abdullahi Shandey Merti-Integrated Development Programme (MID-P) SADES-K Grantees (Result Area 2 - Social and Ethnic Cohesion Strengthened) Executive Director Martha Opilli Keeping Alive Socities Hope (KASH) Consortium SADES-K Grantees (Result Area 2 - Social and Ethnic Cohesion Strengthened) Programs Manager Elly Opondo Champions of Peace Kenya (CoP-K) SADES-K Grantees (Result Area 2 - Social and Ethnic Cohesion Strengthened) Executive Director Bwibo Adieri Anglican Development Services (ADS Kenya) SADES-K Grantees (Result Area 2 - Social and Ethnic Cohesion Strengthened) Executive Director Mercy Jerop Rural Women Peace Link (RWPL) SADES-K Grantees (Result Area 2 - Social and Ethnic Cohesion Strengthened) Executive Director Mercy Tumkou SIKOM Peace Network SADES-K Grantees (Result Area 2 - Social and Ethnic Cohesion Strengthened) Communications Director Phyllis Muema Kenya Community Support Centre (KECOSCE) SADES-K Grantees (Result Area 2 - Social and Ethnic Cohesion Strengthened) Executive Director Zed Mkalla Kenya Community Support Centre (KECOSCE) SADES-K Grantees (Result Area 2 - Social and Ethnic Cohesion Strengthened) Project Officer Chris Mbiti Poverty Eradication Network (PEN) Consortuim SADES-K Grantees (Result Area 3 - Civic & Democratic Spaces Protected) Executive Director Ludivicus Omollo Poverty Eradication Network (PEN) Consortuim SADES-K Grantees (Result Area 3 - Civic & Democratic Spaces Protected) Program Officer Judith Gachathi Civil Society Reference Group (CSRG) SADES-K Grantees (Result Area 3 - Civic & Democratic Spaces Protected) Program Manager CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │94 Name Organization Area Title Eunice Adhiambo Ujamaa Centre SADES-K Grantees (Result Area 3 - Civic & Democratic Spaces Protected) Executive Director David Omwoyo Media Council of Kenya (MCK) Consortium SADES-K Grantees (Result Area 3 - Civic & Democratic Spaces Protected) Executive Director Jacob Nyongesa Media Council of Kenya (MCK) Consortium SADES-K Grantees (Result Area 3 - Civic & Democratic Spaces Protected) Monitoring & Evaluation Eric Oduor Kenya Union of Journalists (KUJ) SADES-K Grantees (Result Area 3 - Civic & Democratic Spaces Protected) Secretary General Rosalia Omungo Kenya Editors Guild (KEG) SADES-K Grantees (Result Area 3 - Civic & Democratic Spaces Protected) Acting CEO Marcelline Nyambala Association of Media Women in Kenya (AMWIK) SADES-K Grantees (Result Area 3 - Civic & Democratic Spaces Protected) Executive Director Hassan Abdi Abdille Muslim Human Rights (MUHURI) SADES-K Grantees (Result Area 3 - Civic & Democratic Spaces Protected) Former Executive Director Beatrice Elachi EMBRACE Women Group In-Kind Support Secretariat Thomas Abol Keeping Alive Socities Hope (KASH) Consortium Grantees involved in the Co-Creation Process Executive Director Kenneth Anusu Institute for Civic Education & Development in Africa (ICEDA) Grantees involved in the Co-Creation Process Executive Director Joseph Akoule SIKOM Peace Network Grantees involved in the Co-Creation Process Executive Director Suba Churchill Civil Society Reference Group (CSRG) Grantees involved in the Co-Creation Process Presiding Convenor Gladys Nabiswa Community Research in Environment & Development Initiatives(CREADIS) Grantees involved in the Co-Creation Process Executive Director Toepista Nabusoba Kenya Union of Journalists (KUJ) Grantees involved in the Co-Creation Process Francis Auma Muslim Human Rights (MUHURI) Grantees involved in the Co-Creation Process Rapid Response Officer Prof. Karuti Kanyinga University of Nairobi￾Institute of Development Studies Institute of Development Studies Director IDS CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │95 Djibouti, February 26, 2020 Name Organization Area Title Alpha Souleymane Diallo USAID Ethiopia USAID Civil Society Organizations Strengthening Program AOR Loula Ali Elabe Djibouti Ministry of National Education (MENFOP) Djibouti Government Representatives Director of Basic Education Fatouma Ali Djibouti Ministry of Health Djibouti Government Representatives Director of Mother’s Health Fardoussa Ahmed Djibouti Ministry of Decentralization Djibouti Government Representatives Director Rokia Ali Djama Djibouti National Women's Union Partner Organizations Program Officer Daoud Houmed Mohamed Ecology of the Village of Adaïlou (Écologie du Village d’Adaïlou (EVA)) Partner Organizations Auditor General Mouna Aboubaker National Agency for People with Disabilities Partner Organizations Director of Prevention, Communication and Cooperation Omeyma Abdallah Sounna Al Haya Partner Organizations President Houlata-Abayazid￾Madina PRECAD Team Other Interviews held USAID, David Jacobstein USAID, Mark Goldenbaum Freedom House, Lisa Dickieson and Laura Adams, Freedom House is the Leader on another LWA – Human Rights Support Mechanism CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │96 ANNEX F: CONFLICT OF INTEREST DISCLOSURE CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │97 ANNEX F. DISCLOSURE CONFLICTS OF INTEREST Name Susan Abbot Title Senior Evaluation Expert Organization NORC at the University of Chicago Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) Prepared under Contract No.: GS-10F-0033M / AID-OAA-M￾13-00013, Tasking N030 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Strengthening Civil Society Globally (SCS Global) Performance Evaluation FHI 360 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1.Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2.Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3.Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4.Current or previous work experience or seeking employment CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │98 with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5.Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6.Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date January 6, 2020 CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │99 ANNEX G: OTHER DOCUMENTS CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │100 ANNEX G: OTHER DOCUMENTS SCS GLOBAL, INPUTS FOR PORTFOLIO REVIEW, OCTOBER 20, 2017 SCS Global Inputs for Portfolio Review October 20, 2017 Managing for Results For existing activities, are they achieving planned results? Which have been the most significant and why? SCS Leader Award. The objective / planned results of the leader award: to identify and development evidence-based, effective approaches to strengthen civil society and media in support of DRG and other development objectives. In year 1, the most significant results have been: • Completion of the Delphi process with more than 100 diverse civil society and independent media experts from all regions of the globe to define a research and learning agenda that would yield practical tools, approaches and methodologies to support practitioners. The process reached consensus to focus on five priority topic areas for investigation: – Closing Space – Press Repression/Media propaganda – Media literacy technology – Countering violent extremism – Informal Organizing • Design, award and initiation of two research projects: – Emerson College: Civic Media Literacy Toolkit to activists better analyze and use new media tools – ICNL: CSO Legal Advocacy Toolkit to prevent the use of new anti-money laundering / anti￾terrorism financial standards as a pre-text for closing civic space • Extensive negotiations on six additional research projects to support the research and learning agenda to ensure the designs are strong and would lead to outstanding knowledge products • The Capacity Development Interventions Guide, a program designer’s manual crafted to provide teams interested in designing and incorporating a capacity development components into their programs; contains techniques, insights from experience and references to additional information to make design decisions. USAID identified this as a gap among existing Agency resources. • Implementation TIPS for USAID Partners: updated a set of 45 easy-to-read articles in a Q&A format on organizational development (e.g., financial management, compliance) topics to help CSOs learn and improve their core systems. CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │101 The knowledge products designed and/or completed during this period directly contribute to filling critical learning needs and improving programming. The careful review of prospective research and learning initiatives ensured that only strong, very promising projects moved forward. It is also enabling USAID and FHI 360 to clarify for planning purposes that other products will need additional effort and as intentional approach to increase the likelihood that products being created will be well utilized. SCS Associate Awards. SCS is also intended to enable USAID to support civil and media organizations and programs around the world, as well as programs that infuse DRG principles (participation, social inclusion, transparency and accountability) in all sectors. The results achieved are as follows: • Designed, refined, awarded and begun implementation on six SCS Associate Awards – Jamaica Local Partner Development – supporting local civil society organizations and social enterprises to increase public safety and security and promote social justice. – Guinea Citizen Involvement in Heath Governance – supporting civic and media partners to aggregate and share citizen input on health service delivery, more effectively collaborate with government, and use evidence to advance health reform. – Tanzania Media and Civil Society Strengthening Activity – supporting an improved media enabling environment, strengthening independent media outlets and community radio stations, and more effective use of media by CSOs to advocate on key issues. TMCS has a strong emphasis on gender and youth integration. – Balkans Media Assistance Program – supporting media in five countries in the region to be more competitive in local and regional marketplaces, strengthen the sustainability of the independent media sector across the region, and foster stronger partnerships to address common challenges, especially in the digital space. – Kyrgyz Republic Cultivating Media Independence – supporting independent media partners and the sector overall to improve sustainability of the sector, the quality and relevance of content and the overall enabling environment. – Civil Society Organization Sustainability Index – conducting USAID’s annual review of the state of civil society in selected countries and refining USAID’s leading tool for monitoring and assessing civil society sustainability to address its limitations. • Two other activities in ongoing procurement – Djibouti SCO Strengthening Program – supporting civic organizations at the national and local levels to improve service delivery (in BAFO). – Haiti Youth Civic Engagement – supporting civic partners to improve youth civic knowledge, increase youth engagement, and strengthen the effectiveness of youth advocacy (proposal due mid-November). • Framework designed to facilitate learning across the leader and associate awards, among SCS consortium partners and beyond. The SCS LWA was designed to both improve the quality of civil society and media-focused programming to achieve development objectives and to serve as a mechanism through which USAID can implement CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │102 such programs. SCS is achieving both. Critical to SCS’s success in successfully negotiating with so many operating units has been its willingness to use its consortium partners (e.g., media-focused awards and CSOSI) and to bring on new partners critical to achieving an award’s objectives (e.g., one of the co￾implementers in the SCS Guinea award is outside the official SCS consortium but has unique expertise in Guinea). What are the most significant mechanism challenges/constraints (e.g. programmatic, management, bureaucratic, interagency, partner-related, etc.) There have been two primary challenges in implementing the SCS Leader Award: • The level of effort required to ensure that research and learning initiatives are well designed to meet a strategic need and use rigorous methodologies to lead to outstanding products of wide applicability to the development community • Balancing the tension for evidence-based products that must focus on specific contexts from which to gather that evidence with the desire for knowledge tools of wide applicability and the costs associated with collecting enough data to ensure wide applicability How are you addressing them? To address these challenges, USAID and the SCS team have agreed to: • Organize co-design processes for the research and learning initiatives for which we do not yet have strong enough designs. These co-design sessions will engage the DRG Learning Team, partners working on similar topics and select outside experts. We have determined that investing upfront in the design is critical to ensuring that the final product meets our quality standards. • Increase the funding available for each research and learning projects and reduce that number of anticipated products. The organizing principle is now “Fewer and Bigger” research projects with more robust designs that yield needed knowledge products. • Strengthen coordination with other DCHA learning initiatives (e.g., DRG Learning Team, Fellows Program, IRI’s Closing Spaces team) Please identify Standard F Indicator(s) that you have or will begin to report on for your activities in FY 2018. See: https://www.state.gov/f/indicators/ [We will discuss at our meeting] Jamaica LPD will report on the following indicators: • Number of human rights organizations trained and supported (F, DR 4.2-1) • Number of micro-enterprises supported by USG assistance (F, EG 5.3) • Number of USG-assisted civil society organizations (CSOs) that participate in legislative proceedings and/or engage in advocacy with national legislature and its committees (F, DR 4.3.1) One year from now, all Associate Awards will report on the following: (1) Number of public policies introduced, adopted, repealed, changed or implemented with citizen input (archived F, DR 2.4.1-12) (2) Number of USG-assisted civil society organizations (CSOs) that participate in legislative proceedings and/or engage in advocacy with national legislature and its committees (F, DR 4.3.1) CONTRACT NO. GS-10F-0033M / ORDER NO. 7200AA18M00016 / DRG-LER II TASKING N030 USAID.GOV SCS GLOBAL PERFORMANCE EVALUATION │103 (3) Number of civil society organizations (CSOs) receiving USG assistance engaged in advocacy interventions (F, DR 4.2-2 – note: will also include information from DR 5.3.-2 specific to media CSOs and DR 4.2-1 on human rights CSOs) (4) Number of human rights organizations trained and supported (F, DR 4.2-1) (5) Number of USG-assisted Media-Sector CSOs and Institutions that serve to strengthen the independent media or journalists (F, DR 5.3-2) (6) Number of training days provided to journalists with USG assistance, measured by person-days of training (F, DR 5.2-1) (7) Number of non-state news outlets assisted by USG (F, DR 5.3-1) (8) Number of USG-supported activities designed to promote or strengthen the civic participation of women (DR 4.1) Financial Management: For your activities, do the historic burn rates differ greatly from the anticipated burn rates in FY 2018? If so, why? Burn rates will accelerate in Year 2 as SCS’s activities are now more labor intensive and sub-awards for research projects are made. U.S. AGENCY FOR INTERNATIONAL DEVELOPMENT 1300 Pennsylvania Avenue, NW Washington, DC 20523