PICTURE CREDIT: KETI JAKELI GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT May 2021 Prepared under Contract No. GS-10F-0033M / Order No. 7200AA18M00016, Tasking N040 This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. DRG LEARNING, EVALUATION, AND RESEARCH ACTIVITY II GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT MAY 2021 Prepared under Contract No. GS-10F-0033M / Order No. 7200AA18M00016, Tasking N040 Submitted to: Brandy Witthoft, USAID COR Submitted by: Ritu Nayyar-Stone, Team Leader Orysia Lutseyvch, Civil Society Subject Matter Expert Tamara Sartania, Local Evaluation Specialist Gia Gotua, Local Evaluation Specialist Miguel Albornoz, Evaluation Specialist Ridhi Sahai, Qualitative Specialist Jack Devine, Quantitative Specialist Zumrat Salmorbekova, Evaluation Specialist and Reviewer Contractor: NORC at the University of Chicago 4350 East West Highway, 8th Floor Bethesda, MD 20814 Attention: Renée Hendley Tel: 301- 634-9489; E-mail: Hendley-Renee@norc.org DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | i TABLE OF CONTENTS ACRONYMS III EXECUTIVE SUMMARY 1 FINDINGS 2 CONCLUSIONS 4 RECOMMENDATIONS 4 1. EVALUATION PURPOSE AND EVALUATION QUESTIONS 6 2. PROJECT BACKGROUND 7 3. METHODS AND LIMITATIONS 9 DESK REVIEW 10 QUANTITATIVE DATA COLLECTION - SURVEY 10 QUALITATIVE DATA COLLECTION 11 LIMITATIONS 13 4. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 14 FINDINGS 14 CONCLUSIONS 51 RECOMMENDATIONS 53 ANNEX A: EVALUATION STATEMENT OF WORK 58 ANNEX B: DATA COLLECTION INSTRUMENT 60 ONLINE SURVEY: ACCESS GRANTEES AND TECHNICAL ASSISTANCE RECIPIENTS 60 MASTER GEORGIA/ACCESS QUESTION BANK 72 ANNEX C: SOURCES OF INFORMATION 91 DOCUMENTS REVIEWED 91 KEY INFORMANTS 92 ACCESS PMP SUMMARY 95 ANNEX D: DISCLOSURE CONFLICTS OF INTEREST 98 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | ii TABLES Table 1. Types and Number of Grants Disbursed by ACCESS 7 Table 2. Desk Review Documents 10 Table 3. KII Respondents 11 Table 4. Target FGD Respondents 12 Table 5. Main Challenges Faced in Engaging Citizens by ACCESS Grantees 20 Table 6. Groups Targeted by Grantee Survey Respondents 22 Table 7. Challenges for Citizen Engagement Overcome with New Methods 27 Table 8. ACCESS PMP Indicator Summary 95 FIGURES Figure 1. ACCESS Objectives and Anticipated Results 8 Figure 2. Top Six Ways of Engaging Citizens (% of respondents; multiple selection) 18 Figure 3. Most Notable Grant Results in Terms of Monitoring and/or Influencing Government Policies and Processes 18 Figure 4. Grants Received by Survey Respondents 23 Figure 5. Average Years of Operation for ACCESS Grantees by Year of First Grant 24 Figure 6. Grantees Who Previously Operated Informally by Year of First Grant 25 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | iii ACRONYMS ACCESS Advancing CSO Capacities and Engaging Society for Sustainability Project CAP Citizen Advocate Program CATI Computer-Assisted Telephone Interview CCE Centers for Civic Engagement CDCS Country Development Cooperation Strategy CiDA Civil Development Agency COG Citizen Outreach Grants COVID-19 Coronavirus Disease of 2019 CN Concept Note CSO Civil Service Organization CSR Corporate Social Responsibility CTC Center for Training and Consultancy DRG-LER Democracy, Human Rights, and Governance – Learning, Evaluation, and Research EQ Evaluation Question ET Evaluation Team EWMI East West Management Institute FGD Focus Group Discussion GOG Government of Georgia IP Implementing Partner KII Key Informant Interview LG Local Government NCCE Network of Centers for Civic Engagement NORC NORC at the University of Chicago NGO Non-Governmental Organization PPAG Public Policy and Accountability Grants R-CSN Regional Civil Society Network TA Technical Assistance USAID United States Agency for International Development USG United States Government CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 1 EXECUTIVE SUMMARY Georgia has been historically characterized by low civic engagement among the general population, both due to poor awareness by citizens of their rights and responsibilities and to low public trust of civil society organizations (CSOs). While there are some highly effective and professional CSOs based in Tbilisi, the rest of civil society is largely ineffective and weak, especially outside the capital. Georgian CSOs have historically been driven more by foreign donor funding than by the concerns of local communities, and are largely dependent on international donor funds to cover operating costs. USAID had three main programs to address this problem over the previous two decades, culminating in the Advancing CSO Capacities and Engaging Society for Sustainability Project (ACCESS), implemented by the East West Management Institute (EWMI) from October 31, 2014 through October 30, 2021. ACCESS incorporates lessons learned and best practices from previous programs and adapts to an evolving operational context to address the Georgian civil society sector’s historical shortcomings. As part of the DRG Learning, Evaluation, and Research (DRG-LER) II Activity, NORC carried out a performance evaluation to assess the effectiveness of ACCESS. Findings are intended to inform similar programming in the future by the United States Agency for International Development (USAID) and other donors, including the upcoming USAID Civil Society Engagement Activity in Georgia. NORC answered the following evaluation questions: • Evaluation Question 1: How effective has the partner been in supporting CSOs to increase citizen engagement? How did problem analysis and activity design through co-creation contribute to addressing citizen identified issues? • Evaluation Question 2: How effective has the partner been in supporting CSOs in monitoring and influencing key government policies and processes? What policy areas do key informants mention as lacking in support? How relevant are these thematic areas/policies to citizens’ priorities and key development issues? What were the areas where ACCESS-supported CSOs achieved most notable results? Were there policy areas where ACCESS should have focused more resources to achieve better results to date, based on key informants and public surveys? • Evaluation Question 3: How successful has ACCESS been in working on local ownership of the Centers for Civic Engagement (CCEs) so that they continue to operate as politically neutral gathering spaces for public discourse? What is the potential of a Network of Centers for Civic Engagement (NCCEs) to diversify services and increase demand for CCE services in each location, strengthen financial diversification, expand services to the private sector, sustain results to date, and continue developing on its own? • Evaluation Question 4: How successful has the partner been in facilitating private sector engagement (exploring areas of mutual interest and CSR initiatives) and other inter-sectoral cooperation including to secure private sector funding and support? • Evaluation Question 5: How successfully did the project’s PMP measure the stated objectives of the project? Are there higher order, more outcome-oriented indicators that could have been employed? Primary data for this evaluation was collected from December 2020 to March 2021. Due to the COVID￾19 pandemic all data collection was remote. NORC’s evaluation approach used a mixture of quantitative CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 2 and qualitative methods, starting with a desk review of program documents, followed by preliminary KIIs with major stakeholders (USAID and ACCESS) to help finalize the evaluation instruments. Additional primary data collection included a web-based survey of CSO recipients of ACCESS grants and technical assistance, key informant interviews (KII), and focus group discussions (FGD) among remaining key stakeholders. The primary data collected for this evaluation is shown in the table below. Key Informant Interviews USAID/Georgia Donors Government of Georgia ACCESS staff Other USAID Implementing Partners ACCESS partners ACCESS grantees Network of Centers for Civic Engagement Media 2 4 7 7 3 6 10 1 3 Focus Group Discussions Centers for Civic Engagement staff General Citizens 1 8 Web-Based Survey ACCESS grantees N = 102 FINDINGS EVALUATION QUESTION 1 CCESS’s contribution to citizen engagement included innovative projects that addressed the wellbeing of Georgian citizens, especially via its citizen outreach grants. Both key informants and focus group discussants reported an increase in citizen engagement compared to prior project implementation. From 2017 ACCESS modified its approach shifting away from citizen awareness to support bottom-up civic action, including informal and newly established groups. With the use of the social labs methodology, ACCESS helped citizen outreach grant recipients develop and implement more innovative, well-thought￾through, and engaging projects. • ACCESS’s own definition of meaningful citizen engagement also evolved over its period of performance, in a direction of enabling active citizens for joint action at various levels, especially communities outside Tbilisi. Almost all key informants reported that ACCESS stood out among other donors by supporting ‘unusual’ thematic projects such as urban development, air quality, quality of life in remote mountainous regions, and cultural heritage. The key drivers of citizen engagement were the relevance of issues, youth being increasingly active in civil society, and improved engagement by ethnic communities. The obstacles to citizen engagement include the capacity of CSOs to successfully outreach and engage citizens, low formal membership and volunteering in CSOs by citizens, weak community mobilization, lack of civic education, disillusionment and a feeling of inability to affect change in citizens, and low trust in local authorities and political parties. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 3 An openness to new thematic areas, regional outreach and a flexible and demand driven approach to citizen outreach and rapid response grants was key to ACCESS’s effectiveness. Also critical was ACCESS’s nurturing of newly established groups to become more viable actors in civil society, and bringing Tbilisi-based CSOs to the regions to support local initiatives. In 2018 ACCESS started using the Social Labs methodology to enable its grantees to better tailor their projects to issues relevant for Georgian citizens. The co-creation tool also emphasized the direct engagement of citizens in leading the design of projects, and participating in actionable steps. This ensured that citizens became active contributors’ not just recipients of projects, and provided a platform for diverse actors to convene in a safe space. Social Labs also boosted ACCESS capacity to award more grants due to an improved quality of projects. EVALUATION QUESTION 2 ACCESS’s support to CSO’s in monitoring and influencing key government policies and processes was channeled through the public policy and accountability grants program. The emphasis of this program was on grass roots level policy initiatives and emerging policy issues. Initiatives supported by ACCESS were directed at influencing public opinion, researching policy problems and initiating and lobbying for policy/legislative changes. Most policy initiatives supported by ACCESS fell into 4 main thematic categories, including: 1) Countering disinformation 2) Environmental issues 3) Cultural heritage and urban development 4) Road safety. Countering disinformation and promotion of Euro Atlantic values is an area in which ACCESS partners achieved the biggest result in terms of introducing a problem into public discussions, making this issue constantly discussed and influencing public opinion and policy as a result. It culminated in the government including the problem of disinformation into its official communication strategy documents. Work on road safety also resulted in amendments to national legislation to address the problem of high fatality as a result of car accidents in Georgia. ACCESS CSOs were successful in changing environmental policy at the regional level, notably in Poti and Rustavi; but work in the cultural heritage and urban development area has had mixed success. CSOs do not focus on socio-economic issues which have remained citizens’ priorities in Georgia over the last decade. Work on these issues are hampered by a shortage of expertize among CSOs on these issues, and the quality of policy research in Georgia. EVALUATION QUESTION 3 Under ACCESS, CCEs have continued their previous role as neutral spaces for public discourse and expanded their role and visibility as a catalyst and enabler of civic engagement outside of Tbilisi. However, despite advances in providing fee-based services and reduced operating expenses, they will not be able to persist without core donor funding. This situation exists independently of the gathering restrictions necessitated by the COVID-19 pandemic, which have nevertheless exacerbated the CCEs’ financial struggles. The NCCE has made critical contributions to the CCEs’ ability to institute fee-based services, negotiate as a large network for grant funding and reduce operating expenses, and collaborate and learn from one another. However, some stakeholders wonder whether the NCCE is worth the significant Tbilisi-based expense when the CCEs’ financial situation is already so poor. For their part, CCE coordinators express gratitude and appreciation for the NCCE’s work, but question its sustainability without core donor funding. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 4 EVALUATION QUESTION 4 Historic engagement between the private sector and civil society in Georgia has been characterized by two parallel and exclusive spheres of operation, leading to mutual mistrust. ACCESS made important progress by creating the Corporate Social Responsibility Club, which served as a platform to facilitate more direct engagement and collaboration between private sector and civil society. Informed stakeholders universally consider “breaking the ice” between private sector and civil society to be a necessary first step for broader engagement in the future. Actors in both groups have indicated interest in taking established relationships further. However, there is still very limited direct funding of CSOs by private sector in Georgia, and even less engagement for other purposes, like influencing government policy for mutual gain. Considerable room for future improvement in private sector engagement remains, particularly in engagement outside of Tbilisi and in creating formal, established channels for CSOs to apply for funding from private companies. EVALUATION QUESTION 5 There is reasonable alignment between the indicators in the PMP and the concepts underlying ACCESS’ objectives and purpose. However, the PMP did not adequately convey ACCESS’ performance. The lack of a cohesive theory of change supporting ACCESS’ results framework posed considerable challenges to the execution of a successful PMP. Alternative MEL approaches such as developmental evaluation and outcome harvesting may have enabled ACCESS to track an evolving theory of change and define indicators to serve as bellwethers of long-term progress toward the project’s objectives. CONCLUSIONS ACCESS was successful in increasing citizen engagement using a flexible and demand driven model that focused on citizens priorities, involved them in the process of co-creation, and included informal and newly established CSO groups. ACCESS also established more direct engagement of the private sector with CSO, but additional modes of engagement, especially outside Tbilisi are needed. Policy changes on the problem of disinformation and road safety were initiated in Tbilisi and ACCESS CSOs had success in changing environmental policy at the regional level. Under ACCESS the Centers for Civil Engagement have continued to serve as a neutral space for public discussion, but their finances have been severely affected by COVID-19, despite an expansion of fee-based services. Finally ACCESS’s performance monitoring plan did not fully capture its performance, and lacked sufficient qualitative indicators to showcase its achievements. RECOMMENDATIONS • Maintain flexible demand-driven grant making, which is accompanied by deep engagement with grantees to help them build capacity. • Focus on strategic communication campaigns and citizen education to counter disillusionment and nihilism in the society. • Further develop institutional capacity of existing community-based organizations and community foundations. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 5 • Further promote the idea of co-creation and enable Georgian CSOs to run the process with the citizens and key stakeholders. • Enable CSOs to advocate for and work on social, health, environmental, and economic issues. • Further build the capacity of CSOs to conduct high quality policy research. • Support enabling factors for success in policy advocacy, including trust, mobilization of public support, and communication capacity. • Identify, and continue to fund CCEs who are providing essential services to communities, while exploring opportunities for self-financing. • Track mutual interests between civil society and the private sector, promote philanthropy, and encourage continued engagement. • Use leading indicators and other innovative MEL approaches such as developmental evaluation to support improved performance measurement. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 6 1. EVALUATION PURPOSE AND EVALUATION QUESTIONS As part of the DRG Learning, Evaluation, and Research (DRG-LER) II Activity, NORC carried out a performance evaluation to assess the effectiveness of the USAID/Georgia Advancing CSO Capacities and Engaging Society for Sustainability Project (ACCESS), implemented by the East West Management Institute (EWMI) under Cooperative Agreement # AID-114-A-14-00008 from October 31, 2014 through October 30, 2021. NORC is pleased to present this Evaluation Report summarizing key findings and recommendations. Findings are intended to inform similar programming in the future by the United States Agency for International Development (USAID) and other donors, including the upcoming USAID Civil Society Engagement Activity in Georgia. NORC answered the following evaluation questions: • Evaluation Question 1: How effective has the partner been in supporting CSOs to increase citizen engagement? How did problem analysis and activity design through co-creation contribute to addressing citizen identified issues? • Evaluation Question 2: How effective has the partner been in supporting CSOs in monitoring and influencing key government policies and processes? What policy areas do key informants mention as lacking in support? How relevant are these thematic areas/policies to citizens’ priorities and key development issues? What were the areas where ACCESS-supported CSOs achieved most notable results? Were there policy areas where ACCESS should have focused more resources to achieve better results to date, based on key informants and public surveys? • Evaluation Question 3: How successful has ACCESS been in working on local ownership of the Centers for Civic Engagement (CCEs) so that they continue to operate as politically neutral gathering spaces for public discourse? What is the potential of a Network of Centers for Civic Engagement (NCCEs) to diversify services and increase demand for CCE services in each location, strengthen financial diversification, expand services to the private sector, sustain results to date, and continue developing on its own? • Evaluation Question 4: How successful has the partner been in facilitating private sector engagement (exploring areas of mutual interest and CSR initiatives) and other inter-sectoral cooperation including to secure private sector funding and support? • Evaluation Question 5: How successfully did the project’s PMP measure the stated objectives of the project? Are there higher order, more outcome-oriented indicators that could have been employed? Across all five questions, the Evaluation Team (ET) assessed implications for the sustainability of the civil society sector in Georgia. The evaluation report reflects opinions and suggestions of key activity stakeholders from the host government, civil society, private sector organizations, other donors, USAID staff, and USAID and non-USAID implementing partners. The ET collected all primary data from December 2020 to March 2021. Due to the COVID-19 pandemic all data collection was remote, including focus group discussions. The ET include Ritu Nayyar-Stone (team lead), Orysia Lutseyvch (civil society subject matter expert), Tamara Sartania (local evaluation specialist), Gia Gotua (local evaluation specialist), Miguel Albornoz (evaluation specialist), Ridhi Sahai (qualitative specialist), and Jack Devine (quantitative specialist). Review of evaluation instrument and the final report was done by Zumrat Salmorbekova (evaluation specialist and reviewer). CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 7 2. PROJECT BACKGROUND Georgia has been historically characterized by low civic engagement among the general population, both due to poor awareness by citizens of their rights and responsibilities and to low public trust of civil society organizations (CSOs). While there are some highly effective and professional CSOs based in Tbilisi, the rest of civil society is largely ineffective and weak, especially outside of Tbilisi. Georgian CSOs have historically been driven more by foreign donor funding than by the concerns of local communities, and are largely dependent on international donor funds to cover operating costs. USAID’s main programming to address this problem over the previous two decades includes the Citizen Advocate Program (CAP) from 2002-2006 and the Good Governance in Georgia (G3) and Policy, Advocacy, and Civil Society Development in Georgia (G-PAC) Project from 2010-2014. ACCESS incorporates lessons learned and best practices from these projects and adapts to an evolving operational context to address the Georgian civil society sector’s historical shortcomings. The purpose of ACCESS is to build the capacity of the civil society sector as an end in itself while also supporting civil society to be a more effective force for demanding more transparent and accountable governance and more responsive policies in Georgia. It does so through citizen outreach grants (COG), public policy and accountability grants (PPAG), and direct technical assistance (TA) to CSOs as shown in Table 1. Individual grants ranged between $8,000 and $300,000, with an average value of around $50,000 per grant. Table 1. Types and Number of Grants Disbursed by ACCESS Type of Grant Type of Issues Addressed No. of Grants Disbursed No. of Grants by Location Citizen Outreach Connect CSOs to citizens on campaigns relevant to their organizational mandate 47 Tbilisi: 24 Regional: 23 Public Policy & Accountability Policy research and analysis in priority areas; monitoring implementation of key legislation and policies 20 Tbilisi: 19 Regional: 1 Rapid Response Flexible support for public awareness or community mobilization on time-sensitive political, economic, and social needs (e.g. COVID-19) or important policy research. 22 Tbilisi: 15 Regional: 7 Technical Assistance Build capacity to manage and operate CSOs and/or conduct policy research and analysis 35 Tbilisi: 21 Regional: 14 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 8 Figure 1. ACCESS Objectives and Anticipated Results Objective 1: Citizens more aware of and involved in CSO activities 1.1 CSOs regularly consult with constituents and informal civil society to identify problems and solutions. 1.2 CSOs seek citizen support for and involvement in their activities. 1.3 CSOs improve cooperation with other sectors, as well as within the civil society sector. 1.4 CSOs expand and diversify their sectoral coverage and activities. 1.5 CSOs are better protected from undue government interference. 1.6 ACCESS networking and citizen engagement activities attract more than 20,000 citizens and representatives of government, media, and business sectors. Objective 2: CSO leadership, organizational capacity, and sustainability, improved across a range of organizations in and outside Tbilisi 2.1 Leadership skills of 100 senior and mid-level managers are enhanced. 2.2 CSO governance and management are strengthened for 60 CSOs. 2.3 Fifty CSOs receive funding from more diverse sources. 2.4 At least two-thirds of mentored CSOs improve their original organizational assessment scores. 2.5 ACCESS-supported non-profit management program remains accessible beyond ACCESS’ lifetime. 2.6 Regional Civil Society Network’s (R-CSN) resources are actively used by Tbilisi-based CSOs and other stakeholders. 2.7 CSOs actively partner with the private sector to promote common public causes and increase sustainable CSR initiatives. 2.8 CSOs are able to secure funds from the private sector. Objective 3: CSOs effectively able to monitor and influence key government policies and processes 3.1 At least eight think tanks produce high quality research of relevance to key policy development processes. 3.2 Civil society engagement with government increased through up to 23 grant-funded monitoring initiatives. 3.3 At least 10 issues identified and advocated for by civil society are included in government policies and legislation. 3.4 CSOs take effective measures against anti￾Western propaganda and disinformation to build resilience against potential hostile foreign electoral interference in Georgia’s elections and sustain popular support for the country's European integration. 3.5 CSOs’ grassroot initiatives build societal resilience against violent extremism in the affected communities and help alter the government’s security-centered approach. 3.6 CSOs positively affect government policies and citizen engagement with critical urban development issues. Objective 4: Local ownership of Centers for Civic Engagement (CCE) so that they continue to operate as politically neutral gathering spaces for public discourse 4.1 At least seven CCEs remain operational beyond the ACCESS lifetime. 4.2 All sustaining CCEs increasingly attract more diverse funding. 4.3 All sustaining CCEs serve as a politically neutral gathering place open to all groups. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 9 ACCESS directly contributes to USAID/Georgia’s Development Objective One under its Country Development Cooperation Strategy 2013-2020: Enhanced democratic checks and balances and accountable governance, and Intermediate Result 1.1: A more informed and engaged citizenry. Its four main objectives and anticipated results are depicted in Figure 1 above. While ACCESS’ objectives have remained constant over its period of performance, two modifications changed the intervention modalities and anticipated results in response to a shifting operational context. In 2019, USAID/Georgia increased the total estimated amount of ACCESS by $2.1 million and extended the duration of the award from October 2019 to April 2021. During this first extension period, ACCESS altered its approach in objective 1 to simplify the grant application process and prioritize partnerships between regional CSOs, Tbilisi-based CSOs, and entities outside the civil society sector through “social lab” and other co-creation strategies, largely in response to low quality grant proposals from regional CSOs with regards to citizen engagement. It also aimed to extend the reach of successful CSO training programs, promote the capabilities of Georgia’s Regional Civil Society Network (R-CSN), and improve CSOs’ trust in and ability to approach the private sector for funding under objective 2. Under objective 3, ACCESS continued to promote priority policy areas that had emerged in its first five years, including anti-western propaganda and disinformation, traffic, air pollution, dereliction of cultural heritage, and violent extremism. Finally, in response to challenges in securing funding from non-USAID donors, ACCESS aimed under objective 4 to focus on fundraising for Centers for Civic Engagement (CCEs) through grants, fee-based services, and private sector partnership. In response to the COVID-19 pandemic in March 2020, ACCESS began to issue rapid response grants (RRG) to amplify citizen voices in pandemic responses, bolster emerging local initiatives, enhance community resource mobilization and volunteer engagement, and engage civil society groups in addressing medium- and long-term implications of the pandemic. In May 2020, USAID/Georgia increased the total estimated amount of ACCESS by $700,000 and extended the duration of the award by seven months, from April 2021 to October 2021. Under the proposed extension, ACCESS will provide additional support for grants and technical assistance to civil society to sustain community vigilance in the face of a prolonged response and recovery period to COVID-19. The extension activities will focus on CSOs with well-defined constituencies, representing vulnerable populations, including economically disadvantaged groups, women, and minorities - groups that are disproportionately affected by the pandemic or may not have equal access to response and recovery efforts, due to language or cultural barriers. 3. METHODS AND LIMITATIONS NORC’s evaluation approach uses a mixture of quantitative and qualitative methods, starting with a desk review of program documents, and followed by preliminary KIIs with major stakeholders (e.g. USAID, ACCESS, etc.) to help finalize the evaluation instruments. Additional primary data collection included a web-based survey of CSO recipients of ACCESS grants and TA, key informant interviews (KII), and focus group discussions (FGD) among remaining key stakeholders. This approach is summarized in more detail in the Evaluation Design Matrix from the evaluation concept note, included at the end of the evaluation statement of work in Annex A. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 10 DESK REVIEW The ET started with a desk review of available documents and secondary data. This included ACCESS quarterly and annual reports, performance monitoring plans and reports; and third party research reports.1 Information from these sources help inform all evaluation instruments. Table 2. Desk Review Documents Document Type No. of Documents ACCESS Program Document 27 Gray Literature 4 Population Based Survey Report 1 USAID Document 3 QUANTITATIVE DATA COLLECTION - SURVEY The team conducted an online web survey of all ACCESS grantees and TA recipients.2 Respondents were identified from EWMI’s list of grant and TA recipients. The total sample included 102 grantees and the response rate was 59.8%. The survey was developed on a modular basis and programmed using Qualtrics software so that respondents are only asked to answer questions relevant to their interaction with ACCESS. The Evaluation Specialist led questionnaire development, supported by NORC staff to ensure the survey elicits information from respondents in a way that reduces observational errors, specifically instrument error, due to ambiguous question wording or illogical questionnaire flow. NORC carefully reviewed the internal logic of each question developed for the survey instrument by the Evaluation Team to ensure that questions are not cognitively difficult, ambiguous, culturally insensitive, or include words that may create biased responses. In addition, questions were analyzed for their ability to link back to key indicators and evaluation questions, and for their contribution to the overall evaluation design. NORC’s secure online Qualtrics survey platform allowed the team to predetermine which modules to present based on information included in the respondent database or on real-time screening questions, or to allow respondents to select modules from a list. The online survey was made available in Georgian and English to cover the language preference of participating respondents. In addition to gathering relevant insights from ACCESS grantees and TA recipients, data from the survey also helped determine interesting grantees (based on the type(s) of grants received and focus of their work) that were interviewed for a deeper understanding of the project. 1 Most relevant among the research reports were those pertaining to public opinion and civic engagement of citizens in ACCESS-targeted regions. Special interest was paid to any that are representative at the regional level and/or for age-, gender-, or minority-disaggregated groups. 2 This refers to CSOs/organizations. Individual participants in ACCESS grants and TA were not targeted by this survey. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 11 QUALITATIVE DATA COLLECTION Our qualitative approach was based on KIIs and FGDs to provide concrete examples that illustrate in greater detail the influences and results of ACCESS activities. KII and FGD guides were designed to understand the complexities around ACCESS’ impact on civil society, support and bolster insights emerging from the online web survey, and providing context (or counter-examples) for surprising quantitative results. Where needed, the interview and discussion guides were reviewed and modified throughout the actual interview process, with appropriate probing for nuanced responses. Given the ongoing COVID-19 pandemic, maximum precautions were taken to ensure the safety and comfort of KII and FGD respondents. The Evaluation Team utilized telephone and/or video-conferencing software for all data collection. Restrictions in Georgia were respected and precautions exceeded those requested by local authorities out of an abundance of caution (i.e. collecting data remotely even if limited in-person gatherings were allowed).3 KEY INFORMANT INTERVIEWS In-depth KIIs were conducted with key stakeholders and partners, providing the core source of qualitative data collection. The ET worked closely with USAID and EWMI to identify all relevant stakeholders and determine the best means of contacting interview subjects. NORC conducted 43 total KIIs, as shown in Table 3. Table 3. KII Respondents ORGANIZATION(S) NO. TYPE OF KEY INFORMANT CANDIDATE USAID/Georgia 2 DRG Technical Staff Donors active in civil society sector 4 European Union, Soros Foundation, Dutch Embassy, SIDA Government of Georgia 7 Former Member of Parliament, Tbilisi mayor’s office, EU and NATO Integration Center, former member of Start Com Division, Ozurgeti Council, Ozurgeti Mayor’s Office, Transportation Department, Public Defender ACCESS Staff 7 COP, key technical and management staff Other USAID IPs in civil society sector 3 Media for Transparent and Accountable Governance Project, Good Governance Initiative, Momavlis Taoba Project ACCESS key partners 3 Civil Development Agency (CiDA), Regional Civil Society Network (R-CSN), and Center for Training and Consultancy (CTC) ACCESS Grantees 10 4 receiving COG, 4 receiving PPAG, 2 receiving RRG NCCE 1 Executive Director or delegated management staff ACCESS Private sector partners 3 GPI Holdings, Crystal, Efes Georgia Media 3 Batumelebi, Radio Marneuli, Samkhretis Karibche 3 See U.S. embassy website. Note all data collection protocols for this evaluation were reviewed and approved by NORC’s Institutional Review Board before field data collection. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 12 The team balanced the need to interview as diverse a set of stakeholders as possible for such a multi￾faceted, sector-level intervention and the need to achieve sufficient saturation of perspectives within similar groups of respondents. Logistical constraints precluded NORC’s ability to interview GOG respondents with expertise in the full universe of policy areas where Georgian CSOs intervene while still achieving saturation; some policy areas were combined or omitted. In some cases, the ET interviewed former government officials due to their experience and familiarity with the policy area of interest. The same concept was applied to the different regions where ACCESS implemented interventions in the local civil society sector. NORC developed semi-structured, open-ended interview guides tailored for each category of KIIs. The ET conducted the majority of interviews during a 5 week period mid-February to end of March 2021 in Georgia. KIIs were recorded only with the permission of interviewees, and notes were prepared by the interviewers directly in English for input and coding into Dedoose. All interviews with USAID personnel were conducted by the Team Leader and/or Evaluation Specialist. FOCUS GROUPS DISCUSSIONS The Evaluation Team conducted a total of 9 FGDs to gain a diverse set of perspectives from citizens and CCE staff in regions outside Tbilisi. FGDs offered an opportunity to discover rich information on complex questions related to information that respondents may have found more or less useful, and the environmental, organizational and individual-level factors that contribute to such views. The focus group structure allowed for feedback among participants, therefore generating responses that may not have emerged from the KIIs or web survey. Although the results are not generalizable to the broader population, they provide invaluable points of departure for the evaluation, as well as source material for vignettes to illustrate more general conclusions. The team conducted FGDs as described in Table 4. Table 4. Target FGD Respondents TARGET GROUP NO. SELECTION Citizens 8 One FGD with 6-8 citizens selected randomly* from CCE databases in each of the following groups, intended to solicit a range of perspectives on the topics of discussion: – Women only, Zugdidi – Women only, Sagarejo – Men only, Gori – Men only, Poti – Mixed gender, Rustavi – Mixed gender, Batumi – Ethnic Azeri women only, Marneuli – Mixed gender ethnic Armenians only, Akhaltsikhe CCE Staff 1 Management staff representing each of the 10 regional CCEs *See some exceptions in “Limitations” section As with the KIIs, the Evaluation Team consulted with USAID and ACCESS on the most effective means of making contact with FGD participants, using available relevant databases from ACCESS or the CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 13 regional CCEs. The local team contacted participants and conducted FGDs remotely using videoconferencing software to comply with COVID-19 safety protocol, since all participants were outside Tbilisi. The ET faced some difficulties in targeting the appropriate mix of participants and attendance at some FGDs. The target group for FGDs was citizens who were not closely affiliated with CSOs, but who have participated in events organized by CCEs. The evaluation team used a database provided by the NCCE to identify participants. However, for some CCEs, most citizens sampled from the database were closely involved in the work of CSOs, regular participants in events, volunteers, or otherwise working with CSOs. The actual composition of FGD respondents thus differed in some cases from the intended composition. LIMITATIONS The following limitations should be considered when interpreting results from this report: • Data availability and data quality: The research team collected all the primary data from KIIs, FGDs, and the web survey of 107 ACCESS grantees. None of the primary data was collected face-to￾face and was all collected remotely, which resulted in potential selection bias as described below. • Selection bias: As some key informants that the research team sought to interview declined to be interviewed, there is a possibility of selection bias, i.e. those respondents who choose to be interviewed might differ from those who do not in terms of their attitudes and perceptions, affiliation with government/non-government structures, and socio-demographic characteristics and experience. Given the remote nature of the data collection, there is also the concern that the officials and citizens we are able to reach electronically may be better-off economically, which could also result in selection bias. • Recall bias: As several questions raised during the interviews addressed issues that took place in the past (e.g., CSO and private sector engagement prior to ACCESS’s work on CSR), recall bias may have affected the responses provided. • Halo bias: There is a known tendency among respondents to under-report socially undesirable answers and alter their responses to approximate what they perceive as the social norm, called halo bias. In the case of this research this may manifest itself via the extent to which respondents will be prepared to reveal their true opinions and experiences when they are asked to provide feedback on governments’ response, collaboration and challenges associated with accepting CSO research and advocacy for policy changes. In other words, they may not be willing to reveal information that might reflect negatively on their government. However, in the context of the research design outlined here, this limitation should not have a significant impact as we are obtaining information from a variety of stakeholders and ensuring anonymity to their response. • Remote data collection: Collecting data remotely was complicated by connectivity and reception issues, as well as by challenges with reaching the intended key respondents by phone or email. Multiple follow up calls were often required to ensure the participation of key informants in government positions, and in some cases KIIs and surveys had to be completed in multiple sessions. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 14 • Sampling for citizen FGDs: The ET intended to select 6-8 respondents for citizen FGDs selected randomly from the list of citizens maintained by CCEs in each region. In practice, many of the “citizens” in these databases were in fact meaningfully affiliated with formal civil society organizations themselves. The ET desired the perspective of citizens who were not so affiliated with civil society for these groups. The sample frame corresponding to this profile in each area was as small as a few dozen potential respondents, of which several declined to participate in focus groups. In many cases (e.g. Rustavi, Gori, and Poti), the ET thus turned to local NGOs to assist in recruiting citizens for FGDs, screening for a diverse group of respondents that was not overly affiliated with civil society. This approach introduces potential sampling bias by relying on networks of individual CSOs, although it does so at the benefit of including perspectives in the sample that are more representative of the citizen population unaffiliated with CSOs. Additionally, for the FGD in Marneuli, the ET needed to resort to purposive sampling to assemble a focus group of Azeri women in Marneuli. Most female Azeri citizens in the Marneuli CCE database (students) were not eligible for the study based on their age, and thus random sampling was not feasible. 4. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS FINDINGS EQ1: How effective has the partner been in supporting CSO’s to increase citizen engagement? How did problem analysis and activity design through co-creation contribute to addressing citizen identified issues? Increasing engagement of citizens in various types of civic activism remains one of the key challenges for the sustainability of Georgian civil society. USAID Sustainability Index states that: ‘Constituency building remains a challenge for CSOs, especially at the grassroots level. CSOs lack close and long-lasting bonds with their beneficiaries, mainly because of their reliance on foreign funding, which keeps them accountable to their financial donors and open to changing the focus of their work depending on the availability of donor funding and shifts in donors’ priorities.4 Addressing this challenge was one of ACCESS’ key objectives. ACCESS’s contribution to citizen engagement included innovative projects that addressed the wellbeing of Georgian citizens, especially via its citizen outreach grants. Both key informants and focus group discussants reported an increase in citizen engagement compared to prior project implementation. From 2017 ACCESS modified its approach shifting away from citizen awareness to support bottom-up civic action, including informal and newly established groups. With the use of the Social Labs methodology, ACCESS helped COG recipients develop and implement more innovative, well-thought-through, and engaging projects. DEFINING CITIZEN ENGAGEMENT Various KII respondents most commonly defined citizen engagement as citizens working with CSOs, actively stating their positions on various issues, pushing collectively to solve community problems and 4 USAID Sustainability Index, 2016, p. 95 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 15 keeping the government in check. Citizens attending FGDs most often mentioned engagement as an action of active citizens to solve their community problems and influence change. “Civic engagement is being active, volunteering, or doing good things to solve problems in our society. That is, doing a lot to solve the problems in our community, in our region, even in the country. Making a positive impact is very important to me. Every citizen must have a responsibility and we must be active to do something good for our community.” [Citizen FGD, Marneuli] ACCESS’s own definition of meaningful citizen engagement has also evolved over its period of performance, in a direction of enabling active citizens for joint action at various levels, especially communities outside Tbilisi. During KIIs the ACCESS team pointed to lack of meaningful engagement strategies by the well-established CSOs and the pipeline of proposals that included citizens merely as beneficiaries rather than agents of change. “They (CSOs) identify problem, identify activity plan, monitor, publish report, do a press conference, all very closed. You don’t engage citizens by just giving final findings and running recommendations through a focus group.” [KII, ACCESS staff] From 2017 ACCESS’s modified its approach ‘to meet citizens where they are’ and supported bottom-up civic action, including by informal, newly established groups, with active contribution from the citizens, and formal CSOs creating space for this engagement SHIFTING FOCUS FROM CITIZEN AWARENESS TO CITIZEN MOBILIZATION AND ENGAGEMENT In its initial years ACCESS invested in promoting awareness among citizens on various issues. For example, it collaborated with CivilPitch5 to engage wider audiences in advocacy efforts, ultimately leading to 2 targeted COGs after screening documentary films promoting urban development.6 ACCESS gradually shifted its emphasis from delivering advocacy outcomes to the value of mobilizing citizens around public issues, debates, and initiatives regardless of the outcome. In 2018-2019 ACCESS grant projects and civic activities engaged over 41,000 citizens to address some of the most important problems facing Georgia’s regions.7 When reflecting on ACCESS contribution to citizen engagement many key informants mentioned innovative projects addressing needs closely related to the wellbeing of Georgian citizens that lacked the support of other donors. Literature review also points to donor-driven funding as the main problem for CSOs’ limited experimentation, and the lack of timely, innovative initiatives.8 Almost all key informants reported that ACCESS stood out among other donors by supporting ‘unusual’ thematic projects such as urban development, air quality, quality of life in remote mountainous regions, and cultural heritage. Bus tours were an innovative way to reach beyond Tbilisi and promote activism. ACCESS did 13 bus tours 5 CivilPitch is a platform that gives an opportunity to documentarians, film directors, and civil society activists to combine their skills, knowledge, and experiences for producing documentary films addressing social issues and human rights. It is hosted by the Open Society Georgia Foundation. https://osgf.ge/en/civil-pitch-documentary-projects/ 6 ACCESS Third Annual Report, 2017 7 ACCESS Fifth Annual Report, 2019 8 USAID Sustainability Index, 2016, p. 97 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 16 from July 2015 – September 2020 across villages in Ajara, Guria, Kvemo Kartli, Kakheti, Samtskhe￾Javakheti, Shida Kartli, Upper Svaneti, Lower Svaneti, and Racha-Lechkhumi.9 The desk review indicates ACCESS’s direct encouragement of its CSO partners to innovate, and take reasonable risks by experimenting with new approaches without risking losing grant funds.10 Both key informants and focus group discussants reported an increase in citizen engagement compared to 2014. Almost all citizens attending the FGDs stated how the number of active people has increased, more initiatives and organizations have emerged that attempt to solve community problems collectively with other citizens. Drivers of Engagement In FGDs active citizens indicated that what motivates and enables their engagement is knowledge of their rights as citizens, ability to bring positive change and hope, solve problems that directly impact them, contribute to their community, as well as meet new people and acquire new knowledge. Grey literature confirms the importance of drivers such as growing expectations from participation, importance of localism, new technology enabling participation and predominance of more spontaneous and unstructured ways of modern engagement.11 Growing motivation to participate is supported by positive dynamic growth of emancipative values (individual freedoms and rights) in Georgia, despite regional regression in Armenia and Azerbaijan.12 Relevance of issues is a strong motivator for citizens to engage. For example, in an FGD CCE coordinators indicated that in Guria the construction of hydropower plant is a big issue, citizens meet with each other, post petitions, share information in media, and attend official public meetings. Similar activities addressing various issues could be observed in Poti, Rustavi, Zugdidi. FGDs and KIIs provided several examples of successful citizen engagement, i.e. return of a castle in Gori to the local community, equipping busses for people with disabilities, stopping construction of a prison, introduction of municipal transport in Telavi, displaying maps in Tbilisi metro, opening youth center with informal education for the villages of Khundarisa and Molakvi, setting up a knowledge café in Tsori, and connecting the eastern mountainous regions to the internet. Youth are increasingly active in civil society. Citizens participating in FGD (Marneuli, Akhaltsikhe) stressed how young people are more active due to having more free time, being active on social media and by working with teachers who encourage civic activism. This finding is also supported by external literature, which states that young people between the ages of eighteen and thirty-five are more likely to volunteer (26%) compared to people between the ages of thirty-six and fifty-five (22%) and over fifty-six (15%).13 Local CECs were mentioned as playing a facilitating role in harnessing various groups of volunteers, activists and local CSOs for various regional projects [more details on CCEs are provided in findings for 9 ACCESS Annual Reports 1 through 6. 10 ACCESS Fourth Annual Report, 2018 11 https://www.involve.org.uk/sites/default/files/uploads/Briefing-paper-2-What-are-the-drivers-of-participation1.pdf 12 From Apathy to Action: Attitudes to Civic Engagement in the Eastern Partnership, p. 15 13 2019 Caucasus Barometer Georgia CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 17 evaluation question 3]. In terms of various constituencies, local communities (68%), youth aged 15 to 25 (66%), women (66%) and NGOs/civic activists (66%) are the most served group by ACCESS grantees (see Table 6 below). The Desk review indicates that many young people were engaged in distributing packages or helping vulnerable groups claim COVID-19 related benefits.14 In an interview an ACCESS grantee reported that according to Facebook statistics, younger women (less than age 30) are much more sensitive to environmental issues. Engagement of ethnic communities. Most key informants among donors, ACCESS staff and grantees pointed to a slight improvement in the way ethnic minorities engage in civic action. In a KII an ACCESS grantee from the Azeri community in Kvemo Kartli stated that many things have changed over the last 10-15 years and ‘now we feel as full-fledged members of the country’. Some key informants from donors indicated that the Armenian community is organized to a lesser extent. Types of Engagement During FGDs citizens indicated various modalities of citizen engagement. They mentioned rallies, petitions, attending meetings of local Sakrebulo or village assemblies, identifying issues and then brainstorming about resolving them, and trainings. Some mentioned forming an organization, mobilizing voluntary groups or small unions where people can discuss issues, drafting official documents and presenting them to respective organizations. There was an emphasis on constructive and collaborative engagement with the local government. For example, following a citizen petition facilitated by the Poti Civic Platform the municipality has installed automatic air monitoring stations in the most contaminated locations. #Gavigudet movement for clean air in Rustavi in addition to campaigning has joined public consultations on important environmental bills.15 ACCESS grantees worked towards promoting various forms of citizen engagement. Center for Civic Activities (CCA) mobilized ten communities in Pankisi and five communities in Tusheti, including raising awareness on citizen engagement mechanisms from the local self-government, facilitating attendance of Sakrebulo meetings, developing petitions, and engaging in follow-up advocacy. CCA held 15 discussions focusing on issues of local importance, like development of tourism and infrastructure and preserving cultural heritage. Young Scientists' Club of Ozurgeti developed an online citizen-petition platform for Ozurgeti municipality and taught citizens from 29 administrative units in Ozurgeti municipality the petition preparation and submission process.16 ACCESS grantees employed a variety of methods to engage citizens. In the online survey many CSOs report quite strong focus on engaging citizens in planning activities (52%) and identifying priorities (61%) – see Figure 2 for the six most frequently selected ways of engaging citizens (respondents were allowed to select multiple responses). This is indicative of the importance of co-creation efforts and CSOs being responsive to them. The most popular way of engagement is citizen participation in trainings and seminars (75%). 14 ACCESS Sixth Annual report, 2020 15 ACCESS Sixth Annual report, 2020 16 ACCESS Fourth Annual Report, 2018 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 18 Figure 2. Top Six Ways of Engaging Citizens (% of respondents; multiple selection) Source: Web survey of ACCESS grantees. KIIs with ACCESS suggest that this emphasis on educating citizens, doing things with them and not for them has grown over the years. Consistently with this finding grantees consider their most notable results to be related to increasing public support to the monitored issues (47.4% of respondents), countering disinformation (34.2% of respondents), and increasing media attention on an issue (44.7% of respondents) – see Figure 3 for other results selected. Figure 3. Most Notable Grant Results in Terms of Monitoring and/or Influencing Government Policies and Processes Source: Web survey of ACCESS grantees Obstacles to Engagement Despite relevant and salient focus of many projects, ACCESS grantees still struggled to improve their capacity in citizen engagement. Majority of ACCESS CSOs (18 out of 30) showed no improvement in 75% 61% 55% 52% 39% 38% Participants in trainings/seminars Identifying priorities Recruit as volunteers Planning activities Public pressure campaigns Signatories of petitions 47% 45% 34% 29% 29% Increased public support to the monitored issue Increased broad media attention to the monitored issue Countered disinformation Research and advocacy impacted state policy or initiated drafting law to address issues Research help increased public awareness of social media disinformation CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 19 citizen engagement score according to OCAT (organizational capacity assessment tool). OCAT was provided by ACCESS as it is widely used by USAID for its various civil society programs. ACCESS partner CSOs demonstrate only marginal improvement of their average community outreach and engagement capacity score over the time of the project from 3.84 to 4.08. According to KIIs with an ACCESS partner, grantees rarely prioritized citizen engagement as an area that needs improvement. Among 30 CSOs OCAT scores show that only 5 grantees slightly improved their citizen engagement score. Among the groups that have achieved improvement most have targeted constituencies (rural children, vulnerable groups, ethnic minorities). Two are working at a grass-root level (EMC, KRDF) and are helping to solve issues of direct concern of citizens. For example, Kakheti Regional Development Foundation conducted youth engagement and capacity development activities in Duisi, Marneuli, and Sakobiano. It has engaged 60 participants in a youth evening program to produce documentaries about youth problems and present to local communities.17 Formal membership and volunteer engagement in CSOs remains low. According to an ACCESS partner key informant, working on capacity building outreach and engagement with respective constituencies remains a significant problem for the CSOs. The web survey of ACCESS grantees indicates that close to majority of CSOs (46%) have less than 10 active members and 62% have less than10 volunteers. Only 8% of CSOs have more than 300 members. Only 6% of ACCESS grantees engages citizens by recruiting them to become members and only 13% views them as potential volunteers. PMP (Y6-7) sets a quite low target for volunteer recruitment of 800 with 2700 target for the life of the project.18 Despite weak formal engagement with CSOs, ACCESS grantees report a strong sense of legitimacy and connection with their constituency. Seventy-one percent of ACCESS grantees in the web survey strongly agreed that ‘my CSO serves a clearly identified constituency that is aligned with its mission’. At the same time majority (57%) strongly or somewhat disagree that their CSOs can draw upon a core of monetary support from their communities and constituents (i.e. crowdfunding). Optimism in non￾monetary support is slightly higher with 42% saying they could draw on such support. Weak community mobilization. FGDs with citizens point to insufficient community mobilization, with citizens describing the co-existence of an active minority and skeptical and nihilistic majority. The latter passive group includes all generations. This group would expect the Government to solve all their problems. Citizens felt CSOs could inspire more people to get together and give them hope that their voice will be heard. The desk review supports this finding indicating that in the past 6 months only 6% attended the town hall meeting, 3% volunteered, 4% attended rally, and 1percent signed a petition.19 “So, today more groups emerge from below trying to influence processes or just to defend itself, but still distance between population and civil society remains, which is reflected in polls.” [KII, ACCESS key partner] 17 ACCESS Fourth Annual Report, 2018 18 With the current target of 2700 volunteers, assuming these are 30 grantees, ACCESS hopes to have an average of 90 volunteers per CSO. Polling data suggests a much higher potential for engagement. If we only consider youth (15 to 29) currently roughly 23,000 report volunteering (3% @IRI poll). This means that ACCESS’s current target taps only 11% of the current volunteer base. The ET suggests that ACCESS aim to reach at least a quarter of already active citizens increasing the target to 6,000. Furthermore, the potential for increasing volunteering is quite high with 49% of Georgians stating that it is rather important for a good citizen to volunteer. 19 IRI Public Opinion Poll, 2019 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 20 Lack of civic education. The ET recorded several reasons for weak engagement. The ACCESS grantee survey, KIIs with most donors and many ACCESS grantees reported lack of civic education as one of the main obstacles to citizens’ engagement. A majority of the web survey respondents (58.9%) stated that citizens have a low level of civic education about their rights and have limited knowledge of civic participation (see Table 5 below for a full list of challenges for citizen engagement cited by survey respondents). As the result of this low awareness even those methods of participation that are legally outlined remain underutilized. KII with USAID IPs pointed to low ranking of citizen participation in the budgeting process (28 out of 100) despite Georgia scoring 5th globally in the Open Budget Index (81 out of 100).20 Table 5. Main Challenges Faced in Engaging Citizens by ACCESS Grantees RESPONSE OPTIONS PERCENTAGE OF RESPONDENTS Citizens believe they cannot influence change 60.7 Citizens have a low level of civic education about their rights 58.9 Citizens have limited knowledge of civic participation 58.9 Our organization has limited financial resources 53.6 Citizens have different priorities due to a decline in their quality of life 53.6 Citizens lack of trust in government 50.0 Citizens lack trust in CSOs 37.5 Citizens believe that reform efforts will fail 30.4 Citizens lack of attention to public affairs 30.4 Citizens believe that CSOs are politically motivated 26.8 Citizens do not understand the importance of issues that our CSO addresses 25.0 Citizens fear retaliation against activists 10.7 Our staff is fearful of retaliation against activists 7.1 Our staff have insufficient training/knowledge 3.6 Source: Web survey of ACCESS grantees Disillusionment and a feeling of inability to affect change. One of the top reasons of weak engagement as seen by ACCESS grantees is the fact that citizens believe they cannot influence change (60.7%). However, the desk review reveals that the citizens themselves believe that ordinary people can 20 https://www.internationalbudget.org/open-budget-survey/country-results/2019/georgia CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 21 influence decisions in the country. Those who say influence is very or somewhat likely is 46% and outweighs the 31% who say this is very unlikely.21 Low trust in local authorities and political parties. Half of ACCESS grantees who responded to the online survey named citizens’ lack of trust in government as an obstacle to engagement. “The population that does not know whom to trust – you turn to one Channel you hear one thing and then turn to another – and hear a completely different story. So people try to limit themselves to their family and think about survival and the rest they do not care about.” [Citizen FGD, Gori] Stalled decentralization was named by citizens and several IPs as contributing to apathy. A centralized system of governance where Prime Minister’s appointed Governors are more powerful than elected Mayors makes local activism difficult. FGD with CCEs coordinators mentioned increasing distance between citizens and local authorities in the last 7 years due to centralization of governance. Grey literature review suggests that Georgia also suffers from low levels of generalized trust with only 6% trusting their fellow citizens in 2019.22 Lack of capacity among CSOs to successfully outreach citizens. Key informants from other donors spoke about the gap between citizens and Tbilisi-based CSOs who speak complicated and sophisticated language that common citizens outside Tbilisi do not connect with. Lack of outreach to the regional level was viewed as contributing to mistrust. ACCESS staff pointed to lack of internalizing by CSOs of engagement and treating it simply pro-forma as ‘participant lists’. ACCESS grantees have a certain ‘blind spot’ with regards to their internal capacity and, only 1% of ACCESS grantees stated in the web survey that their staff’s insufficient training and knowledge is an obstacle to citizen engagement. Another internal factor linked to challenge with engagement as noted by the ACCESS staff was lack of transparency among CSOs to the public. Only 3% of ACCESS grantees invite citizens to join their Board and few make annual reports available to the public. Shrinking civic space. A 2019 study has 60% respondents in Georgia saying that a shrinking civic space is a problem.23 This was confirmed during KIIs with ACCESS and USAID, other donors and some ACCESS grantees who stated that the current political crisis, increasing polarization and de-facto one party system of governance makes it more challenging for CSOs to advocate, and constructively engage with the political parties. Several KIIs respondents spoke about a backlash against leading Tbilisi-based CSOs in the public media, casting a negative shadow over wider civil society sector. At the local level excessive politicization of civic space leads to some fear among civil servants, including teachers to expose themselves as cooperating with CSOs. A Focus on Citizens’ Priorities By awarding COGs ACCESS supported a variety of issues that were embedded in citizen priorities. Among them internet access, metro infrastructure, tunnel access, quality of air and water, preservation of cultural heritage (Batumi), intercity transport safety, disinformation, road safety, access to educational resources in rural areas, development of mountainous regions. Many ACCESS grantees such as Platform 21 IRI Public Opinion Poll, 2019 22 From Apathy to Action: Attitudes to Civic Engagement in the Eastern Partnership, p.11 23 From Apathy to Action: Attitudes to Civic Engagement in the Eastern Partnership, p.17 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 22 Salami, Orbeliani, Mtis Ambebi all focused on solving local problems. Thematically many ACCESS grants focused on urban development and environmental issues trying to address the needs of Georgian citizens. Literature review points to the following top three priorities for Georgian citizens at the local level: first being unemployment, second – roads, and third - quality of water.24 Moreover, roughly half (48.3%) of ACCESS grantees participating in the web survey report working closer to the citizens on the regional and sub-regional levels. This focus on more silent issues increases the chances for the citizens to engage with CSOs. Literature review also indicates that when asked if citizens would be interested in ‘becoming a member of an organization that works to ensure food safety in Georgia’, 43% of the population expressed interest in joining.25 The top 5 items selected by ACCESS grantees when asked which target groups their organization works with were local communities/local authorities, youth aged 15 – 29 years, women, NGO/civic activists, and media. See Table 6 for a full breakdown of target groups selected by survey respondents. Table 6. Groups Targeted by Grantee Survey Respondents RESPONSE OPTIONS PERCENTAGE OF RESPONDENTS Local communities/ local authorities 67.9 Youth aged 15 to 25 years 66.1 Women 66.1 NGO / civic activists 66.1 Media 46.4 Children aged up to 14 42.9 People with disabilities 35.7 Private sector 35.7 Non-dominant ethnic or religious groups 30.4 Internally displaced people (IDPs) 25.0 Elderly/retired 19.6 Voters 19.6 LGBTI (Lesbian, gay bisexual, transgender, and intersex) people 17.9 Prisoners 10.7 Sportsmen 5.4 Veterans 3.6 Source: Web survey of Access grantees 24 IRI Public Opinion Poll, 2019, slide 8 25 G-PAC Civic Engagement Survey, 2014, page 4 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 23 “Citizens are more responsive to the work of CSOs as they saw CSOs as organization that helped them – they are not only grant eaters. Now citizens have more positive understanding of citizen work.” [KII, ACCESS staff] FDGs with active citizens endorsed this improvement of the view of CSOs in the communities. Those citizens that engaged with CSOs reported during FDGs that they trust them, and this trust is increasing thanks to positive outcome of actions and belief that CSOs can help their community. In the survey of ACCESS grantees 20.8% of respondents stated that using the new project design methods for ACCESS grant funding helped them overcome citizens’ lack of trust in CSOs. FLEXIBLE AND DEMAND DRIVEN APPROACH TO GRANTS One way to ensure wider societal relevance in a highly volatile environment of Georgia was to embed flexibility and velocity in the programming. The ET found that this openness to new thematic areas, regional outreach and flexibility was highly appreciated by many key informants among ACCESS grantees and other donors. ACCESS achieved this by announcing calls for proposals without pre-defined priorities. This was achieved by providing COGs and RRGs to various organizations. According to the survey of ACCESS grantees, 50% of respondents received COGs and 28% RRGs– see Figure 4 for a breakdown of types of grants received by survey respondents. Figure 4. Grants Received by Survey Respondents Source: Web survey of Access grantees The use of RRGs -- flexible and rapid funding tools that enabled ACCESS to respond to citizen needs -- became especially prominent when the COVID-19 pandemic hit Georgia. Many key respondents from donors, CSOs and IPs stressed the importance of RRGs. In 2020 ACCESS awarded 22 RRG. RRGs enabled quick mobilization of civic action to inform citizens about the risks of the pandemic, provided support to vulnerable groups (especially ethnic, and children), distributed motivational content, and countered COVID-19 conspiracies and disinformation. The ACCESS team also expanded the thematic focus of donor-supported projects partially bridging the gap between citizens and CSOs. A desk review of ACCESS annual reports reveals the evolution of grant-making from process-oriented projects to more community-based problem-solving activities. 50% 18% 29% 14% Citizen Outreach Grant Public Policy and Accountability Grant Rapid Response Grant Technical Assistance CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 24 NURTURING AND ENABLING NEW ACTORS ACCESS nurtured their partners, especially newly established groups to become more viable actors in the civil society. In the web survey, 39% of ACCESS grantees reported operating informally before registering as a formal CSO. By working with informal and newly registered groups key informants believe that ACCESS was supporting those whose projects are more grounded in citizen priorities and delivering results to citizens. In interviews, other donors’ staff expressed admiration and interest in ACCESS work with individual activists. More than a quarter (31%) of ACCESS grantees participating in the survey were registered after 2014. The average years of operation for grantees responding to the web survey, defined as the time passed since they began operations, is 13.5 years. As shown in Figure 5, ACCESS gives grants to younger organizations over time, the sharpest reduction is seen for PPAG recipients. Figure 5. Average Years of Operation for ACCESS Grantees by Year of First Grant Source: Web survey of Access grantees The segment of informal groups cooperating with ACCESS remains small, with only 2 ACCESS grantees participating in the online survey indicating that they are currently not registered formally with the Georgian government as a non-commercial entity. Fourteen respondents, however, indicated that they had previously operated informally. The distribution of these respondents over time are shown in Figure 6. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 25 Figure 6. Grantees Who Previously Operated Informally by Year of First Grant Source: Web survey of Access grantees In KIIs ACCESS grantees repeatedly mentioned how open-minded, engaging, and helpful the ACCESS team was in helping them fine-tune their proposal ideas, suggesting best outreach tools, connecting to other stakeholders, and encouraging them to consider the relevance and interests of the core groups of beneficiaries they serve. ACCESS grantee Orbeliani help citizen projects to develop their online fundraising plans and set up the tailored crowdfunding campaigns. This finding was also confirmed by KIIs with ACCESS staff where they deliberately simplified application process paying more attention to the idea and needs of the community rather than technical criteria or capacity to present elaborate budgets. Enabling a platform for crowdfunding activities such as Orbeliani ACCESS helped build mobilization of communities, promote and fund ideas of these nascent groups. ACCESS funding matched the amount fundraised from the citizens on the platform. In 2020 10 out of 12 projects reached their targets and four campaigners – the Bediani Climbing Club, New Life for Okami Park, Healthy Yards for Tbilisi, and 2 GELs to solve 2 Problems – were particularly creative in mobilizing supporters and fundraising. 26 WORK WITH REGIONAL ACTIVISTS Many key informants from the donor community and ACCESS staff acknowledge the risk-taking nature of ACCESS. As the program progressed ACCESS shifted focus from bringing Tbilisi-based CSOs to the regions to supporting local emerging initiatives and CBOs. It was aiming to meet active citizens where they were. Review of ACCESS annual reports indicates that ACCESS staff encouraged regional CSOs to engage diverse groups (outside the usual suspects of the civil society sector), including local teachers, students, entrepreneurs, journalists, artists, civic activists, pensioners, etc. in identifying the target problems and developing their solutions. 26 ACCESS Sixth Annual report, 2020 1 4 4 4 1 0 1 2 3 4 2011 2013 2015 2016 2017 2018 2019 2020 2021 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 26 ACCESS staff pointed to a cohort of new community-based organizations (CBO) that they aided to emerge. Currently there are 6 CBOs registered that mobilize communities to address local issues. These grass-root groups inspire citizens by example. “When other citizens see their fellow citizen’s engagement – they become interested. This brings more trust and confidence.” [KII, ACCESS staff] PROBLEM ANALYSIS AND ACTIVITY DESIGN THROUGH CO-CREATION In 2018 ACCESS introduced new tool to enable its grantees better tailor their projects to the issues relevant for Georgian citizens. Social Labs methodology was first introduced to the Georgian civil society by Reos Partners. It combines the following key approaches: addressing root causes, developing solutions, involving all key stakeholders, learning by doing, proto-typing solutions, and keeping a stable platform of actors over time.27 Desk review,28 KIIs with ACCESS grantees, and the web survey suggests that Social Labs helped the COG recipients develop and implement more innovative, well-thought-through, and engaging projects. As of March 2021 ACCESS has co-created 13 projects with 15 more projects planned for years 6-7. According to the Year 6 PMP, ACCESS co-created 17 activities through a social lab approach, including 12 COGs and 5 PPAGs. A 2020 literature review by ACCESS’ key partner CTC (here) indicates: “Social labs are comprised of multiple teams and people – conveners, secretariat, facilitators, managers, retreat leaders, historians who document the entire process, hosts, and core lab teams with up to 30-35 people in each. Not all labs need that many people, and their size and structure depend on the nature and scale of the challenge the lab is addressing.” The ET found that this co-creation tool emphasized direct engagement of people impacted by the projects to ensure that they: (1) are grounded in the needs of the community, (2) lead the design of the project, and (3) join actionable steps. It enables collaboration between ACCESS, its local partners and active citizens. After receiving concept papers ACCESS engaged with the shortlisted grantees/partners to further develop their ideas during Social Lab meeting. These lab discussions brought together project team, diverse groups and individuals, especially those directly affected by the proposed project and allowed for key stakeholder engagement from the start. With the citizens – not just for the citizens. The ET established that Social Labs provided a space for citizens themselves to be active contributors not just recipients of projects. Most key informants from ACCESS, USAID and grantees highly valued the fact that this format provided a platform for diverse actors (artists, marketing specialists, SMEs, parents, elder citizens) to convene in a safe space. Many key informants spoke about the importance of this collaborative process as compared to a more common donor focus on just achieving tangible restudies. According to ACCESS staff it is exactly this co-creation from the start that ensures stronger success in the end. 27 Reos Partners, Applied Social Lab Methodology Handbook 28 ACCESS Fifth Annual Report, 2018 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 27 ‘People believe in what they help to create. You cannot achieve anything for citizens without citizens. Citizens support what they help to create, when they know they get to influence the result’. [KII, ACCESS staff] Table 7. Challenges for Citizen Engagement Overcome with New Methods RESPONSE OPTIONS PERCENTAGE OF RESPONDENTS Citizens believe they cannot influence change 43.4 Citizens have limited knowledge of civic participation 41.5 Citizens have a low level of civic education about their rights 33.9 Our organization has limited financial resources 24.5 Citizens lack trust in CSOs 20.8 Citizens lack of attention to public affairs 20.8 Citizens do not understand the importance of issues that our CSO addresses 15.1 Citizens lack of trust in government 15.1 Citizens believe that reform efforts will fail 11.3 Citizens have different priorities due to a decline in their quality of life 11.3 Citizens believe that CSOs are politically motivated 5.7 Citizens fear retaliation against activists 1.9 Our staff is fearful of retaliation against activists 1.9 Our staff have insufficient training/knowledge 1.9 Source: Web survey of Access grantees The web survey suggests that ACCESS grantees view these new tools of citizens’ engagement as the one helping to empower and educate citizens about their rights and pathways of civic participation. See Table 7 above for a full breakdown of challenges that grantee respondents indicated they had overcome using the new methods for citizen engagement. The ET found that ACCESS partners appreciated this innovative approach and found it useful. KIIs with ACCESS grantees show that this group valued this new skill. The web survey supports this finding with 79% of respondents who used Social Labs or a similar participatory approach indicating that this was a new method for their CSO. This is also supported by the desk review where, according to the CSOs this was the first time that they were engaged in this kind of grant-making and project design process and they found the process extremely practical and productive, as it encouraged constructive criticism, creativity, and multi-angle perspective.29 29 ACCESS Fifth Annual Report, 2018 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 28 ACCESS staff expressed confidence that co-creation in some cases helped bridge the gap between CSOs and citizen priorities. Social Labs ensured that projects were designed jointly with the community and reflected local needs. Several ACCESS staff members mentioned that in initial consultations with stakeholders, citizens’ priorities were often not connected with those of CSOs. By facilitating these meetings and discussions ACCESS’s grantees could hear the feedback from diverse groups and adjust their work plans. Some ACCESS grantees indeed pointed to changes in the projects that they implemented after these community discussions. The online survey supports this finding with 96% of respondents who used new methods strongly or somewhat agreed that these methods improved their ability to address constituents’ priorities. This adaptability was also possible thanks to ACCESS staff emphasis that Social Labs were part of a wider program culture that nurtured relevance to community needs rather than rigid commitment to implement projects as they were approved by the funder. Social Labs also boosted ACCESS capacity to award more grants due to an improved quality of projects. Prior to Social Labs ACCESS staff felt that many proposals had weak citizen engagement strategies and many applicants lacked ideas about creative engagement. According to ACCESS staff Social Labs aided to elevate the quality of proposals by better embedding citizen interests. ACCESS grantees report the use of other methods that ensure alignment of their action with the interest of their constituencies. Grantees report directly involving citizens in focus groups, conducting interviews with key constituents, and including citizens in strategic planning. Compared to these other tools Social Lab is applied slightly less, with 28.6% of ACCESS grantees using it. This is confirmed by the desk review pointing to the study by CTC, which states that the co-creation methods are not well￾known in Georgia and are therefore significantly underutilized. CTC will use the research data to develop a manual for Georgian CSOs and civic activists on how to organize co-creation processes in the Georgian context and support innovation and participatory planning.30 EQ2: How effective has the partner been in supporting CSOs in monitoring and influencing key government policies and processes? What policy areas do key informants mention as lacking in support? How relevant are these thematic areas/policies to citizens’ priorities and key development issues? What were the areas where ACCESS-supported CSOs achieved most notable results? Were there policy areas where ACCESS should have focused more resources to achieve better results to date, based on key informants and public surveys? ACCESS’s support to CSO’s in monitoring and influencing key government policies and processes was channeled through the PPAG program. The emphasis of this program was on grass roots level policy initiatives and emerging policy issues. Initiatives supported by ACCESS were directed at influencing public opinion, researching policy problems and initiating and lobbying for policy/legislative changes. In addition to funding the policy related initiatives of CSOs ACCESS was also directly involved in organizing public discussions via bus tours in which representatives of ACCESS and partner organizations met citizens in rural areas and discussed important policy problems with them31 . Most policy initiatives supported by ACCESS fell into 4 main thematic categories, including: 1) Countering disinformation 2) Environmental issues 3) Cultural heritage and urban development 4) Road 30 ACCESS Sixth Annual Report, 2020 31 Based on ACCESS annual reports CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 29 safety. In addition, there were projects on teaching policy analysis, improving labor standards as well as issues such as supporting media reform environment, and explaining principles of European agricultural policy to relevant stakeholder groups. COUNTERING DISINFORMATION This area was selected by the ACCESS and USAID/Georgia as an issue which had not received sufficient attention from other NGOs and donors. According to one ACCESS staff: “We realized that there is fundamental complex problem with disinformation, like we were telling people about technical regulations and they [were] responding with speaking about gay rights and how EU wants to spread them. Nobody was funding disinformation at that point except for a minor project of one of the German foundations and that triggered public discussions so that now disinformation is recognized by nearly 40% of participants of the survey as a problem.” [KII, ACCESS] Countering disinformation and promotion of Euro Atlantic values is an area in which ACCESS partners achieved the biggest result in terms of introducing a problem into public discussions, making this issue constantly discussed and influencing public opinion and policy as a result. Disinformation which was only marginally represented in media and public opinion before ACCESS, became one of the major problems discussed in the public sphere. The number of CSOs and donors working on this issue multiplied, with organizations supported by ACCESS becoming active in researching/monitoring traditional and social media, the latter being the most important channel for spreading disinformation in Georgia. Work in this area did culminate in influencing state policy on countering anti-western disinformation. For example the Parliament communicated the findings of CSO monitoring work to executives, and this influenced the government to include the problem of disinformation into official communication strategy documents32. The Strategic Communication Department was established in the Prime Minister’s office last year, which is improving prospects for cooperation between civil society and government. However, further coordination between the government and CSOs is required, and a high level of distrust between government and civil society still exists. The negative attitude of representatives of government toward NGOs working in this area was summarized by a GoG key respondent: ‘It was very difficult to use any of this research in our work - when we realized researchers , donors, interviewers were persons affiliated with political groups and interviews were also done with experts affiliated to political groups….It is not a mistrust toward CSO , but rather some question marks about motivation and qualification of people who were doing it. When you do this kind of research you have to look comprehensively and not from one angle.’ [KII, GoG] ENVIRONMENTAL ISSUES ACCESS CSOs were successful in changing environmental policy at the regional level. In Rustavi, a local organization “Gavigudet” mobilized the local community around the problem of industrial air pollution. Through a combination of information campaigns, protest actions and lobbying activities they achieved the introduction of desired regulations into legislation. In Poti, a coalition of local NGOs and community activists achieved recognition of the problem of pollution by the local government, who has now begun 32 Government of Georgia (2017) Communication Strategy on Georgia's Membership to the EU and NATO for 2017-2020 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 30 monitoring the level of air pollution in and around the city. However, according to local community activists, there is still insufficient engagement on this problem from other citizens, civil society organizations and experts. As a result, they think that there is not enough pressure on government to resolve the problem. Currently, according to activists, there are little or no signs of willingness from the side of government to tackle this issue, beyond monitoring. In another project, ACCESS is supporting NGOs to help activists in their fight against construction of Namakhvani hydropower station in Western Georgia, currently an important political and policy issue in Georgia. ACCESS has also helped its grantees working on environmental issues to exchange their experience by organizing meetings between them. CULTURAL HERITAGE AND URBAN DEVELOPMENT ACCESS partner’s work in the cultural heritage and urban development area has had mixed success. ACCESS has organized policy forums on urban issue in which more than 200 citizens took part together with experts and activists33. In Tbilisi, good practices of cooperation between municipality and ACCESS –supported CSOs working on cultural heritage were developed, highlighting participation in monitoring of the renovation work of historical districts, voice in decision-making and cooperation in preparation of legislative amendments. Important new issues related to urban development policy were raised in an ACCESS funded project on Tbilisis’ architectural biennale. For the biennale, ACCESS’s partner organized several studies and public discussions and prepared media materials for popular magazine and TV stations. While relations between civil society and local government in Tbilisi (which includes more than 1/3 of the total population in the country) remains complicated, with instances of both close cooperation and confrontation, role of civil society in raising urban development problems is unquestionable. These issues are constantly discussed in the media and public sphere in general. The role of ACCESS in supporting civil society and facilitating debates on these issues is easily recognizable, as other donors are not so active in this field.34 However, in Batumi, the struggle for preservation of urban heritage is impeded by lack of cooperation from local and sub-national governments as well as a weak level of mobilization of local society around this problem. As a result, despite constant small scale protest actions, construction of commercial infrastructure is continuing even in sites of historical – cultural importance (Batumi seaside-boulevard etc.) Activism on environmental and cultural heritage has also led to disputes between civil society and the private sector. In the examples of Batumi and Poti described above, as well as several projects on Tbilisi urban development, activities undertaken by the private sector and, in some cases, backing of these activities by local/central level governments, have been criticized by representatives of civil society and scholars as being harmful35. When representatives of private sector resist the introduction of new 33 East West Management Institute. (2017). ACCESS Third Annual Report. 34 East West Management Institute. (2018). ACCESS Fourth Annual Report. 35 Gogishvili, D., & Harris-Brandts, S. (2020). Coinciding practices of exception in urban development: mega-events and special economic zones in Tbilisi, Georgia. European Planning Studies, 28(10), 1999-2019; CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 31 regulations that benefit citizens, and are criticized by civil society, the government can step in and play the role of a mediator between civil society and businesses. According to a civil society key informant: ‘Civil society can get involved when the other side is a private company – supporting trade unions, etc. And civil society should not be afraid of going against the private sector. Now the government is trying to use Public-private partnerships more and we should realize this new context.’36 [KII, ACCESS Grantee] ROAD SAFETY Road safety is an area in which ACCESS partners have achieved the biggest success not only in influencing public opinion, but also in finding common language with the government and pushing important legislative changes as a result. The Government’s willingness to tackle the problem has played an important role in this success. ACCESS partner, “Alliance on road safety”, together with government institutions, prepared amendments to national legislation to address the problem of high fatality as a result of car accidents in Georgia. These legislative changes were supported both by the ruling party and opposition. Enactment of these legislative changes has resulted in a decrease in fatalities by 11% and in injuries by 14%37. Another project on road safety, particularly on causes of road accidents involving so called “marshrutkas” (minibuses for travel outside of Tbilisi), was conducted with support from ACCESS by a partner research organization. The project was innovative for the Georgian policy context, using an experimental approach, and produced important findings. However, the policy impact of this project was limited due to the absence of a government agency that can introduce relevant regulations. FACTORS EFFECTING CSO SUCCESS IN CHANGING POLICY The ET found that the context for each specific policy area plays an important role in successfully changing policy. These contextual factors include attitudes of government officials and bureaucracy working in a particular area towards civil society, the extent of cooperation, including the existence of more or less stable ties between representatives of civil society and bureaucracy or politicians, citizen mobilization around an issue, and level of convergence on the issue/problem between representatives of CSOs and government. In both Rustavi and Poti, success in dealing with air pollution was achieved by engaging citizens. However, in Batumi both weaker mobilization of the community and lack of political cooperation failed to achieve results. One of the most important factors affecting the attitudes and level of cooperation is the extent to which the policy issue is “politicized”, or, in other words, is under debate between the opposition and ruling party. As noted by a Georgian civil society key informant, ‘You can develop a good policy paper or research but it depends on which stance the government has. For example in [EU-Georgia] association agreement related issue government wanted the same things as we and the government was responsive.’ As noted in Lomia and Lomia (2020), the dialogue between the government and civil society takes place in one of two ways. First is a dialogue of "constructive cooperation", related to the "good governance" approach which has had considerable influence over governance during the past decades and was 36 The CSO was implying that PPPs are used to implement and justify projects that have been criticized by CSOs. 37 East West Management Institute. (2018). ACCESS Fourth Annual Report. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 32 consistently supported by donors38. In the dialogue of “constructive cooperation”, civil society organizations are represented as potential or actual allies in the process of solving concrete problems and as valuable contributors to the discussion over these problems. Dialogue with civil society is seen as an essential part of democratic process. Key informants from GoG that met with the ET often emphasized the active role of government in building this cooperation, sometimes without paying attention to the facilitating role of donors or international organizations in this process39. Some government officials stressed that while the last word in the relation with civil society belongs to the government, all initiatives coming from civil society are discussed and many fully or partially adopted. An example of this type of dialogue was provided by a GoG key informant: ‘We always consider some of the recommendations of the civil society – there is no process where their opinions are not included, for example in constitutional or election reform. Not all recommendations of course. In last four years civil society was actively involved in all issues: they submitted opinions, provided recommendations. .... In general, when the political party sees something as right, they follow what they think is right. But there are other cases, when CSO recommendations are not taken into account...because there is no mechanism for the implementation of the proposed law.’ [KII, GoG official] For some representatives of government, active involvement of civil society is an additional safeguard to prevent criticism from some parts of society. These key informants were strongly in favor of measures such as creation of a public board of trustees or other kind of consultative body. They perceived civil society as an ally in the process of communication with society and in monitoring problems of common interest. The experience and qualifications of representatives of civil society is seen as a valuable addition to experience and qualification of representatives of government and bureaucracy. The second type of dialogue is more critical toward civil society, with the government having a perception that civil society creates obstacles to cooperation. Critical attitude toward civil society is based on several points. One point is perception of some organizations as being affiliated with oppositional parties. This bias, according to one GoG key informant, is influencing results of studies produced by representatives of civil society. Another line of criticism is related to the qualification of representatives of civil society. Government representatives sometimes mention a lack of sufficient knowledge and skills in civil society, which, according to them, makes it difficult to consider civil society as an equal partner. Their recommendation to donors is to work on increasing the qualification of representatives of civil society and to attract new, more qualified people into civil society. In interviews civil society key informants mentioned the clear deterioration in relations between civil society and government. The current period is often compared to the first years after the coming of "Georgian Dream" to power, roughly before 2016. That period, according to participants, was characterized by openness of government and willingness to accept initiatives from civil society. Currently, there is more distance between the two and the government's position is often hostile. Main 38 Lomia, T., & Lomia, E. (2020). Economic and political support of the European Union to Georgia: retrospective analysis of the EU-Georgia relations. International Journal Vallis Aurea, 6(1), 35-43. 39 On donor involvement, see Rinnert, D. (2015). The politics of civil service and administrative reforms in development— explaining within-country variation of reform outcomes in Georgia after the Rose revolution. Public Administration and Development, 35(1), 19-33. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 33 reason for this hostility is the increasing influence of informal power on government and centralization of decision-making40. Often, there are major agreements between views of politicians or bureaucracy but these agreements are never realized due to the unpredictability and volatility in state policy led by informal actors. Open attacks from representatives of ruling parties are appearing more often. This includes attacks on the public image of leaders of civil society, with frequent reference to previous affiliation of some of them with the Saakashvili period ruling party or state apparatus. NGOs are depicted as part of an opposition coalition led by the United National Movement. Also, attacks on NGO sometimes have a form of exclusion from civil and political dialogue and denial of previously recognized role of facilitator. This last point was visible during the post - election crisis, when the ruling party openly went against participation of NGOs in dialogue as facilitator. According to one of the representatives of civil society: ‘Oh, they will soon arrest us all.... I think that the support to NGOs should be oriented on institutional support, not a project based support. As situation changes quickly and we adjust – we had an institutional grant from OSGF and could do new projects as the needs emerged. You can’t always tell beforehand how the situation will change. So donors should provide institutional support to well-established organizations. This does not exclude support to small and regional organizations of course.’ [KII, CSO] Other representatives of civil society suggest a more nuanced picture of relation with government. According to them, there are issues where cooperation is more successful and issues where this cooperation is less successful. Success is related mostly to less "politicized" policy issues, in which there were no disagreements between political parties and there was a high level of willingness to solve problems. Key informants from the government also agreed with the above and considered this to be important. Some of the projects supported by ACCESS were focused precisely on this kind of issue, including road safety regulation and some projects on urban heritage. For government officials, NGOs are a source of valuable ideas and legitimacy. Around “non-politicized” or “technocratic” issues, more or less stable coalitions or policy networks are formed. Some civil society groups and bureaucracy are cooperating as a part of these informal networks. This kind of network exists in case of problem of road safety, where ACCESS is actively involved. Another example of a network (informal) is one formed around urban development issues, consisting of representatives of the Tbilisi Mayor’s office and civil society activists. At the same time, in other policy areas, there are civil society groups which are closely cooperating with government and groups which are excluded from these networks. Communication between the latter group of civil society and government tends to be minimized or often has a confrontational character. POLICY AREAS LACKING IN SUPPORT Citizen priorities, as reflected in public opinion polls, have remained relatively unchanged in the last decade in Georgia. The top priorities are socio - economic issues, such as unemployment and poverty, 40 On informal power Aprasidze, D., & Siroky, D. S. (2020). Technocratic populism in hybrid regimes: Georgia on my mind and in my pocket. Politics and Governance, 8(4), 580-589 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 34 followed by problems related to COVID-19 during the last several months41. However, the work of civil society does not reflect these priorities. Key informants from the government and civil society noted that attempts to switch focus to social and economic issues are hampered by a shortage of expertize among CSOs on these issues. Another problem noted by informants from various sectors is the quality of policy research in Georgia, including policy research in areas where civil society expertize is most developed such as good governance and human rights. As a result, positions of civil society in policy dialogue are weakened. According to respondents, conclusions of some research studies are often reflecting pre-existing views of authors of these studies rather than findings based on empirical data collected during the research. Problems with the low level of competencies with policy analysis was also mentioned in ACCESS annual reports42. According to this source, regional CSOs have difficulty formulating policy problems, which is one of the reasons why the number of ACCESS’s grants to local CSOs were less than targeted. In the web survey of ACCESS grantees, 48% of respondents agreed that there are critical issues in the community where their CSO operates that are underserved by civil society organizations. When asked to list underserved issues the following were mentioned; education, primary health care, unemployment, child protection, persons with disabilities, LGBTQ populations, economic inequality, engaging citizens in the decision-making process/civic activism, and grass roots initiatives. In some FGDs with citizens education was named as a social priority which requires more attention from donors and civil society. According to one respondent, an important problem in this area is the high level of centralization and control by the government in the school system. Other focus group discussants noted that the problem with the education policy is a constant, rather chaotic attempt to reform it with many reforms that never get completed. According to other discussants, there is a need for more informal and adult education initiatives, which will create opportunities both for youth and older people in the regions. Representatives of CSOs in interviews, and citizens in FGDs agreed that civil society should be more active in the socio-economic sector. Some of the civil society activists suggested limited interventions, such as empowerment of communities in the process of local development and more active role in informal education, as a measure not exceeding power of the CSO sector in Georgia. Others pointed at the necessity of increasing expertise in these fields and increasing participation in policy debates on social and economic problems. Increased involvement in solving social and economic problems was named by members of the ACCESS team as one of the most important goals of the whole program. However, while ACCESS has been mostly successful in increasing citizen debate and engagement, on technocratic issues (for example, urban development, road safety, and environment) and effecting policy in some areas, social and economic issues which are more political and encounter resistance from policy-makers have had a limited focus. Limited capacity of CSOs in this area also continues to be a barrier. Representatives of all stakeholder groups that the ET talked with agreed that the current agenda of civil society is not always reflecting everyday problems of members of society. Poor communication between CSOs and other stakeholders also harms the image of civil society in the country. In a KII one 41 According to the latest 2000 Caucasus Barometer survey, conducted by the Caucasus Research Resource Center, the most important concerns for citizens were unemployment (33%), COVID - 19 (17%) and poverty (17%). In a February 2021 poll, commissioned by the International Republican Institute (IRI), unemployment was stated as the highest concern by 49% of respondents, followed by high prices (16%), poverty (8%) and COVID-19 (7%). 42 Fourth annual report. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 35 representative of the sector suggested: “People just don’t understand what NGOs are doing. Sometimes they just want charity”. Explaining the linkage between bad governance and socio-economic problems is also needed. One donor key informant noted that ‘it is necessary to explain to the public how high levels of corruption affect ordinary citizens, for example in the price of bread.’ POLICY AREAS WHERE ACCESS SHOULD HAVE FOCUSED MORE RESOURCES TO ACHIEVE BETTER RESULTS ACCESS representatives mentioned several directions of work in which they could have achieved better results if not facing a shortage of funds. Lack of sufficient support to social and economic issues was partly a result of insufficient funds and partly a result of shortage of good applications on this issue. ACCESS had limited possibility of continuing working in some areas, for example urban policy, after a large share of funds went to disinformation issues, which was a priority for USAID. Same is true with regard to policy forums. Informants state that overall, it was difficult to concentrate on a single or several policy directions as this would hurt the flexibility of the program in selecting applicants. EQ3: How successful has ACCESS been in working on local ownership of the Centers for Civic Engagement (CCEs) so that they continue to operate as politically neutral gathering spaces for public discourse? What is the potential of a Network of Centers for Civic Engagement (NCCE) to diversify services and increase demand for CCE services in each location, strengthen financial diversification, expand services to the private sector, sustain results to date, and continue developing on its own? Under ACCESS, CCEs have continued their previous role as neutral spaces for public discourse and expanded their role and visibility as a catalyst and enabler of civic engagement outside of Tbilisi. However, despite advances in providing fee-based services and reduced operating expenses, they will not be able to persist following ACCESS without core donor funding. This situation exists independently of the gathering restrictions necessitated by the COVID-19 pandemic, which have nevertheless exacerbated the CCEs’ financial struggles. The NCCE has made critical contributions to the CCEs’ ability to institute fee-based services, negotiate as a large network for grant funding and reduce operating expenses, and collaborate and learn from one another. However, some stakeholders wonder whether the NCCE is worth the significant Tbilisi-based expense when the CCEs’ financial situation is already so poor. For their part, CCE coordinators express gratitude and appreciation for the NCCE’s work, but question whether it is a luxury they can afford without core donor funding. CCE BACKGROUND Ten Centers for Civic Engagement (CCE) were established in 2011-2012 through the USAID-funded Good Governance in Georgia (G3) activity under a different political context: their main function was to provide a safe space for dialogue and civic engagement in the regions of Georgia in the run-up to the 2012 Parliamentary Elections. Following the transition of power and ensuing political polarization in the country, the CCEs continued to operate as spaces for discussions and multi-stakeholder engagement, quickly gaining popularity and demand for the services provided. ACCESS inherited the CCEs from G3 with an objective to enhance the local ownership of the CCEs to ensure their long-term continuity independently of USAID funding. In addition to diversifying CCEs’ funding and ensuring their financial sustainability, ACCESS was tasked to maintain their core function as CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 36 a “politically neutral gathering space for public discourse”.43 The following sections describe the extent to which ACCESS and CCEs have achieved these objectives to date. CCES AS “NEUTRAL SPACES” THAT PLAY A CENTRAL ROLE IN CIVIC ENGAGEMENT OUTSIDE TBILISI KII respondents and FGD participants who were aware of the CCEs commonly described them as a “space,” often further specifying them as a “neutral space.” This perception fulfills the CCEs’ original conception as “politically neutral community centers providing safe space” for public discourse to a variety of stakeholders. Though originally established to encourage political participation during the dominant rule of the United National Movement (UNM), they remain relevant especially in the context of the power consolidation by the Georgian Dream government. With increased political polarization and allegations of intimidation and persecution of dissenting opinions, KII respondents maintained the view that CCEs serve as a safe haven for opposition parties as well as CSOs and citizens expressing critical views of the government and/or its policies. “Opposition still has hard time finding a neutral space… they feel safer in our space” [FGD, CCE Coordinators] “As some opposition parties can’t find space in regions due to the hidden control of the government, centers are helping them as well. No similar infrastructure exists” [KII, USAID Implementing Partner] Beyond serving as a neutral space for public gathering, FGD and KII respondents express myriad ways that CCEs encourage or catalyze civic engagement in communities outside Tbilisi. The CCEs are often described as spaces that give equal access to everyone. In some cases, as in Marneuli and Sagarejo, the local CCE is the only available space for civic engagement in the region. In all cases except Zugdidi and Gori, the CCE is one of only a few public spaces that are accessible to persons with disabilities. FGD and KII respondents describe CCEs as spaces that serve the following functions in their communities: • Creating space for communities to come together to discuss problems and find solutions; • Providing a venue for local or central government to present their reports or policy visions; • Permitting citizens to interact with local government on equal footing and pose pressing questions; • Catalyzing the formation of new CSOs, new ideas from local activists, and new cooperation between local actors; • Enabling newly emerged civic activists and/or newly formed CSOs to get organized and meet like￾minded individuals; • Connecting local activists with opportunities for grant funding; • Serving as an entry point to regions for Tbilisi-based CSOs and donor organizations; and 43 ACCESS Objectives and Anticipated Results, Objective 4. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 37 • Disseminating information and new opportunities for engagement for citizens. The latter function is especially true for youth and particularly profound in regions densely populated by ethnic minorities. For example, during the initial stage of the COVID-19 pandemic, the CCEs in Marneuli and Akhaltsikhe provided information on state regulations in Azeri and Armenian languages and also assisted the local residents with the registration process to receive aid from the Government of Georgia. Some KII respondents and FGD participants credited CCEs with increasing civic engagement in their respective communities and elevating locally important issues, including through Women’s Clubs which have emerged with the support of CCEs. The quotes below portray how meaningful some of these functions were to relevant stakeholders who spoke with the ET. “CCE is giving free access which is very important: even students can go and raise their idea or protest” [KII, Media] “The CCE for me was like a beacon of democracy because if anything happened in Marneuli, it was the Center for Civic Engagement where it took place” [FGD, Marneuli] “I have been to the Center for Civic Engagement several times and it is a very interesting space. There you will learn about the main issues that exist in your city and your municipality…Of course, civic activism won’t exist without such spaces” [FGD, Zugdidi] During elections, CCEs play an additional important role in local civic engagement. Once the elections are called, the CCEs organize meetings with competing parties and candidates: each gets an opportunity to present their platform and program so that citizens can make an informed decision. In addition, CCEs serve as election media centers from which journalists can report on election-day; as not all media have regional offices. While CCEs have fulfilled their intended function to date, they are increasingly competing with alternative spaces emerging in the regions of Georgia. The Ministry of Justice (MoJ) of Georgia currently operates 85 Community Centers throughout the country where, in addition to receiving state services, local and central governments use the space to meet with communities. CSOs can use the space for meetings with their beneficiaries. In addition, with the support of the European Union, several regional CSOs increasingly play the role of hubs for local civic engagement activities. In recent years, hotels have increasingly offered conference facilities in the regions competing with CCE services, especially for the segment of the market that is able to pay for the use of such facilities (e.g. businesses and international organizations). “They need better infrastructure, especially if they want to be self-sustainable because there are lot of alternatives like hotels.” [KII, ACCESS Key Partner] The emergence of these alternative spaces has coincided with depreciation of CCE premises and equipment and relocation of CCEs from prime locations in their respective cities to less visible, lower cost locations. The confluence of these factors has been met with dissatisfaction among primary users of CCEs during a time that USAID had hoped they could begin to operate on their own. However, the CCEs still maintain three core competitive advantages over alternative spaces: 1. A reputation of being free and accessible to everyone; CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 38 2. Good connections with local governments resulting in an ability to serve as a convener; and 3. A large and diverse database of local residents, which is invaluable for those outside the region wishing to engage with local populations. PROSPECTS FOR LOCAL OWNERSHIP AND FINANCIAL SUSTAINABILITY OF CCES ACCESS undertook a number of actions to assist the CCEs with diversification of their funding and increasing their sustainability: ACCESS commissioned market research in Year I (October 2014 – September 2015) to calculate potential revenues from fee-based services and launched paid services in all CCEs in Year 2. In Year 2, ACCESS established the Network of Center of Civic Engagement (NCCE) to “provide guidance and overall support to CCEs”. The same year, the NCCE was registered as an independent Georgian NGO. In Year 3 the NCCE developed a marketing strategy and action plan to promote CCE services and expand its customer base. Operation costs across the network were reduced by $200,000 USD (from $570,000 a year to $370,000). In addition, ACCESS has tried to secure free space for CCEs from local governments, though no such space has been secured to date. As a result of those interventions, ACCESS was able to support the CCEs to generate about 20% of their income from paid services while the NCCE was able to secure around $250,000 (USD) in grants from different donors in Years 3-6. However, since the onset of the COVID-19 pandemic, the income of the CCEs has sharply declined owing to repeated lock-downs and social distancing requirements across the country. Since early 2020, the demand for meeting spaces – the key revenue-generating service that CCEs offer – is negligible. Alternative strategies to pursue local ownership have yet to provide a solution for ensuring CCEs’ survival without core donor funding. The ET finds that the CCEs will not be able to operate as independent entities without core donor funding after ACCESS’ conclusion. Pessimism regarding the financial sustainability of CCEs. ACCESS’ Performance Monitoring Plan (PMP) operationalizes the objective of “local ownership of CCEs” with a goal that seven out of the ten CCEs operate independently of project funding, covering 100% of operational expenses through non￾USAID funds. As of this report, no CCE has achieved this goal. Although the share of operating expenses covered by non-USAID funds increased from 0% to 36% from the project inception to October 2019, it has decreased to 19.4% as of October 2020 due in large part to the gathering restrictions necessitated by the ongoing COVID-19 pandemic. An overwhelming majority of KII respondents with whom the ET met believe that the CCEs will not become fully financially sustainable and would need donor funding in the foreseeable future. Important challenge for self -sustainability is an absence of own office space and consequently, rent expenses. Many of those more familiar with the CCEs (coordinators, USAID, ACCESS) expressed an opinion that core donor funding must come from USAID, as the CCEs are so closely affiliated with USAID that it is hard for them to attract significant funding from other donors. Indeed, the CCEs are known by some citizens and other donors as “USAID Centers.” According to members of CCE staff, this image of being closely affiliated with USAID is sometimes creating a barrier in the process of fundraising as big donors do not want to spend funds on projects which are already supported by other big donor. Unlike many other CSOs in Georgia, CCEs do generate a portion of their income through commercial activities. However, market research commissioned by ACCESS found that, even at full capacity, the CCEs cannot generate sufficient revenue to cover all of their operating expenses. Some KII respondents further raised concerns that commercializing CCEs would negate their core function as open spaces to CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 39 which everyone has equal access. Free access is the core characteristic for which CCEs are most known and used. “I know there are talks about making them self-sustainable but I do not really believe Georgian economy is strong enough to sustain this”, [KII, ACCESS Grantee] “As it is becoming commercialized, people can’t come” [KII, Media] “If they want CCEs to be a public space, it can’t be commercialized. If it is commercialized, it’s a corporate space, not a public space”, [KII, ACCESS Grantee] Challenges and opportunities for CCEs presented by the COVID-19 pandemic. The COVID￾19 pandemic and subsequent social distancing requirements and restrictions on public gatherings virtually halted the main income-generating service the CCEs provided: renting space for different events, such as trainings, meetings, conferences, etc. Although revenue through fee-based services was effectively eliminated, CCEs were able to reduce their operating expenses by half with reduced rent and utility expenses. The CCEs further seized opportunities to expand their base by offering new services to the affected communities, such as assistance with registration for state aid and dissemination of information about the novel coronavirus. These opportunities helped the CCEs engage with citizens more directly than they had historically, and served to expand their role beyond being perceived merely as a space. Some of activities of the CCEs moved online, generating more viewership and interactions on social media. While these opportunities create the potential for expanded reach with citizens once the pandemic and associated gathering restrictions end, they do not address the absolute revenue shortfall for the CCEs at precisely the time they were supposed to operate independently from USAID funding. Alternative strategies to pursue local ownership of CCEs. KII respondents suggested multiple alternative strategies that CCEs could pursue or are pursuing to improve their financial sustainability and promote local ownership. These include diversifying their customer base, decentralizing to operate as individual NGOs, attempting a community foundation model, and identifying niche demands in their community which CCEs are uniquely placed to fulfill. We summarize each of these suggestions below, though none have yet proven promising for ensuring CCEs’ survival without core donor funding. To date, CSOs have been the main user of CCEs, with lesser use reported by professional and business associations and community leaders/civic activists. No other group constituted more than 5% of CCEs’ users as of October 2019.44 The user profile has remained more or less constant since ACCESS’ inception. The NCCE has worked to diversify the profile of users with the aim of diversifying the demand for paid services. CCEs’ venues are now sub-leased to a number of international and local organizations and a representative of the Public Defender. There are some efforts underway to engage businesses (e.g. insurance companies) to market their products through CCEs. However, at the time of evaluation, the results of those efforts were insignificant. Attempts to secure private sector support have generally not succeeded (see EQ4 for additional detail). 44 For this reason, although the ET intended to sample citizens who were not formally involved with civil society for FGDs, citizens sampled from the database were mostly formally affiliated with CSOs in some way. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 40 “We started to register CCEs as independent entities which would give them opportunity to negotiate small projects with donors and to talk to local enterprises and intensify talks on CSR” [KII, NCCE] “Decentralization would be very good for financial viability of the centers” [FGD, CCE Coordinators] Currently the NCCE is in the process of registering each CCE as an independent NGO in an attempt to decentralize ownership. The objective of this decentralization is to attract funding for small projects, to have more independence in terms of the projects for which they could apply, and to seek opportunities to engage local businesses. All of these are currently unavailable to CCEs in their capacity as a national network, but would become available to them in their new capacity as local NGOs. In addition, with annual income being reported at the CCE level rather than aggregated to the NCCE level, CCEs will be able to apply for value added tax (VAT) exemption, thus avoiding an expense equal to 18% of their income. This decentralization is equally supported by CCE coordinators and the NCCE, though they understand that decentralized ownership is more advantageous for some CCEs than for others. Relevant KII respondents named coordinators’ skills and CCE infrastructure as factors determining the level of sustainability of individual centers. Some coordinators have more entrepreneurial skills or are better at fundraising or outreach than others. However, the CCE infrastructure is depreciating in all locations - something that has been noted also by those KII respondents who had used CCE premises. In addition, the CCE coordinators thought that paying for rent for ten and more years instead of purchasing the property has been a lost opportunity on behalf of USAID to make CCEs sustainable without its support. ACCESS hoped to transform some of the CCEs into community foundations, which would aggregate resources from individuals, the private sector, government grants, and other sources to redistribute within their local community as determined by local community members. Early efforts to this end were delayed when the COVID-19 pandemic caused ACCESS to cancel planned peer-to-peer exchanges with community foundations in Eastern Europe. Regardless, KII respondents expressed doubt that sufficient funding could be aggregated for a community foundation when insufficient laws, regulations, and cultural norms exist to incentivize public philanthropy nation-wide. While state funding for CSOs is growing, it is mostly focused on service delivery, and local municipalities are still not allowed to award grant funding.45 Private sector engagement remains low. In general, the culture of giving in Georgia is emerging and is mainly directed on concrete problems - such as helping those in need, collecting money to build a playground/a community café, etc. As one KII respondent summarized: “Maintenance of CCE would be little abstract goal for which people may not donate”. [KII, USAID IP] KII respondents believe that, for CCEs to remain relevant, they need to find a unique, niche role they are playing in relevant regions. Some CCEs are better positioned to do so based on their locations, others face competition from other CSOs and/or private and commercial entities. Some of the ideas expressed by KII respondents and FGD participants included offering co-working space, language classes, assistance to students for entrance exams, bridging local challenges with CSR initiatives, and assisting communities with access to information. The NCCE once entertained an idea that the CCEs would be well￾positioned to play a larger role in promoting Georgia’s European integration agenda through serving as 45 For more information see 2019 CSO Sustainability Index. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 41 local consultation for raising awareness about DCFTA.46 None of these candidate niche roles have advanced past the ideation stage, however, and none are likely to do so by ACCESS’ conclusion. “There are quite a few NGOs in this country. So unless they provide something unique, just becoming yet another NGO won’t have much impact” [KII, ACCESS Grantee] An alternative strategy for local ownership that is not recommended by respondents is assistance from local government, which many respondents feel would compromise CCEs’ neutrality. In early years of implementation, ACCESS tried and failed to negotiate with the central and local governments for free provision of space to the CCEs. The ET asked KII respondents and FGD participants specifically about how they perceive such local government support would affect the CCEs’ hard won reputation for neutrality. The overwhelming majority feel that significant government funding would cause CCEs to no longer be perceived as neutral spaces that are accessible to everyone. In addition, some fear funding from government might affect the CCEs’ staff, for example with coordinators being replaced by people close to government. They fear this might further affect the content of the events the CCEs organize. “If the Center receives funding from Sakrebulo, they will hold governmental meetings, they cannot be neutral, somehow they will be supporters of government. Therefore I do not want it to be funded by Sakrebulo” [FGD, Marneuli] “If Sakrebulo… funds CCEs, can you imagine what would happen, first of all, to people working there? This would become yet another bureau of local government, which is against their initial aims” [KII, ACCESS grantee] Some respondents indicated that the selection of a neutral location (e.g. town library versus City Hall) or enactment of clear access policies might mitigate some of the risks of losing perceived political neutrality. Even so, some respondents opposed local government support of CCEs for other reasons, including possible crowding out of funds for other citizen concerns. Regardless of rationale, there is very little support for local governments providing significant support to the CCEs. ROLE OF THE NCCE While the NCCE has played a critical role advancing the CCEs’ toward ACCESS’ objectives to date, there are open questions as to whether its continued services are worth its substantial cost of operations. As established in previous sections, the NCCE has helped CCEs make significant advances in financial independence since its inception in 2015. These include negotiating lower rent costs, commissioning market research to identify and rolling out fee-based services, and successfully applying for various grants. NCCE staff assert that these advances and similar ones in the future are possible because the CCEs can negotiate as a unified, nation-wide network rather than ten separate NGOs. They further assert that large donors are more comfortable working with such a large network than they would be working with unaffiliated local NGOs. The NCCE provides administrative, marketing, capacity-building, and fundraising support to individual CCEs while fostering the connection among CCEs. CCE 46 NCCE role in socioeconomic development, page 1 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 42 coordinators express gratitude for the NCCE’s work on administration and financial management, leaving more time for the coordinators to work substantively on civic engagement. “Main value of network is that there is network of centers with similar values and we need individual CCEs to sustain this network. No one, including donors, needs separate CCEs, just one more local NGOs” [KII, NCCE]. However, respondents outside of the NCCE express doubt that the CCEs can afford maintaining a sort of headquarter office with four staff based in Tbilisi while they are struggling so much with covering an estimated $370,000 in annual operating expenses across the network and regional centers.47 CCE coordinators express gratitude and appreciation for the NCCE’s work, but question whether it is a luxury they can afford without core donor funding. The CCE coordinators believe that decentralization will offer them more opportunities to attract funding, including for small-scale individual projects, without needing the assistance of the NCCE. As most of the coordinators have remained in place since the CCEs’ establishment, they express strong interpersonal connections that they feel would persist and enable inter-CCE coordination of activities without needing further coordination by the NCCE. EQ4: How successful has the partner been in facilitating private sector engagement (exploring areas of mutual interest and CSR initiatives) and other inter-sectoral cooperation including to secure private sector funding and support? Historic engagement between the private sector and civil society in Georgia has been characterized by two parallel and exclusive spheres of operation, leading to mutual mistrust. ACCESS made important progress by creating the CSR Club, which served as a platform to facilitate more direct engagement and collaboration between private sector and civil society. Informed stakeholders universally consider “breaking the ice” between private sector and civil society to be a necessary first step for broader engagement in the future. Actors in both groups have indicated interest in taking established relationships further. However, there is still very limited direct funding of CSOs by private sector in Georgia, and even less engagement for other purposes, like influencing government policy for mutual gain. Considerable room for future improvement in private sector engagement remains, particularly in engagement outside of Tbilisi and in creating formal, established channels for CSOs to apply for funding from private companies. NASCENT ENGAGEMENT OF PRIVATE SECTOR WITH CSOS/CIVIC LIFE According to the ET’s document review and corroborated by interviews with CSOs and private sector representatives, the cooperation between civil society and the private sector is in a very nascent form. Both CSOs and private businesses express reservations when it comes to engagement. In a 2019 assessment of the civil society sector in Georgia commissioned by ACCESS, CSOs rank businesses as the least cooperative (1.5 on a five-point scale) on addressing the most important national issues, ranking behind international partners, media, civic activists, academia, and the government.48 According to the same report, businesses perceive that CSOs are “political” due to their watchdog 47 Per ACCESS Year Four (2018) Annual Report, the most recent estimate available. 48 Assessment of Civil Society Sector in Georgia, page 23. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 43 activities and thus shy away from association with them. On the other hand, CSOs believe cooperation with businesses might be damaging to their reputation.49 The same sentiments are expressed in the 2019 CSO Sustainability Index report: “Businesses are most likely to engage in apolitical charitable activities but are wary of forming links with vocal watchdogs. Advocacy CSOs similarly view cooperation with large businesses as a potential risk to their reputations.”50 The Civil Development Agency’s (CiDA) CSR Survey, conducted for the first time in 2019, found that the private sector rarely expressed desire to partner with NGOs. Among 1,053 small, medium and large-size businesses surveyed, only 25% considered getting involved in a partnership project with an NGO. 51 KII respondents the ET interviewed expressed similar views. While they noted some progress in the relationship between the civil society and private sector in recent years, this engagement remains infrequent, unstructured, and in need of further development. “I can say that those relationships between the civil society and the private sector have progressed positively so far and it's still progress. However, there is still huge room for improvement.” [KII, Private Sector] “There is some relation. In cases when they need each other – experience, speakers. But both sides view each other as separate tribes. Businesses think they know what they are doing, NGOs think they should be talking to business like experts.” [KII, ACCESS Grantee] According to key informants, one of the factors determining the weak engagement between the two sectors is Georgia’s low level of economic development. Businesses are under-developed and very few companies have dedicated CSR budgets. CiDA’s CSR Quantitative Survey indicates that only about 6-7% of medium- and large-size enterprises plan CSR activities in their annual budgets. The COVID-19 pandemic has further deteriorated the economic situation in the country, especially affecting small and medium size enterprises. As such, businesses are left with little disposable income to fund CSO initiatives. Those businesses with enough income to partner with CSOs prefer to partner with leading, established CSOs rather than with CSOs that are less known. “Development of country means strong civil society and strong business. However, these relations are currently very weak. Only big companies can afford this cooperation. It is about resources and funds available to companies and leaders” [KII, Private Sector] Mutual mistrust is the main obstacle for cooperation with companies who can afford it. The ET finds that businesses do not have confidence in the capacity of CSOs and are not able to check credibility of their proposals – either due to lack of time or internal procedures to do so. ACCESS’ 2019 civil society assessment found that businesses feel they are asked for donations by CSOs without any assurances as to mechanisms for accountability and reporting. At the same time, CSOs are blamed for being attuned to donor funding and having no skills on how to attract attention from the private sector. Businesses have suggested CSOs join business associations to attract this attention, but CSOs thus far have been hesitant to do so. 49 Ibid 50 CSO sustainability Index, page 100 51 CSR Quantitative Survey. Page 50. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 44 One reason for this hesitation is the recent trend of civil society actors who try to protect citizen interests vis-à-vis big companies and corporations. As such, they perceive the private sector as an enemy, rather than a friend. CSOs’ reportedly suspect that private companies only donate money to sanitize their public image rather than pursuing concrete improvements for citizens. Some CSOs, especially watchdog organizations, refuse to receive funding or in-kind contributions to avoid undermining their own reputations. “We don’t need SOCARS [Azerbaijani Oil Company] money to support democracy. They have to share our values…If you look at the big businesses, if you tell them – we want a project on kleptocracy, of course, they will be against you, as they benefit from the current system. There are very few businesses with right values. Until businesses are independent and judiciary is free, I can’t imagine cooperation between NGOs and private sector.” [KII Respondent, ACCESS Grantee] “They [CSOs] start criticizing private sector instead of finding common ground. We all know that labor conditions in Georgia are bad, but when you are at an event where you are trying to pitch an idea to business, you do not start your speech with saying that businesses underpay their employees.” [KII Respondent, ACCESS] “Private sector is more willing to finance CSOs than CSOs are to take risk of partnering with private sector and damaging their reputation. We need to continue working with civil society. In addition to believing in private sector, we need to teach civil society how to talk to businesses.” [KII respondent, ACCESS] While findings from the ET’s literature review and qualitative data are consistent for the broader civil society sector, responses to the ET’s online survey suggest CSOs who received ACCESS grants may be more willing to work with the private sector than their counterparts in Georgian civil society writ large. Specifically, over half of the surveyed respondents at least somewhat agree that “companies and other private sector actors are valuable and trustworthy partners to CSOs like mine.” Less than 20% disagree with the same statement. Nearly half of the surveyed respondents report receiving funding from the private sector in the past. Only three of the twenty four respondents who have never sought such funding indicated that they would refuse to ever consider accepting funding from a private company or other private sector organization. The other twenty one indicated that they at least might consider such funding. Those who refuse to consider accepting funding cite the reputational damage this may cause or the risk this poses toward the values of their organization. ACCESS’S CONTRIBUTION TO FACILITATING PRIVATE SECTOR ENGAGEMENT ACCESS primarily encouraged private sector engagement with civil society through the promotion of corporate social responsibility (CSR) in Georgia with the assistance of CiDA. Key informants credit ACCESS and CiDA’s CSR Club with “breaking the ice” between civil society and the private sector. While key informants from the private sector and otherwise acknowledge the ad hoc and fragile nature of this connection, they also expressed willingness to take developing relationships further. Although ACCESS considers breaking the ice between the two parties to constitute a necessary first step in broader engagement, there is no clear vision for the development of such broader engagement at present. ACCESS and CiDA’s CSR club. According to KIIs with ACCESS staff, ACCESS aimed to address the mutual mistrust between CSOs and private sector through “baby steps” centered on creating a shared CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 45 space for interaction in lieu of the parallel worlds in which CSOs and private businesses previously operated. ACCESS thus partnered with CiDA to facilitate private sector engagement, mainly through supporting the establishment of the first-of-a-kind CSR club. The CSR club aimed to bring the two sectors to sit at the same table. In the beginning, the club membership included 71 business organizations, including small-, medium- and large-size companies. Throughout the program years, the CSR club had notable success in strengthening linkages between the two sectors, mainly through organization of CSO fairs and compiling a database of the CSR priorities of the CSR club members to match CSO and citizen priorities and map opportunities for partnerships. In addition, some examples of successful engagement included employment of CSO beneficiaries by CSR club members, allocation of funds for program funding (including assistance to relief efforts and families affected by flooding in Tbilisi), and awareness-raising activities around sexual harassment, gender equality, and disability issues. The CSR club’s annual CSR awards allowed ACCESS to coach club members towards sustainable, systemic CSR initiatives rather than one-off financing initiatives and inspired more businesses to join the club. According to KII respondents, the CSR club played an instrumental role in starting the dialogue between the private sector and civil society. KII respondents credit the CSR Club with raising the private sector’s awareness about the work of CSOs as well as with expanding the definition of corporate social responsibility. Many noted that businesses began to understand that CSR is “not only about charity.” In addition, the club created networking opportunities for CSOs and enhanced understanding of businesses preferences, leading to some small-scale projects which were implemented collaboratively by businesses and CSOs. “Biggest change was in personal attitudes of representatives of business. We are in the beginning of dialogue. For many private companies it was completely new that funding of CSOs could come not only from donors; that NGOs can assist business in solving social and environmental problems. Most important result of work of CSR club is this.” [KII Respondent, Private Sector] “What worked the best was awareness raising. CSR club would get together and talk. They organized CSR awards. And this made a big impact. Because everyone wanted to get the award.” [KII Respondent, Private Sector] ACCESS support to CiDA and the CSR club was put on hold in program Year Five due to alleged fraud committed by the Executive Director. The suspension of the club activities negatively affected the outreach to private companies since 2019. Private sector engagement from ACCESS outside of the CSR club. The vast majority of ACCESS’ work in private sector engagement was through the CSR club. According to USAID’s private sector engagement policy, private sector engagement should advance beyond CSR to leverage private sector funds to increase the scale of CSOs’ impact, leverage private sector’s relationship with governments to advance CSOs’ policy interests, and to reduce any systemic constraints that prevent private sector from fully supporting CSOs. ACCESS staff indicated that private sector engagement outside the context of CSR was minimal, though ACCESS and CiDA leveraged relationships formed through the club to encourage direct business financing of CSO activities. Financing emerging from this effort was minimal, with a few $10,000 - $15,000 projects financed for purposes such as pandemic￾related charity events, design of a public garden, protection of cultural heritage, educational activities, sports events, environmental activities, and some crowdfunding initiatives. However, ACCESS considers CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 46 the “breaking of ice” between CSOs and private sector and the creation of a platform for their engagement to be a necessary condition to permit future engagement, including engagement aligned with USAID’s private sector engagement policy. Implications for future private sector engagement with civil society. In general, KII respondents across all sampled groups agree that the cooperation of private sector and civil society is important for both groups and should continue beyond ACCESS. However, there is little understanding among CSOs of how to reach out to the private sector or what types of projects businesses might fund. In addition, key informants from the private sector indicated that businesses who are open to collaboration with CSOs tend to have a set of predefined priorities that are usually developed in consultation with a public relations company or based on the sector in which the business is involved. There are usually no formalized procedures for CSOs to apply for opportunities for funding from the private sector. The cooperation and funding that have occurred to date have relied mostly on personal contacts to identify opportunities. As most businesses who are large enough to support CSOs are concentrated in Tbilisi, this reliance on personal contacts further hinders cooperation between the private sector and regionally based CSOs. “We tested cooperation with business and when we talk to business, unless you offer something that gives them benefit, they are not interested. And so far we could not find such a joint thing. We have to come up with a service/idea that benefits business.” [FGD Respondent, CCE FGD] “There are no big businesses in regions, only branches. And they can’t decide themselves without the support from the HQ.” [FGD Respondent, CCE FGD] Ultimately, there is no expectation that funding from the private sector can replace financial assistance from donor organizations in the foreseeable future. Beyond CSR, some KII respondents note that CSOs can be a valuable resource for the private sector with their knowledge, expertise, and access to international organizations and resources. In addition, civil society can be instrumental in supporting the businesses in interaction with government, especially around certain policies and legal framework when the interests of both parties coincide. Key informants assert that donor support will have a crucial ongoing role to this end—in educating businesses on working with civil society and in educating civil society on how to work with businesses. They assert this education should show how the collaboration between the two can be beneficial for both parties and society at large. While ACCESS – through CiDA - made initial steps in this direction, more is needed to solidify the results achieved so far. One possible candidate that key informants feel is well suited to carry this work forward is the United Nations (UN) Global Compact Network in Georgia. The UN Global Compact Network is the world’s largest corporate sustainability initiative. It unites businesses committed to promotion of a sustainable and inclusive global economy through integration of human rights, labor rights, environment, and anti￾corruption into their activities. CiDA currently serves as the Secretariat of the Network in Georgia. Some KII respondents anticipate the ground work laid by the CSR club to continue through the UN Global Compact as most see the need for further mediation and facilitation of private sector and civil society engagement. They especially note the need for matching CSR priorities with CSO and citizens interests – either through constantly updating the database developed by CiDA or through CSOs informing businesses about citizen priorities. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 47 “CSOs can provide information to businesses on what should be their priority. There is need for such information. No one is putting these problems together in an understandable way.” [KII Respondent, USAID IP] “There is a need for further mediation and facilitation of this process. This culture of cooperation is not sustainable yet. CIDA without ACCESS funding is very committed and they will contribute.” [KII Respondent, ACCESS] EQ5: How successfully did the project’s PMP measure the stated objectives of the project? Are there higher order, more outcome-oriented indicators that could have been employed? There is reasonable alignment between the indicators in the PMP and the concepts underlying ACCESS’ objectives and purpose. However, based on a review of project documentation and interviews with key stakeholders from ACCESS and USAID, the PMP did not adequately convey ACCESS’ performance. The lack of a cohesive theory of change supporting ACCESS’ results framework posed considerable challenges to the execution of a successful PMP. Alternative MEL approaches such as developmental evaluation and outcome harvesting may have enabled ACCESS to track an evolving theory of change and define indicators to serve as bellwethers of long-term progress toward the project’s objectives. ACCESS has four stated objectives: • Objective 1: Citizens more aware of and involved in CSO activities • Objective 2: CSO leadership and organizational capacity, as well as sustainability, improved across a range of organizations in and outside Tbilisi • Objective 3: CSOs effectively able to monitor and influence key government policies and processes • Objective 4: Local ownership of CCEs so that they continue to operate as politically neutral gathering spaces for public discourse Project documents imply that these objectives work together towards increasing civil society’s effectiveness as a force for demanding more transparent and accountable governance in Georgia. ACCESS has a well-defined results framework and Performance Monitoring Plan (PMP) elaborating which results are expected in each of these four objectives. Table 8 (see Annex D) summarizes ACCESS’ PMP, including comments on indicators which have been modified, added, or removed over the first six years of the project. According to ACCESS staff and USAID respondents, this PMP has measurement and conceptual shortcomings that, especially in project years one through five, limited the PMP’s success. The ET explains each of the PMP’s primary shortcomings below. MEASUREMENT SHORTCOMINGS Reliance on the USAID DRG Indicators survey. Three key indicators which were meant to add population-level information on changes in citizen engagement with civil society relied upon USAID’s DRG Indicator survey as a data source. While these indicators were measured for the first three years of ACCESS, USAID respondents report that competing demands for limited questions on the survey and poor reliability due to sampling issues dictated that USAID remove the questions required to CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 48 capture ACCESS’ indicators. These indicators were thus not reported in contract years 4 and 5, and have ultimately been removed from the PMP in contract year 6. Reliance on the USAID CSO Sustainability Index. The USAID CSO Sustainability Index, like the DRG Indicators survey, permitted ACCESS to monitor any changes in Georgian civil society writ large over the project’s lifetime. Indicators relevant to this index still appear in ACCESS’ PMP. However, USAID CSO Sustainability Index dimensions are scored based on the expert opinions of seven panelists on the state of the civil society sector in Georgia. They are not a direct measurement of these dimensions, per se. ACCESS staff members expressed doubt that their project’s activities could reasonably move the needle on such an indicator given the limited scale of their project. For example, they wonder if advances made in the advocacy capabilities of a few dozen CSOs would register any change in the way a panel of experts would score the advocacy capabilities of Georgian civil society as a whole. ACCESS thus does not consider these indicators as valid measurements of project performance. In any case, studies of previous USAID civil society projects have validated that the population of USAID program participants and the general population are often categorically different in many behaviors and attitudes with respect to civil society. 52 It is thus reasonable to assert that measures linked to the overall population may not reflect outcomes with the participant population. Quality concerns with ACCESS monitoring data for Objective 1. ACCESS’s, grantees’, and CCEs’ records of citizens’ participation in CSO activities constitute the PMP’s main data source for citizen engagement. In PMP reports and in interviews with ACCESS staff, there are two serious problems with the quality of these records. First, they are considered to be under-estimates. ACCESS staff feel that the project does not have the staff or resources to adequately collect the details of each citizen who participates in an activity or serves as a volunteer. This situation has been exacerbated during the COVID-19 pandemic, when most events have occurred outdoors. Second, ACCESS staff express that quantitative measures of participation do not capture qualitative differences in the nature of participation. Stakeholders from ACCESS and USAID agree that ACCESS has adapted over time through experimentation to maximize strategies that lead to more meaningful engagement between CSOs and citizens. For example, ACCESS shifted increasingly toward regional bus tours in more recent years given their ability to bring CSOs closer to issues that were more meaningful to citizens. Counting attendance for a bus tour the same as attendance for a generic informational event obscures meaningful differences in the quality of these interactions between CSOs and citizens. CONCEPTUAL SHORTCOMINGS “Although we do not have formal theory of change, we had one in our mind.” [KII, ACCESS] Lack of a cohesive theory of change. Although ACCESS has a well-documented results framework, there is no record that it has ever had a cohesive theory of change linking project outputs and outcomes for individual CSOs and citizens to the expected impact on the civil society sector writ large. Given ACCESS’ established difficulty with population and sector-level indicators, it is important for its PMP to demonstrate progress on near-term outcomes that are predictive of future sector-wide changes. Without a clear theory of change, it is unclear how, for example, changes in individual OACA scores or increased crowdfunding initiatives by CSOs will lead to improved citizen engagement and policy changes. 52 See Pact Ukraine’s 2018 “Survey of USAID/Ukraine ENGAGE participants: differences and similarities with population-based poll” CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 49 The absence of a declared theory of change may reflect ACCESS’ experimentation over time. ACCESS staff and counterparts at USAID express that the project’s focus shifted over time as it experimented with different approaches to citizen engagement and influencing government policies. ACCESS ultimately settled on the approaches that seemed most successful. As such, any theory of change codified in early project documents would not have applied to the project’s main implementation strategies after six years. “ACCESS evolved so much, the quantitative indicators don’t really capture the novelty of work…These are traditional indicators. I don’t think they capture the variety of ACCESS’ approaches.” [KII, USAID] Absence of ACCESS’ primary accomplishments. There are very few measures in ACCESS’ PMP aligned with what many stakeholders believe to be ACCESS’ primary success: engaging atypical CSOs such as informal organizations and individual activists and promoting the use of novel methods for citizen engagement, such as co-creation of CSO activities (see Flexible and demand driven approach to grants in response to EQ1). Key stakeholders within ACCESS feel that these accomplishments cannot be captured by quantitative indicators, because they represent qualitative changes in the way that civil society works in Georgia. Updates made to the PMP in Year 6 attempt to capture some of these changes, such as the increased use of co-creation and crowdfunding for CSO activities. However, ACCESS and USAID staff feel that these indicators still do not reflect what is effectively a paradigm shift in Georgian civil society, where issues that CSOs engage with can be determined by citizens and not donors, and where informal organizations and individual activists who are closest to these issues can leverage funding to better monitor and influence government policies. Even if this paradigm shift does not lead to changes in sector-wide measures of CSOs’ engagement with citizens and ability to influence government policy during ACCESS’ lifetime, many stakeholders consider that it creates a critical opportunity for such changes in the long term. ALTERNATIVE INDICATORS TO MEASURE ACCESS’ PERFORMANCE Respondents from across the Georgian civil society landscape, including CSOs, ACCESS, USAID, and other donors, acknowledge that individual projects such as ACCESS are limited in their capacity to make transformative change in the civil society sector on their own. As such, many more closely monitor output indicators in PMPs (or analogous documents) than higher-order indicators such as those from the CSO Sustainability Index or the DRG Indicators survey. “We are modest from the beginning when we assess the projects realizing it won’t change much. We are contributing some tiny drops to the ocean…. We look at outputs – it’s easy. With impact it takes time and more money than we can offer.” [KII, non-USG donor] When substantial impact on higher-order indicators is not expected in the project timeframe, a PMP must instead monitor outputs and short-term or intermediate outcomes that are reasonably predictive of long-term project objectives according to a robust theory of change (i.e. “bellwether” indicators). Based on published literature, there are three possibilities for measures that could serve as bellwether indicators for a project such as ACCESS. Based on Nancy Bermeo’s framework for civil society development, two conditions that are particularly conducive to civil society’s development are a climate CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 50 of toleration by political elites and well-defined linkages with other domains of the polity.53 Even if these conditions are not the end goal of a civil society program, they might be expected to change in the near term and indicate a stronger civil society by more direct measures in the longer term. A third possible set of bellwether indicators are self-expression values as measured in the World Values Survey.54 Literature suggests that self-expression values give high priority to “rising demands for participation in decision making in economic and political life.”55 The World Values Survey is already conducted in Georgia, so these indicators are more readily available than measures of elite toleration and linkages with other domains of the polity. For their part, ACCESS staff profess to have implemented a version of this strategy, albeit without ever formally recording their theory of change. As indicated in the previous section, the theory of change morphed over the project lifetime as ACCESS experimented with different approaches for engaging citizens. The quote below shows how ACCESS staff were more concerned with building potential for future results than with demonstrating results in the near term. “We said we would evaluate the success of a grant project not by achieving a result but by the support you mobilized and the people you engaged on your issues. Not just one time engagement, more consistent engagement... Not solving the problem is ok because at least you have a base to keep working on the issue.” [KII, ACCESS] After periods of intense activity on a given work stream, ACCESS staff would reportedly gather to debrief while impressions were fresh and reflect on their strategy. Regrettably, these sessions were not documented. Documenting these sessions and analyzing the documentation to iteratively update a theory of change would have provided an opportunity to align the PMP with the project’s most recent thinking on how to increase citizen engagement and monitor and influence key government policies. Rather than pursue a rigid MEL system that assumes a fixed theory of change, ACCESS might have pursued established MEL techniques that support evolving theories of change and emerging outcomes. Two helpful possibilities would have been developmental evaluation and outcome harvesting. Developmental evaluation is expressly meant to support innovation and adaptation of projects in dynamic environments by embedding evaluation within project implementation. USAID has previously used developmental evaluation for the express purpose of “rebooting theories of change.”56 Embedding a developmental evaluation within ACCESS would have made evolution in the theory of change explicit and sought opportunities to elucidate causal links between project outputs and the project purpose. Outcome harvesting, in which some ACCESS staff expressed interest, “works backwards to identify, describe and verify contribution to results, both expected and unexpected.”57 Coupling outcome harvesting with experimental shifts in the project’s theory of change could have clarified what outcomes were emerging and assess the prospects of these leading to project objectives. Outcome harvesting 53 Kubik, J. (2005). “How to Study Civil Society: The State of the Art and What to Do Next.” East European Politics and Societies, 19(1) 105-120. 54 https://www.worldvaluessurvey.org/WVSNewsShow.jsp?ID=427 55 Ronald Inglehart; Wayne E. Baker; Modernization, Cultural Change, and the Persistence of Traditional Values, ASR 2000, Vol 65,1 pp. 19-51 56 https://usaidlearninglab.org/library/using-developmental-evaluation-reboot-theories-change 57 https://usaidlearninglab.org/library/outcome-harvesting-complexity-aware-monitoring-approach CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 51 could also help identify a more complete set of bellwether indicators that portend progress toward project objectives in the long term. CONCLUSIONS EQ1: Effectiveness of ACCESS in supporting CSO’s to increase citizen engagement; did problem analysis and activity design through co-creation address citizen identified issues? ACCESS was effective in supporting CSOs to increase citizen engagement in the country. This can be attributed to the following factors. First, ACCESS evolved its definition of citizen engagement, shifting from advocacy and citizen awareness at the start of the project to enabling citizens to actively participate on issues that focused on citizen wellbeing, especially in communities outside Tbilisi. This included issues such as urban development, air quality, quality of life and cultural heritage. Second, ACCESS encouraged its grantees to experiment with new approaches for engaging citizens -- involving them in planning activities, identifying priorities, recruiting them as volunteers, involving them in public pressure campaigns, signing petitions, and conducting trainings and seminars. Third, ACCESS was able to focus on citizen’s priorities by having a flexible and demand driven approach to its community outreach and rapid response grants, and by not specifying pre-defined priorities in its calls for proposals. From 2017 ACCESS supported grass roots civic action, including working with informal and newly registered CSOs on projects grounded in citizen priorities. Finally, ACCESS encouraged regional activists to engage with a diverse group of individuals. The process of co-creation with the Social Labs methodology helped grant recipients develop and implement more innovative, well-thought-through, and engaging projects. According to ACCESS staff Social Labs helped to elevate the quality of proposals by better embedding citizen interests. Factors that encouraged citizen engagement include a more active youth population, the relevance of issues focused on by CSOs, and increased civic engagement by ethnic communities. Key obstacles to citizen engagement are that formal membership and volunteer engagement in CSOs remains low, weak community mobilization by CSOs, lack of civic education, citizen’s belief that they cannot affect change, low trust in local authorities and political parties by citizens, lack of capacity among CSOs to successfully outreach citizens, and a shrinking civic space in the country. The lack of capacity among CSOs was confirmed by the organizational capacity assessment tool. It showed that several ACCESS grantees struggled to improve their capacity in citizen engagement despite the project, and rarely prioritized citizen engagement as an area that needs improvement. EQ2: Effectiveness of ACCESS in supporting CSOs in monitoring and influencing key government policies and processes? What policy areas lacked support? Where were most notable results achieved? ACCESS-supported projects were mostly successful in influencing public opinion on policy issues and spotlighting important issues/problems in the country. However, examples of actual change in policy are more limited. Countering disinformation and promotion of Euro Atlantic values is an area in which ACCESS partners achieved the biggest result in terms of introducing a problem into public discussions, making this issue constantly discussed, and influencing public opinion and policy as a result. ACCESS’ efforts culminated in the GoG including the problem of disinformation into official communication strategy documents. Work on road safety also resulted in amendments to national legislation to address the problem of high fatality as a result of car accidents in Georgia. USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 52 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 ACCESS CSOs were successful in changing environmental policy at the regional level, dealing with air pollution in Rustavi and Poti, and helping activists in their fight against the construction of the Namakhvani hydropower station in Western Georgia. ACCESS partner’s work in the cultural heritage and urban development area has had mixed success. In Tbilisi, CSOs have been successful in having urban development issues discussed in the media and public sphere, but not yet changed policy. In Batumi, the struggle for preservation of urban heritage is impeded by lack of cooperation from local and sub-national governments as well as a weak level of mobilization of local society around this problem. Factors that helped in influencing government policy included attitudes of government officials and bureaucracy working in a particular area towards civil society, the extent of cooperation, the existence of more or less stable ties between representatives of civil society and bureaucracy or politicians, citizen mobilization around an issue, and level of convergence on the issue/problem between representatives of CSOs and government. However, cooperation between civil society and government was often complicated by long-term distrust between the two and by interests of other actors such as the private sector. CSOs have not focused on socio-economic issues which have remained citizens’ priorities in Georgia over the last decade. Work on these issues are hampered by a shortage of expertise among CSOs on these issues, as well as the quality of policy research in Georgia. EQ3: What is the success of ACCESS in increasing local ownership of CCEs? What is the potential of a network of centers for civic engagement (NCCE) to increase demand for CCE services in each location, get funding and be sustainable? Under ACCESS, CCEs have continued their previous role as neutral spaces for public discourse and expanded their role and visibility as a catalyst and enabler of civic engagement outside of Tbilisi. However, despite advances in providing fee-based services and reduced operating expenses, they will not be able to persist following ACCESS without core donor funding. The COVID-19 pandemic halted the main income-generating service the CCEs provided, namely the space for different events. CCEs began to offer new services to affected communities, such as assistance with registration for state aid and dissemination of information about COVID-19. However, while CCEs have expanded their base, they have experienced a severe revenue shortfall during this period. The NCCE has made critical contributions to the CCEs’ ability to institute fee-based services, negotiate as a large network for grant funding and reduce operating expenses, and collaborate and learn from one another. However, some stakeholders find this Tbilisi-based expense high and not sustainable outside of donor support. EQ4: How successful has the partner been in facilitating private sector engagement, including to secure private sector funding and support? ACCESS made important progress in creating a platform to facilitate more direct engagement and collaboration between private sector and civil society. This was considered by stakeholders to be a necessary first step for broader engagement in the future. ACCESS primarily encouraged private sector engagement with civil society through the promotion of CSR in Georgia. The CSR club had notable success in strengthening linkages between the two sectors, mainly through organization of CSO fairs and CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 53 compiling a database of the CSR priorities of the CSR club members to match CSO and citizen priorities and map opportunities for partnerships. Georgia’s low level of economic development is one of the factors affecting weak engagement between CSOs and the private sector. Businesses are under-developed and very few companies have dedicated CSR budgets. Additionally mutual mistrust is the main obstacle for cooperation. There is still very limited direct funding of CSOs by private sector in Georgia, and even less engagement for other purposes, like influencing government policy for mutual gain. Considerable room for future improvement in private sector engagement remains, particularly in engagement outside of Tbilisi and in creating formal, established channels for CSOs to apply for funding from private companies EQ5: How successfully did the project’s PMP measure the stated objectives of the project? Are there higher order, more outcome-oriented indicators that could have been employed? There is reasonable alignment between the indicators in the PMP and the concepts underlying ACCESS’ objectives and purpose. However, the PMP did not adequately convey ACCESS’ performance. Measurement shortcomings such as reliance on the USAID DRG Indicators Survey, which due to reliability concerns and requirements for prioritization removed three key ACCESS indicators led to an inability to report results for these indicators in contract years 4 and 5. Measuring project results based on USAID’s CSO Sustainability Index was also difficult, given the scale of ACCESS and challenges in moving the needle on the state of civil society sector for the whole country. Finally there are concerns that citizen’s participation in CSO activities have been under estimated. Conceptual shortcomings in the PMP such as the lack of a cohesive theory of change, and indicators that capture ACCESS’ accomplishments make it difficult to measure and capture ACCESS’ results. ACCESS acknowledges that they failed to adequately record and capture debriefing sessions on their strategy, and to align the PMP with the project’s most recent thinking on how to increase citizen engagement and monitor and influence key government policies. RECOMMENDATIONS The ET recommends the following for USAID/Georgia’s consideration as it prepares to implement the Georgia Civil Society Engagement Activity: EQ1: Effectiveness of ACCESS in supporting CSO’s to increase citizen engagement; did problem analysis and activity design through co-creation address citizen identified issues? Maintain flexible demand-driven grant making, which is accompanied by deep engagement with grantees to help them build capacity. • Support further civic mobilization and enlarge the number of actors in civil society. • Ensure that the program offers a variety of tailored assistance depending on the level of development of a civic initiative. For informal groups offer technical assistance, mentoring, and networking to encourage evolution into more effective groups. • Provide targeted TA for grantees to develop an understanding of how citizen engagement links with accountability and legitimacy of CSOs and expand their skills in citizen engagement. TA could CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 54 include increasing capacity for volunteer management, building individual donor relations, developing strategies for growing formal membership, and assistance with inclusive forms of governance. • Ensure measurement of citizen engagement is measured not only in numbers but also in quality/depth of engagement. One potential model would be for USAID to introduce an Engagement Account for grantees to measure and differentiate engagement across participants, volunteers, donors, members, and trustees. Such an account could be submitted annually and include metrics such as social media followers, event participants, donations from citizens, volunteer hours per year, number of members, and number of board meetings. • Ensure the Civil Society Engagement Activity’s “Ecosystem Approach” covers all the components of the ecosystem not just with funding, but in knowledge sharing and development of political participatory culture. Focus on strategic communication campaigns and citizen education to counter disillusionment and nihilism in the society. • Promote existing tools for citizen engagement among active citizens, COBs, regional CSOs • Where possible in USAID/Georgia’s portfolio of democracy, human rights, and governance programming, support the expansion of civic education programs to cover more schools and reach out to students. Accompany knowledge building by offering practical opportunities to implement community projects. • Focus on knowledge sharing and learning: peer-to-peer exchanges about lessons learned from citizen engagement, developing volunteering, promoting charitable giving, campaigning with the local and national governments. • Showcasing positive examples empowers citizen’s confidence to become advocates of positive change. The case of various engagement methods (Social Labs, focus groups) proves effective to overcome this obstacle for engagement. Further develop institutional capacity of existing community-based organizations and community foundations. • Consider connecting these groups across Eastern Europe to learn from peers. Ukrainian foundations would be an especially strong resource, given similar socioeconomic context and strong tradition of community foundations with many currently operating successfully. • Where possible and if genuine interest is expressed, build CCE’s capacity for community mobilization. Enlarge their scope by serving the needs of local residents more effectively. Train community organizers. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 55 • Support work to defend and expand space for civic action in Georgia. Ensure a wide coalition of CSOs conducts advocacy and monitoring to defend democratic norms for citizen participation. Further promote the idea of co-creation and enable Georgian CSOs to run the process with the citizens and key stakeholders. • Georgian CSOs should engage citizens and key stakeholders in externally facilitated events and gradually preparing organizations to run the process on their own. • ACCESS organizational development partners like CTC could train and develop learning toolkits in Georgian. • Further develop proficiency and confidence in using new methods of co-creation. • Promote voluntary culture and charitable giving (crowdfunding): deliver public awareness campaigns, improve regulatory framework, offer civic education. EQ2: Effectiveness of ACCESS in supporting CSOs in monitoring and influencing key government policies and processes? What policy areas lacked support? Where were most notable results achieved? Enable CSOs to advocate for and work on social, health, environmental, and economic issues. • Build the capacity of CSOs to meet the demand from citizens to address social, health, environmental, and economic issues in the Civil Society Engagement project and award grant funding to emerging informal actors who intend to work on these issues. • Promote the formation of nationwide coalitions of local activists and CSOs to advocate for policy issues related to urban development and the environment. These coalitions should aim to ensure citizen priorities are reflected in policy equally with private sector interests. Further build the capacity of CSOs to conduct high quality policy research. • Build capacity to use more rigorous and novel methodological approaches, including experiments, cost-benefit analysis, participatory research methodologies • Foster ongoing thematic cooperation and exchange of information between civil society groups and relevant government offices Support enabling factors for success in policy advocacy, including trust, mobilization of public support, and communication capacity. • Include concrete steps in the Civil Society Engagement Activity to build trust between CSOs and government. Increased trust would yield many benefits, including an improved ability to counter disinformation. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 56 • Assessment criteria for future grant proposals seeking policy change should promote the inclusion of mobilizing public support alongside policy research/advocacy activities. • Build the communication capacity of CSO to showcase their work and results in the country. EQ3: Success of ACCESS in local ownership of CCEs so that they continue to operate as politically neutral gathering spaces for public discourse? What is the potential of a network of centers for civic engagement (NCCE) to increase demand for CCE services in each location, get funding and be sustainable? Identify, and continue to fund, CCEs who are providing essential services to communities, while exploring opportunities for self￾financing. • Conduct a community center landscape analysis to assess the geographic and services overlap of MoJ community centers and CCEs. This analysis should identify which CCEs, if any, have redundant services in their community with other service providers. • Given their critical role in civil society outside Tbilisi, continue to fund all CCEs which are not identified as being redundant based on the aforementioned landscape analysis during the lifetime of the upcoming Civil Society Engagement Activity. • Support the decentralization of the NCCE network and explore opportunities for self-financing over the life of the Civil Society Engagement Activity. Continue funding the NCCE with a mandate to further increase the proportion of revenue generated independently of core donor funding. EQ4: How successful has the partner been in facilitating private sector engagement, including to secure private sector funding and support? Track mutual interests between civil society and the private sector, promote philanthropy, and encourage continued engagement • Include scope in the upcoming Civil Society Engagement Activity to keep the database developed by CiDA matching private sector and CSO interests up to date. Advance this analysis by identifying opportunities for CSOs and private sector to lobby government together for mutual gain. • Pursue improvements in the enabling environment to promote philanthropy, especially to promote giving to regionally-based CSOs. Advocate with large Tbilisi-based businesses to permit regional branches to have their own CSR budget. • Continue to support and expand a platform for CSOs and the private sector to interact on a regular basis. Try to engage top managers and decision-makers in those meetings. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 57 EQ5: How successfully did the project’s PMP measure the stated objectives of the project? Are there higher order, more outcome-oriented indicators that could have been employed? Use leading indicators and other innovative MEL approaches such as developmental evaluation to support improved performance measurement. • For the upcoming Civil Society Engagement Activity, commission an evidence review with the explicit purpose of identifying feasible indicators to include in PMP as short-term intermediate outcomes of intended long-term impacts in civil society. Also leverage any learning done by USAID’s DRG Center on the subject. • Ensure that the upcoming Civil Society Engagement Activity includes sufficient resources for the implementing partner to collect its own set of robust monitoring data. • For similar programs to ACCESS, where USAID intends to promote experimentation, embed developmental evaluation in the program to document modifications to the theory of change and iteratively update the project’s MEL plan and PMP. If a developmental evaluation is not possible, be sure to document the outcome of pause and reflect sessions as relevant to improving project MEL procedures. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 58 ANNEX A: EVALUATION STATEMENT OF WORK USAID DRG-LER II Tasking Request #40 Date of Request: 08 September 2020 Type of Task: Performance Evaluation Statement of Work (SOW): The purpose of the evaluation is to assess the effectiveness of the Advancing CSO Capacities and Engaging Society for Sustainability Project (ACCESS) in enabling a more constituent-connected cadre of civil society organizations (CSOs); supporting CSOs in monitoring and influencing key government policies and processes; building local ownership of the Centers for Civic Engagement; and facilitating private sector engagement and other inter-sectoral cooperation, since its inception in October 2014 to the date of evaluation. Evaluation questions are as follow (and will be refined in consultation with the Evaluation Team): EQ1: How effective has the partner been in enabling a more constituent-connected cadre of CSOs? How did problem analysis and activity design through co-creation contribute to supporting citizen driven civic initiatives? EQ2: How effective has the partner been in supporting CSOs in monitoring and influencing key government policies and processes? What policy areas do key informants mention as lacking in support? How relevant are these thematic areas/policies to citizens’ priorities and key development issues? What were the areas where ACCESS-supported CSOs achieved most notable results? Were there policy areas where ACCESS should have focused more resources to achieve better results to date, based on key informants and public surveys? EQ3: How successful has ACCESS been in working on local ownership of the Centers for Civic Engagement (CCEs) so that they continue to operate as politically neutral gathering spaces for public discourse? What is the potential of a network of Centers for Civic Engagement (NCCEs) to diversify services and increase demand for CCE services in each location, strengthen financial diversification, expand services to the private sector, sustain results to date, and continue developing on its own? EQ4: How successful has the partner been in facilitating private sector engagement (exploring areas of mutual interest and CSR initiatives) and other inter-sectoral cooperation? EQ5: How successfully did the project’s PMP measure the stated objectives of the project? Are there higher order, more outcome-oriented indicators that could have been employed? Deliverable(s): Each of the deliverables below are explained in detail in the attached SOW: CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 59 4. Evaluation Work Plan 5. Evaluation Design 6. In-briefing 7. Mid-term Briefing and Interim Meetings 8. Final Exit Briefing 9. DVC for Recommendations Development 10. Final Presentation (if requested by USAID) 11. Draft Evaluation Report 12. Final Evaluation Report 13. Submission of Dataset(s) to the Development Data Library 14. Submission of Final Evaluation Report to the Development Experience Clearinghouse Dates of Performance and Timeline: April 2020, with final report to be submitted in June 2021. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 60 ANNEX B: DATA COLLECTION INSTRUMENT ONLINE SURVEY: ACCESS GRANTEES AND TECHNICAL ASSISTANCE RECIPIENTS A. Consent and Preloads Question Response Options Type and Constraints A1. NORC, a research center associated with the University of Chicago, has been contracted by USAID to conduct an independent evaluation of the Advancing CSO Capacities and Engaging Society for Sustainability (ACCESS) project in Georgia. The primary goal of this study is to inform future USAID programming that will work with local civil society organizations in Georgia. This survey will ask you to describe basic characteristics of your civil society organization and the grant your organization was awarded through the ACCESS program. You will also be invited to give recommendations for similar programs in the future. It will require approximately 15 minutes to complete. Your participation in this study is voluntary—there are no consequences for choosing not to participate. However, your feedback will be very useful and help design future USAID programs in Georgia. Your response will be kept confidential and anonymous. The information you provide will be used in summary form only after aggregation with information from other respondents. None of your responses will be linked with your identity. If you have questions please e-mail Ritu Nayyar-Stone, the Team Leader for this evaluation at nayyarstone-ritu@norc.org or Georgian Evaluation Specialist Tamara Sartania at tsartania@gmail.com. Please select “Yes” to indicate your consent to participate in this survey, or select “No” to decline participation. o Yes, I consent to participate in this survey o No, I do not consent to participate in this survey (→F6) Select one A2. Before continuing, please confirm which type(s) of grant(s) your organization received from ACCESS (select all that apply): □ Citizen Outreach Grant □ Public Policy and Accountability Grant □ Rapid Response Grant □ Technical Assistance Select multiple CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 61 Question Response Options Type and Constraints A3. (Repeat for each type of grant selected above): How many of these types of grants did you receive? _______ Integer (maximum 9) A4. (Repeat for each grant received): Please enter the starting and ending month and year for this grant. If grant is ongoing, please enter planned ending date. Start date: MM/YYYY End date: MM/YYYY Date (Month/Year) B. Summary Characteristics Question Response Options (Note: Order of options will be randomized for most questions) Type and Constraints B1. At what geographic level does your CSO work (select all that apply) □ National □ Regional □ Sub-regional (e.g. municipal) Select multiple B2. Is your CSO formally registered as a non￾commercial legal entity with the Georgian government? o Yes o No (→B9) Select one B3. In which year was your CSO formally registered? ____ (→B5) o Don’t know Integer Constrained 1900 – 2021, including don’t know option B4. Please take an approximate guess regarding which year your CSO was formally registered _______ Integer Constrained 1900 – 2021 B5. Is your CSO separately registered as a charity organization? o Yes o No (→B8) o I don’t know (→B8) Select one B6. In which year did your CSO separately register as a charity organization? ____ (→B8) o Don’t know Integer Constrained 1900 – 2021, including don’t know option B7. Please take an approximate guess regarding which year your CSO registered as a charity _______ Integer Constrained 1900 – 2021 B8. Did your CSO operate informally before it was formally registered as a non-commercial legal entity with the Georgian government? o Yes o No (→B10) o I don’t know (→B10) Select one B9. In which year did your CSO begin operation? _______ Integer Constrained 1900 – 2021 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 62 Question Response Options (Note: Order of options will be randomized for most questions) Type and Constraints B10. How many of the following does your organization have: a. Active members b. Volunteers o Less than 10 o 10-19 o 20-49 o 50-99 o 100-299 o 300 or more Select one (repeat for a and b) B11. Which of the following best describes the profile of your organization? (Select all that apply) □ CCE □ Community-based organization □ Professional Association / Professional Union □ Think tank/Analytical center □ Service provision CSO □ Charity Organization / Foundation □ Oversight / Watchdog organization □ Media organization □ Advocacy CSO □ University □ Private company □ Network □ Independent community leader(s) or activist(s) □ Other (please describe:_____________) Select multiple B12. Please indicate the extent to which you agree with the following statements regarding your CSO’s organizational capacity: a. My CSO has a clearly defined Mission b. My CSO serves a clearly identified constituency that is aligned with its Mission c. My CSO incorporates time-bound strategic planning in its decision-making d. My CSO has a clearly defined management structure e. My CSO engages professional services from accountants, IT managers, and/or lawyers f. My CSO uses modern, basic office equipment (computers and software, internet access, telephone, printer/scanner, etc.) o Strongly Agree o Somewhat Agree o Neither Agree nor Disagree o Somewhat Disagree o Strongly Disagree Select one for each question, ideally present in matrix form CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 63 Question Response Options (Note: Order of options will be randomized for most questions) Type and Constraints B13. Please indicate the extent to which you agree with the following statements regarding your CSO’s financial viability: a. My CSO raises a significant percentage of its funding (at least 20%) from within its municipality/town b. My CSO raises a significant percentage of its funding (at least 20%) from within its region c. My CSO raises a significant percentage of its funding (at least 20%) from within Georgia d. My CSO raises a significant percentage of its funding (at least 20%) from foreign donors e. My CSO raises a significant percentage of its funding (at least 20%) from the private sector f. My CSO earns revenue from services, products, rent, or membership fees For a-f, use: o Yes o No Select one for each question, ideally present in matrix form B13. Please indicate the extent to which you agree with the following statements regarding your CSO’s financial viability: g. My CSO can draw upon a core of monetary support from my community and constituents (i.e. crowdfunding) h. My CSO can draw upon a core of non￾monetary support from my community and constituents i. My CSO has diverse sources of funding j. My CSO has enough resources to remain viable for the next year k. My CSO has enough resources to remain viable for the next 5 years l. My CSO has processes, procedures, dedicated personnel, and tools in place to manage financial resources and operations B14. My CSO operates in a transparent manner, including independent financial audits and the publication of annual reports with financial statements For g-l, use: o Strongly Agree o Somewhat Agree o Neither Agree nor Disagree o Somewhat Disagree o Strongly Disagree Select one for each question, ideally present in matrix form CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 64 C. Citizen Engagement and Service Provision Question Response Options (note, order of options will be randomized for most questions) Type and Constraints C1. Which of the following target groups does your organization work with? (Select all that apply) □ Youth aged 15 to 25 years □ Children aged up to 14 □ Elderly/retired □ Women □ LGBTI (Lesbian, gay bisexual, transgender, and intersex) people □ Internally displaced people (IDPs) □ People with disabilities □ Non-dominant ethnic or religious groups □ Veterans □ Private sector □ Local communities/ local authorities □ Media □ NGO / civic activists □ Sportsmen □ Voters □ Prisoners □ Other (please describe:____________) Select multiple C2. Which of the following ways does your organization engage citizens in its work? (select all that apply) □ Identifying priorities □ Planning activities □ Signatories of petitions □ Participants in trainings/seminars □ Recruit as volunteers □ Flash mobs □ Public pressure campaigns □ Public hearings □ Funders □ Recruit as members □ Invite to join CSO Board □ Other, specify:___________ Select multiple CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 65 Question Response Options (note, order of options will be randomized for most questions) Type and Constraints C3. What are the main challenges your organization faces in engaging citizens? (Please select all that apply). □ Our organization has limited financial resources □ Our staff have insufficient training/knowledge □ Our staff is fearful of retaliation against activists □ Citizens lack trust in CSOs □ Citizens believe that CSOs are politically motivated □ Citizens have a low level of civic education about their rights □ Citizens have limited knowledge of civic participation □ Citizens do not understand the importance of issues that our CSO addresses □ Citizens fear retaliation against activists □ Citizens lack of attention to public affairs □ Citizens lack of trust in government □ Citizens believe that reform efforts will fail □ Citizens believe they cannot influence change □ Citizens have different priorities due to a decline in their quality of life □ Other (please specify:_______________) C4. What methods did your CSO use to ensure your ACCESS-funded grant reflected the priorities of your constituents? (select all that apply) □ Included citizens in strategic planning □ Interviews with key constituents □ Focus group discussions with citizens □ Used knowledge and experience from previous projects □ Baseline survey of target constituents □ Researched available public opinion data □ Researched other academic literature □ Social Lab, or similar participatory project design approach □ Participated in bus tours □ Other, specify:________________ CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 66 Question Response Options (note, order of options will be randomized for most questions) Type and Constraints C5. Were the methods you used to ensure your grant reflected the priorities of your constituents new to your CSO, or had you employed these methods before? o All were new (→C9) o Some were new, some we had used before o We had used all these methods before (→C9) Select one C6. Please indicate which of the methods you used were new to your CSO (select all that apply) □ Included citizens in strategic planning □ Interviews with key constituents □ Focus group discussions with citizens □ Used knowledge and experience from previous projects □ Baseline survey of target constituents □ Researched available public opinion data □ Researched other academic literature □ Social Lab, or similar participatory project design approach □ Participated in bus tours □ Other (previously specified) Select multiple, ideally these would populate from previously selected options C7. To what extent do you agree with the following statement: the new method(s) we pursued during project design for our ACCESS grant improved our ability to address issues that are prioritized by our constituents? o Strongly Agree o Somewhat Agree o Somewhat Disagree (→C9) o Strongly Disagree (→C9) Select one CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 67 Question Response Options (note, order of options will be randomized for most questions) Type and Constraints C8. Which challenges do you think these new methods helped your organization overcome in engaging citizens? (Please select all that apply). □ Our organization has limited financial resources □ Our staff have insufficient training/knowledge □ Our staff is fearful of retaliation against activists □ Citizens lack trust in CSOs □ Citizens believe that CSOs are politically motivated □ Citizens have a low level of civic education about their rights □ Citizens have limited knowledge of civic participation □ Citizens do not understand the importance of issues that our CSO addresses □ Citizens fear retaliation against activists □ Citizens lack of attention to public affairs □ Citizens lack of trust in government □ Citizens believe that reform efforts will fail □ Citizens believe they cannot influence change □ Citizens have different priorities due to a decline in their quality of life □ Other (please specify:_______________) Select multiple C9. Do you feel that your grant(s) achieved notable results in terms of monitoring and/or influencing government policies and processes? o Yes o No (→C11) Select one CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 68 Question Response Options (note, order of options will be randomized for most questions) Type and Constraints C10. Please select which responses best describe your grant(s)’ most notable results in terms of monitoring and/or influencing government policies and processes? (select all that apply) □ Increased broad media attention to the monitored issue □ Increased public support to the monitored issue □ Introduced policy regulations and changes in relevant law due to constructive dialogue and engagement with the government/policy-makers □ Research and advocacy impacted state policy or initiated drafting law to address issues □ Preserved cultural heritage and historic monuments □ Countered disinformation □ Research help increased public awareness of social media disinformation □ Other, specify:___________ Select multiple C11. Please indicate which responses best describe what obstacles your grant faced in achieving notable results in terms of monitoring and/or influencing government policies and processes (select all that apply) □ Government and policy-makers were not receptive □ Insufficient support from the public □ Insufficient funding or scale □ Issues with grant implementation □ Issues with grant strategy □ Other, specify:___________ Select multiple C12. Do you feel that your grant(s) achieved notable results in terms of addressing issues prioritized by your constituents? o Yes o No (→C14) Select one CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 69 Question Response Options (note, order of options will be randomized for most questions) Type and Constraints C13. Please select which responses best describe your grant(s)’ most notable results in terms of addressing issues prioritized by your constituents? □ Increased broad media attention to an issue important to my constituents □ Advocated for policy regulations and changes in relevant law on an issue important to my constituents □ Achieved changes in policy regulations and relevant law on an issue important to my constituents □ Raised public awareness of an issue important to my constituents □ Informed my constituents of their rights regarding an issue that is important to them □ Empowered disadvantaged ethnic or religious groups that represent my constituents □ Other, specify:___________ Select multiple C14. Please select which responses best describe what obstacles your grant faced in achieving notable results in terms of addressing issues prioritized by your constituents? □ The grants priorities were not aligned with citizens’ priorities □ Government and policymakers were not receptive □ Insufficient support from the public □ Insufficient funding or scale □ Issues with grant implementation □ Issues with grant strategy □ Other, specify:_____________ Select multiple D. Underserved Priority Issues Question Response Options Type and Constraints D1. Are there critical issues in the community where your CSO operates that you think are underserved by civil society organizations? o Yes o No, all important issues are covered by CSOs (→E1) Select one D2. Please list any issues you feel are underserved, separating different issues using a semi-colon (“;”) ________________________ Open text CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 70 E. Private Sector Engagement Question Response Options Type and Constraints E1. To what extent do you agree with the following statement: “Companies and other private sector actors are valuable and trustworthy partners to CSOs like mine.” o Strongly Agree o Somewhat Agree o Neither Agree nor Disagree o Somewhat Disagree o Strongly Disagree Select one E2. Has your CSO ever received funding from a private company or other private sector organization? o Yes (→E5) o No, we sought funding and were unsuccessful (→E5) o No, we have not sought funding from the private sector Select one E3. Would your CSO ever consider accepting funding from a private company or other private sector organization? o Yes (→E5) o Maybe (→E5) o No Select one E4. Please explain why your grant would not consider accepting funding from a private company or other private sector organization _________________________ Open text E5. Has your CSO ever provided pro bono services to a private company or other private sector organization? o Yes o No o I don’t know Select one E6. Would your CSO ever considered providing pro bono services to a private company or other private sector organization? o Yes o Maybe o No Select one E7. What recommendations would you give to USAID in terms of encouraging private sector engagement with civil society in Georgia? _________________________ Open text (not required) CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 71 F. Conclusion Question Response Options Type and Constraints F1. What do you believe are the main challenges facing civil society in Georgia in the next 5 years? (select all that apply) □ Financial viability □ Polarization of society □ Lack of pluralism □ Lack of cooperation and solidarity among CSOs/competition □ Lack of CSO capacity in key issues of citizen concern □ Creation of government-organized NGOs (GONGOs) by state and private sector □ Disinformation □ Pressure from government/ruling party □ Decreasing public trust or support □ Other, specify:___________ Select multiple F2. Thank you for participating in this survey. May a member of the research team call you to clarify any responses that are unclear? o Yes, you may call me to clarify any response on this survey o No, you may not call me to clarify any responses (→F4) Select one F3. Thank you! Please share your telephone number, including the country code _________________________ Integer, constrain to Georgian number formats F4. We intend to follow up with some ACCESS grant recipients for one-hour interviews regarding their participation in ACCESS and associated lessons learned. Would your organization be willing to participate in a follow-up interview? o Yes, my CSO will participate in a follow-up interview, if requested o No, my CSO will not participate in a follow-up interview (→F6) Select one F5. Thank you! Please indicate the email address of the individual we should contact for a follow-up interview, if selected __________________________ Open text Thank you. Your responses have been recorded. This concludes the survey. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 72 MASTER GEORGIA/ACCESS QUESTION BANK NOTE TO INTERVIEWER: THIS TABLE INCLUDES ALL QUESTIONS THAT WILL BE ASKED IN THE EVALUATION AND INDICATES TO WHICH RESPONDENT(S) EACH QUESTION WILL BE ASKED. BELOW THE TABLE, CONSOLIDATED LISTS OF QUESTIONS WILL BE INCLUDED IN A DEDICATED PROTOCOL FOR EACH RESPONDENT. IF REVISIONS TO QUESTIONS ARE REQUIRED, PLEASE MAKE THEM IN THE QUESTION BANK FIRST BEFORE PROCEEDING TO PASTE REVISIONS INTO EACH RELEVANT PROTOCOL. NOTE WORDING MAY BE SLIGHTLY REVISED FROM THE QUESTION BANK TO BE RELEVANT TO EACH RESPONDENT. No. Evaluation Question and Sub Questions KII Stakeholders USAID Donors GoG USAID IPs ACCESS staff ACCESS Key ACCES Private ACCESS Grantees NCCE Media 0.1 Please introduce yourself and describe your role within your organization.           0.2 Please describe your understanding of the various ways that ACCESS is working with the civil society sector in Georgia. a. In your opinion how have ACCESS activities influenced civil society’s development in Georgia?           0.3 Describe briefly your cooperation with ACCESS to date? no ne no ne  no ne no ne     no ne EQ1. How effective has the partner been in supporting CSOs to increase citizen engagement? How did problem analysis and activity design through co-creation contribute to addressing citizen identified issues? 1.1 To your knowledge how does the ACCESS program define civic engagement? a. If organization is not ACCESS: Does your organization have a formal definition of citizen engagement? How is it different from ACCESS’ definition? b. In your personal view, would you modify these definitions in any way to consider other aspects? c. For GoG/ Media only: from your perspective, how would you define civic engagement?       no ne  no ne  CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 73 No. Evaluation Question and Sub Questions KII Stakeholders USAID Donors GoG USAID IPs ACCESS staff ACCESS Key ACCES Private ACCESS Grantees NCCE Media 1.2 What changes have you seen in the ways citizens engage with civil society since ACCESS began in 2014? How do you feel ACCESS has contributed to these changes? [Interviewer: be sure to probe according to how they defined engagement. E.g., if they said awareness, attendance, and action, probe about each of these. You may also probe on some of aspects below, if they do not come up naturally] a. How does your CSO/project engage citizens? b. What are the main drivers (motivations) of citizen engagement? c. What are the main obstacles to citizens’ engagement? d. Can you give examples of successful engagement? e. Are ethnic minorities and other disadvantaged groups equally engaged with civil society? What strategies do CSOs employ to engage these groups? f. What has worked best in the Georgian context to increase citizen engagement? Why? g. What strategies would you say were less effective/ failed to increase citizen engagement? h. How is citizen engagement different in Tbilisi compared to the regions? i. How do you believe the local community perceives CSOs?           1.3 What sorts of issues did CSOs prioritize prior to ACCESS intervention (before 2014)? Have these issues changed since? What role, if any, has ACCESS played in this change?  no ne    no ne no ne  no ne no ne CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 74 No. Evaluation Question and Sub Questions KII Stakeholders USAID Donors GoG USAID IPs ACCESS staff ACCESS Key ACCES Private ACCESS Grantees NCCE Media 1.4 Was your organization involved in any co￾creation (e.g. “social lab”) activities with ACCESS to discuss issues regarding citizen engagement or activity design? a. To your knowledge how did your organization determine citizen priorities? b. In your opinion, are there any differences in citizen priorities among minority and other disadvantaged groups? Are there regional differences? If so, what are they? c. Do these priorities coincide with top issues/concerns in the country/region? Did they change over time? d. How has co-creation impacted other activities? e. What would you change about the activity design in future programming? Why? no ne no ne no ne  no ne    no ne no ne 1.5 To your knowledge how has ACCESS gone about problem analysis and co￾creation of activity design? How has this affected the nature of activities and the issues they address? a. Please describe the co-creation process. To your knowledge how did ACCESS use this process to determine citizen priorities? How have you ensured inclusion of the determined citizen priorities in your activities? b. In your opinion, are there any differences in citizen priorities among minority and other disadvantaged groups? Are there regional differences? If so, what are they? c. Do these priorities coincide with top issues/concerns in the country/region? Did they change over time? d. How has co-creation impacted other activities? e. What would you change about the activity design in future programming? Why?  no ne no ne no ne  no ne no ne no ne no ne no ne CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 75 No. Evaluation Question and Sub Questions KII Stakeholders USAID Donors GoG USAID IPs ACCESS staff ACCESS Key ACCES Private ACCESS Grantees NCCE Media EQ2. How effective has the partner been in supporting CSOs in monitoring and influencing key government policies and processes? What policy areas do key informants mention as lacking in support? How relevant are these thematic areas/policies to citizens’ priorities and key development issues? What were the areas where ACCESS-supported CSOs achieved most notable results? Were there policy areas where ACCESS should have focused more resources to achieve better results to date, based on key informants and public surveys? 2.1 To your knowledge which are the major policy areas where CSOs (your CSO in particular) have attempted to influence government policies and processes with ACCESS’ support? a. For each area, please describe to what extent these attempts have been successful. b. If some failed, why?       no ne  no ne  2.2 To your knowledge are there important policy areas supported by ACCESS which should have been given more resources to achieve better results? Please elaborate.  no ne no ne no ne     no ne no ne 2.3 Are there any important policy/thematic areas that you feel have lacked support from ACCESS (donors in general)? a. Would ACCESS be well positioned to provide support for these areas? b. Are these areas that have lacked support important citizen priorities? If yes, please elaborate why you think this. Why are these policy areas important? c. Are there any issues of national/local concern that lacked support from ACCESS? If so, what are these issues? Why was there a lack of support?    no ne     no ne  EQ3. How successful has ACCESS been in working on local ownership of the Centers for Civic Engagement (CCEs) so that they continue to operate as politically neutral gathering spaces for public discourse? What is the potential of a Network of Centers for Civic Engagement (NCCEs) to diversify services and increase demand for CCE services in each location, strengthen financial diversification, expand services to the private sector, sustain results to date, and continue developing on its own? 3.1 a. What would you say is the main function of CCEs? What needs do they address? b. What is the social profile of people who tend to use the CCEs’ services most often?           CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 76 No. Evaluation Question and Sub Questions KII Stakeholders USAID Donors GoG USAID IPs ACCESS staff ACCESS Key ACCES Private ACCESS Grantees NCCE Media 3.1 To what extent do you think CCEs were used by local communities? c. What is the perception / public image of CCE’s in their local communities? Are they perceived as politically neutral places? Are there any groups (e.g. based on gender, ethnicity, age, etc.) in the community that perceive them differently? d. What aspects/services of CCE’s are important for your community? Why? e. How long do you think it will take for them to return to full operations once the COVID-19 pandemic ends? What it will require? f. For private sector only: [briefly explain what CCEs are/services they provide] have you used the services provided by CCEs? Which services? g. For private sector only: would you be willing to pay competitive fees to use the services of the CCEs? h. For donors/other IPs only: did you cooperate with CCEs, what are your impressions (probe about neutrality) and would you be willing to support their work? i. How can CCEs best serve their communities’ needs in the future? For which needs are they most suitable? Is there any role they may perform in the 2021 local elections?           3.2 To what extent do CCEs rely on USAID funding to remain financially viable? a. What is the share of USAID’s funding for most CCEs? b. For NCCE: what was the share of non-USG funding for CCEs prior to Covid-19? What is current situation?  no ne no ne no ne  no ne no ne no ne  no ne CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 77 No. Evaluation Question and Sub Questions KII Stakeholders USAID Donors GoG USAID IPs ACCESS staff ACCESS Key ACCES Private ACCESS Grantees NCCE Media 3.3 What strategies have CCEs used to generate alternative revenues? a. What other sources of revenue do they rely on? [Interviewer: please probe regarding funding from other donors, local government, and the private sector, crowdfunding, Facebook fundraising, in￾kind contributions: volunteers, business donations] b. How effective were these strategies? How are they affected by COVID-19? c. For GoG: Would the local government be willing to contribute to funding of the CCEs and/or provide in-kind contribution (space, for example)? If so, how should the CCEs preserve perception of neutrality? d. For NCCE: which CCEs do you think can be sustainable? What internal and external factors contribute to their sustainability? e. For NCCE: Could CCEs survive using community foundation model? f. Which additional fee-based services or projects do you think would be a good idea for CCEs to pursue in the future? Which existing services/projects can be replicated in different settings?  no ne  no ne  no ne no ne   no ne 3.4 How has the creation of the NCCE affected the services offered by CCEs, demand for CCE services, partnership with the private sector, and diversification of CCE funding? a. Does the NCCE has a value-added to CCEs? What is it? b. Are there any other important ways the NCCE supports CCEs or civil society in general? c. To what extent do you feel the NCCE is sustainable and can develop on its own? Why do you feel so?  no ne no ne no ne  no ne no ne no ne  no ne CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 78 No. Evaluation Question and Sub Questions KII Stakeholders USAID Donors GoG USAID IPs ACCESS staff ACCESS Key ACCES Private ACCESS Grantees NCCE Media EQ4. How successful has the partner been in facilitating private sector engagement (exploring areas of mutual interest and CSR initiatives) and other inter-sectoral cooperation including to secure private sector funding and support? 4.1 Please describe the relationship between civil society and the private sector in Georgia. [Interviewer: probe about any collaboration either in the past or possibly in the future] a. How has this relationship changed over ACCESS’ implementation? b. What role has ACCESS played in this change? c. For private sector: what is your overall perception of the civil society sector in Georgia?          no ne 4.2 Has ACCESS assisted CSOs in securing private sector funding and /or any other support (networking, new partnerships)? a. If yes, what has worked well? What has not worked? What needs to be improved? b. If no, what could ACCESS do to increase funding and support from the private sector for CSOs? c. For private sector: what types of CSOs/activities have you funded/are willing to fund in the future? d. For private sector: are there any pre￾conditions for funding CSOs? How has ACCESS helped CSOs to comply with those preconditions?  no ne no ne no ne     no ne no ne 4.3 Describe your involvement with the CSR club. a. How has your involvement in the CSR club changed your perception of the civil society in Georgia? b. What kind of new partnerships have you identified through your involvement in the CSR club? no ne no ne no ne no ne no ne no ne  no ne no ne no ne CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 79 No. Evaluation Question and Sub Questions KII Stakeholders USAID Donors GoG USAID IPs ACCESS staff ACCESS Key ACCES Private ACCESS Grantees NCCE Media 4.4 What trajectory do you think collaboration between the private sector and CSOs will take after ACCESS has concluded? a. To what extent will additional support be required to deepen and sustain this collaboration? What kind of support? b. What opportunities exist for CSO￾private sector engagement beyond CSR? c. For NCCE only: could NCCE engage private sector in supporting CCEs? Which CCEs are likely to benefit from such an engagement?          no ne EQ5. How successfully did the project’s PMP measure the stated objectives of the project? Are there higher order, more outcome-oriented indicators that could have been employed? 5.1 Are there important changes brought about by ACCESS that you feel were not captured in its PMP? a. Would you describe these as unintended outcomes, or intended outcomes that were not well captured?  no ne no ne no ne  no ne no ne no ne no ne no ne 5.2 What constraints, if any, prevented ACCESS’ PMP from measuring the project’s stated objectives? For a project like ACCESS in the future, which alternative measures and/or M&E strategies should be used to more faithfully measure evaluation results?  no ne no ne no ne  no ne no ne no ne no ne no ne 5.3 How does your organization (CSOs/donor/IP) measure citizen engagement? a. How does your CSO measure impact on policy? no ne  no ne  no ne no ne no ne  no ne no ne Miscellaneous (including context, recommendations for future programming, etc.)What should be the key priorities for USAID’s civil society development program in Georgia in the future? M.1 Please describe the impact of the COVID￾19 pandemic on ACCESS activities and any key outcomes a. For private sector: Please describe the impact of the COVID-19 pandemic on your CSR activities b. for donors/grantees: Impact of COVID-19 on wider civil society development   no ne    no ne  no ne no ne CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 80 No. Evaluation Question and Sub Questions KII Stakeholders USAID Donors GoG USAID IPs ACCESS staff ACCESS Key ACCES Private ACCESS Grantees NCCE Media M.2 Has Georgia’s 2020 election affected the relationship between government and CSOs? Have the elections affected ACCESS activities and any key outcomes?           M.3 Are there any other important events or contextual factors that you believe have influenced ACCESS’ performance to date? Please describe  no ne no ne no ne  no ne no ne  no ne no ne Closing Question for all key informants and FGDs Thank you for discussing these issues with us today. We asked a lot of questions and learned a lot from you. Is there anything that you want to add, or would like to ask us?           CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 81 Focus Group Discussion – Representatives of CCEs Date: Month: ____ Day: ____ Year: ________ Facilitator: Sample Group: Notetaker: Recording filename: Consent granted to record? Yes/No No. FGD Participants Gender CCE (e.g. Batumi, Gori, etc.) 1 2 3 4 5 6 7 8 9 10 Start time: ___ :___ AM / PM (circle one) End time: ___ :___ AM / PM (circle one) Comments on any aspect of the FGD: CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 82 Introduction and Consent Hello, my name is XXXX, and I am an evaluation consultant. I am working with NORC, a research center associated with the University of Chicago in the US, to conduct an independent evaluation on behalf of USAID of the Advancing CSO Capacities and Engaging Society for Sustainability (ACCESS) project. The primary goal of this study is to inform future USAID programming that will work with local civil society organizations in Georgia. The aim of our discussion today is to understand the current status of centers for civic engagement in Georgia and prospects for future sustainability. You each were selected purposively due to your role as coordinating staff of the CCEs. Our role here is to ask questions and listen to your opinions and experiences. Please note that there are no “right” or “wrong” answers in this discussion. We would like everyone to share their experience, and ask that you allow people with different experiences than yours to express themselves freely. This focus group discussion will require 90 minutes of your time. Your participation is completely voluntary and you can choose to not answer any question or stop participating at any time. You are not obligated to answer any question that you are not comfortable with. However your feedback will be very useful and help design future USAID programs in Georgia. Your response will be kept confidential and anonymous. The information you provide will be used in summary form only, and will not identify you as a participant of this interview/discussion. We ask that everyone here respect each person’s privacy and confidentiality, and not repeat what is said during this discussion. But, please remember that other participants in the group may accidentally share what was said. If you have questions please e-mail Ritu Nayyar-Stone, the team lead for this evaluation at nayyarstone￾ritu@norc.org or Tamara Sartania at tsartania@gmail.com. Thank you very much for your participation. We would like to record this discussion with your permission. This recording will enable us to make sure we faithfully capture your thoughts and ideas during this discussion, and will be deleted following completion of the study. Do you agree? May we start the focus group? [GET VERBAL CONSENT FROM EACH PARTICIPANT OF THE FOCUS GROUP ONCE RECORDING STARTS] CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 83 [NOTE TO FACILITATOR—THE QUESTIONS BELOW ARE MEANT TO BE BROAD AND ENCOURAGING OF RICH DISCUSSION. YOUR ROLE IS TO FACILITATE A PRODUCTIVE DISCUSSION AND ALLOW THE DISCUSSION TO FLOW FREELY. IT IS MANDATORY TO ASK EACH OF THE QUESTIONS, BUT NOT MANDATORY TO COVER ALL OF THE PROBES] Questions Facilitator Instructions and Probes A. Introduction First, let’s go around and introduce ourselves. Please share your name and how long you have worked with the CCE. B. Role for Centers for Civic Engagement 1. What community needs do your CCEs address? a. How would you describe the role that your CCEs play in civic life in your communities? b. How has it changed civic life in this community since its founding? Allow respondents to discuss at least a few minutes before probing. If these do not come up naturally in the discussion, probe:  What role do you think CCEs can play in the 2021 local elections? 2. Which of the services provided by your CCEs are most popular in your community? Why? a. Which community groups use the CCEs’ offered services the most? b. Which community groups use the CCEs’ offered services the least? c. In your opinion, what other services should your CCE offer to increase demand for its services? Why? Allow respondents to discuss at least a few minutes before probing. If these do not come up naturally in the discussion, probe:  Which specific services/aspects of CCEs are especially important for your community? Why? To which community groups are these services particularly important?  What services offered by CCEs, if any, are mostly used by disadvantaged and vulnerable groups? 3. How would you describe the value of the NCCE to your CCEs? a. Do you think the NCCE should be retained following ACCESS’ conclusion, or would you prefer to operate independently? Why? If examples of “value” are required, these could include knowledge sharing, administration, negotiation with national or regional organizations, amplification, etc. Allow respondents to discuss at least a few minutes before probing. If these do not come up naturally in the discussion, probe:  In what other ways can the NCCE support CCEs?  In your opinion, to what extent can NCCE sustainably develop on its own? Why do you think so? CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 84 Questions Facilitator Instructions and Probes 4. What are the prospects for your CCEs to remain financially viable after ACCESS concludes? What are the supporting factors? What are the opposing factors? a. What changes are required for CCEs to remain financially viable? How will these changes affect your CCEs’ role in the community? b. What other than USAID funding sources of revenue do your CCEs rely on? c. What are the opportunities, if any, for introducing the fee-based services? What are the limitations for introducing the fee-based services? How could these limitations be mitigated? Allow respondents to discuss at least a few minutes before probing. If these do not come up naturally in the discussion, probe:  What strategies do you employ for generating alternative revenue? Have you considered any fundraising activities (applying for funds from other donors, engaging the private sector and/or local government, crowdfunding, in-kind contributions, donations)? What are the benefits for doing so? What are the drawbacks of doing so for keeping CCEs’ role?  How has the COVID-19 pandemic affected your CCEs? What will it require for your CCE to recover from the impact of the pandemic? How long will it take to return to full operation once the pandemic ends? 5. How, if at all, do you think operating as a community foundation would change your CCE’s role in local civil society? a. What would it take for your CCE to be viable as a community foundation? If the respondents are not familiar with the definition of community foundation, you can share the following definition from the Council on Foundations: “Community foundations are grantmaking public charities that are dedicated to improving the lives of people in a defined local geographic area. They bring together the financial resources of individuals, families, and businesses to support effective nonprofits in their communities.” C. Citizen Engagement 6. I’d like to discuss a few aspects of citizen engagement in your communities: a. How would you describe the culture of civic engagement in your communities? b. In which ways do citizens tend to engage with civic issues in your communities? c. Has it changed since 2014? If so, how? Allow respondents to discuss at least a few minutes before probing. If these do not come up naturally in the discussion, you may probe for some of the following themes:  Is the culture of citizen engagement or the way citizens engage different here than it is in Tbilisi? If so, in what ways it is different?  Does engagement tend to occur between citizens, citizens and government, or citizens and CSOs?  Are there some sub-groups (gender, age, ethnicity, etc.) who tend to engage more, less, or differently than others? 7. How have local CSOs encouraged citizen engagement with civic issues in this community? a. Do you perceive that local CSOs are effective partners in advancing citizens’ interests in this community? You may skip this question if pressed for time You may need to give some examples of relevant local CSOs, or simply ask about organizations, in case it is unclear. Allow respondents to discuss at least a few minutes before probing. If these do not come up naturally in the discussion, you may probe for some of the following themes:  What specific strategies to encourage citizen engagement have been more or less effective? – Has anyone heard of or participated in a “social lab” approach? What are your views on this approach?  How would you rate the level of trust in CSOs in your communities? In your opinion, how are CSOs perceived in your communities? CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 85 Questions Facilitator Instructions and Probes 8. What strategies do you think would be effective in the future for CSOs to better understand and respond to citizens’ needs in this community? Allow respondents to discuss at least a few minutes before probing. If these do not come up naturally in the discussion, probe:  How would you rate the level of CSOs’ engagement of vulnerable and disadvantaged groups (e.g. women, minorities, LGBT, IDPs, etc.) in your communities?  How could CSOs engage different types of citizens than they currently do? D. Private Sector 9. What roles, if any, do CCEs play in strengthening the connection between local civil society and the private sector? What opportunities exist for CCEs to play this role in the future? Allow respondents to discuss at least a few minutes before probing. If these do not come up naturally in the discussion, probe:  Could the CCE play any role in popularizing corporate social responsibility among local business communities?  Have you proactively tried to engage with the private sector? 10. Have your CCEs been supported by the private sector? If so, could you please provide examples? What opportunities exist to secure funding from the private sector in the future? a. To what extent do you believe CCEs maintain a reputation as a politically neutral space in your communities? b. What change, if any, could happen in public perception about political neutrality if CCEs accept funds from local government or the private sector? E. Closing 11. Is there anything else that you’d like to share with me regarding your CCEs or recommendations for future USAID programming? Thank you again for agreeing to participate in this focus group! [FACILITATOR: STOP RECORDER NOW.] [FACILITATOR: WRITE THE END TIME OF FGD ON COVER PAGE.] CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 86 Focus Group Discussion – Citizens in Regions where ACCESS is Active Date: Month: ____ Day: ____ Year: ________ Facilitator: Sample Group: Notetaker: Recording filename: Consent granted to record? Yes/No No. FGD Participants Gender CCE (e.g. Batumi, Gori, etc.) 1 2 3 4 5 6 7 8 9 10 Start time: ___ :___ AM / PM (circle one) End time: ___ :___ AM / PM (circle one) Comments on any aspect of the FGD: CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 87 Introduction and Consent Hello, my name is XXXX, and I am an evaluation consultant. I am working with NORC, a research center associated with the University of Chicago in the US, to conduct an independent evaluation on behalf of USAID of the Advancing CSO Capacities and Engaging Society for Sustainability (ACCESS) project. The primary goal of this study is to inform future USAID programming that will work with local civil society organizations in Georgia. The aim of our discussion today is to understand how citizens like you engage with civil society in your community and country and to understand which issues are important to you. You and other community members in this discussion were selected randomly from a list of individuals who have attended events which were hosted with the support of the ACCESS program. Our role here is to ask questions and listen to your opinions and experiences. Please note that there are no “right” or “wrong” answers in this discussion. We would like everyone to share their experience, and ask that you allow people with different experiences than yours to express themselves freely. This focus group discussion will require 90 minutes of your time. Your participation is completely voluntary and you can choose to not answer any question or stop participating at any time. You are not obligated to answer any question that you are not comfortable with. However your feedback will be very useful and help design future USAID programs in Georgia. Your response will be kept confidential and anonymous. The information you provide will be used in summary form only, and will not identify you as a participant of this interview/discussion. We ask that everyone here respect each person’s privacy and confidentiality, and not repeat what is said during this discussion. But, please remember that other participants in the group may accidentally share what was said. If you have questions please e-mail Ritu Nayyar-Stone, the team lead for this evaluation at nayyarstone￾ritu@norc.org or Tamara Sartania at tsartania@gmail.com. Thank you very much for your participation. We would like to record this discussion with your permission. This recording will enable us to make sure we faithfully capture your thoughts and ideas during this discussion, and will be deleted following completion of the study. Do you agree? May we start the focus group? [GET VERBAL CONSENT FROM EACH PARTICIPANT OF THE FOCUS GROUP ONCE RECORDING STARTS] CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 88 [NOTE TO FACILITATOR—THE QUESTIONS BELOW ARE MEANT TO BE BROAD AND ENCOURAGING OF RICH DISCUSSION. YOUR ROLE IS TO FACILITATE A PRODUCTIVE DISCUSSION AND ALLOW THE DISCUSSION TO FLOW FREELY. IT IS MANDATORY TO ASK EACH OF THE QUESTIONS, BUT NOT MANDATORY TO COVER ALL OF THE PROBES] Questions Facilitator Instructions and Probes A. Introduction First, let’s go around and introduce ourselves. Please share your name. Please also share what “citizen’s engagement with civil society” means to you. Before the discussion, be sure to review which grants were active in this community. If respondents are unfamiliar with ACCESS, ask if they are familiar with the implementers of local grants (e.g. Young Scientists Club) or the local CCE. If citizen engagement with civil society is not well understood, ask: please share a memorable and rewarding experience where you worked with others in your community to address an issue that was important to you and your community. B. Engagement 1. I’d like to discuss a few aspects of citizen engagement in this community: a. In which ways do citizens tend to engage with civic issues in your community? b. Have you observed any change in citizen engagement since 2014? If so, what has changed? Why? Allow respondents to discuss at least a few minutes before probing. If these do not come up naturally in the discussion, you may probe for some of the following themes:  In your opinion, is there a difference in the culture of citizen engagement in your community compared to Tbilisi? If so, in what ways is it different? Why do you think there is a difference?  In what form does engagement tend to occur? Is it more between citizens, citizens and government, or citizens and CSOs?  Which groups in your community (women, men, youth, elderly, people with disabilities, minority groups, IDPs, etc.) tend to engage more in civic issues? Why do you think that is so? What motivates these groups to engage more?  Which groups tend to engage less? What are the main obstacles for these groups to participate?  Which groups tend to engage differently than others? In what ways is this engagement different? Why? CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 89 Questions Facilitator Instructions and Probes 2. Have you participated in activities organized by local CSOs in your community? If so, please elaborate what the activity was, who organized it, who else participated, what the issue was about and the outcome of the activity? 3. What motivated you to participate in that activity? You may need to give some examples of relevant local CSOs, or simply ask about organizations, in case it is unclear. Allow respondents to discuss at least a few minutes before probing. If these do not come up naturally in the discussion, you may probe for some of the following themes:  What CSOs’ approaches work best to increase citizen engagement in your community? What approaches work less effective in increasing citizen engagement? – Has anyone heard of or participated in a “social lab” approach? What are your views on this approach?  How are CSOs perceived in your community? How would you rate the level of trust in CSOs in your community?  In your opinion, do local CSOs equally engage ethnic minorities and other vulnerable and disadvantaged groups (women, youth, religious minorities, LGBT, IDPs, etc.)? If yes, what approaches local CSOs used to engage these groups?  If no, why do you think so? Should CSOs engage with ethnic minorities and other vulnerable and disadvantaged groups? How? C. Monitoring and Influencing Government Policy and Priorities 4. What are the three most important issues that need to be addressed in Georgia? What are the three most important issues that need to be addressed in your community? Allow respondents to discuss at least a few minutes before probing. If these do not come up naturally in the discussion, probe:  Which issues are most important to you, those at the national level, regional level, or local level? 5. Are there any issues where you think civil society should have devoted more time or resources to make progress since 2014? What about any issues that you think should be a priority in the future? Allow respondents to discuss at least a few minutes before probing. If these do not come up naturally in the discussion, probe:  Gauge if CSOs attempted to address these issues and failed or if they failed to address them entirely. In either case, why is this the case?  Do you feel there are any serious barriers to CSOs advocating for solutions to these issues in the future? If so, what are these barriers? How do you think these barriers could be overcome? D. Centers for Civic Engagement I’d like to close by talking a bit about the center for civic engagement in this community. 6. What is your understanding of the main function of CCEs? What citizens’ needs do they address? What value do CCEs bring to your community? Why? Has it changed civic life in this community since its founding? If so, how? Allow respondents to discuss at least a few minutes before probing. If these do not come up naturally in the discussion, probe:  What is your perception of CCEs? Do you consider CCEs to be politically neutral spaces? What makes you think so?  Would your perception change if CCEs accept funds from local government or the private sector? If so, how?  What role do you think CCEs can play in the 2021 local elections? COVID-19 response? CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 90 Questions Facilitator Instructions and Probes 7. Which of the services provided by the CCE do you use? Which do you find most useful? Why? a. In your opinion, which groups in your community use CCE￾offered services the most? b. Which groups in your community use CCE-offered services the least? c. What other services do you think the CCE should add to its offerings? Why? Come prepared with a list of services the CCE offers. If the community doesn’t mention any, probe as to their awareness of these services. Allow respondents to discuss at least a few minutes before probing. If these do not come up naturally in the discussion, probe:  Are there any aspects of CCEs or services provided by CCEs you consider important for your community? IF yes, which ones? Why are they important?  To which community groups are these services particularly important?  Are there any specific services offered by CCE to vulnerable and disadvantaged groups? If so, what are they? E. Closing 8. Is there anything else that you’d like to share with me regarding civil society in your community or the ACCESS project? Thank you again for agreeing to participate in this focus group! [FACILITATOR: STOP RECORDER NOW.] [FACILITATOR: WRITE THE END TIME OF FGD ON COVER PAGE.] CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 91 ANNEX C: SOURCES OF INFORMATION DOCUMENTS REVIEWED NO. DOCUMENT TYPE TITLE YEAR PUBLISHED AUTHORING ENTITY/ INDIVIDUAL 1 ACCESS Program Document ACCESS First Annual Report 2015 EWMI 2 ACCESS Program Document ACCESS Second Annual Report 2016 EWMI 3 ACCESS Program Document ACCESS Third Annual Report 2017 EWMI 4 ACCESS Program Document ACCESS Fourth Annual Report 2018 EWMI 5 ACCESS Program Document ACCESS Fifth Annual Report 2019 EWMI 6 ACCESS Program Document ACCESS Sixth Annual Report 2020 EWMI 7 ACCESS Program Document ACCESS Year 1 PMP Table of Indicators EWMI 8 ACCESS Program Document ACCESS Year 2 PMP Revision Memo EWMI 9 ACCESS Program Document ACCESS Year 2 PMP Table of Indicators (Revised) EWMI 10 ACCESS Program Document ACCESS Year 3 PMP Revision Memo EWMI 11 ACCESS Program Document ACCESS Year 3 PMP Table of Indicators (Revised) EWMI 12 ACCESS Program Document ACCESS PMP Table of Indicators as of 2019 2019 EWMI 13 ACCESS Program Document CiDA Georgian Private Sector CSR Database CiDA 14 ACCESS Program Document National Baseline Assessment - Business and Human Rights in Georgia 2017 CiDA 15 ACCESS Program Document R-CSN Membership Catalogue 2015 R-CSN 16 ACCESS Program Document Women Voters Needs and Priorities Survey 2017 R-CSN 17 ACCESS Program Document Responding to Russian Disinformation: A Case of Georgia 2019 UNAG 18 ACCESS Program Document Assessment of the Civil Society Sector in Georgia 2019 GIPA 19 ACCESS Program Document Applied Social Lab Methodology Handbook 2020 Reos Partners 20 ACCESS Program Document G-PAC Civic Engagement Survey 2014 EWMI 21 ACCESS Program Document NCCE Potential Role Supporting EU Integration and Anti-Propaganda Efforts EWMI CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 92 NO. DOCUMENT TYPE TITLE YEAR PUBLISHED AUTHORING ENTITY/ INDIVIDUAL 22 ACCESS Program Document NCCE Brief 2019 EWMI 23 ACCESS Program Document NCCE Role in Georgia's Socioeconomic Development EWMI 24 ACCESS Program Document CCE Transformation Action Plan Brief 2016 EWMI 25 ACCESS Program Document CCE Marketing Strategy 2017 EWMI 26 USAID Document Georgia DRG Assessment 2020 NORC 27 Population Based Survey Report Georgia Public Opinion Survey 2019 IRI 28 Gray Literature From Apathy to Action: Attitudes to Civic Engagement in the Eastern Partnership 2020 GMF 29 ACCESS Program Document OACA Score Database 2020 CTC 30 Gray Literature CSR Quantitative Survey 2019 CiDA 31 ACCESS Program Document Assessment of the Civil Society Sector in Georgia 2019 EWMI 32 USAID Document 2019 CSO Sustainability Index - Europe and Eurasia 2019 USAID 33 Gray Literature 2020 Open Democracy 34 USAID Document USAID Private Sector Engagement Policy 2020? (actually not marked anywhere) USAID 35 Gray Literature Social Laboratory Literature Review 2020 CTC KEY INFORMANTS No. Stakeholder Type Organizational Affiliation Name Title 1 Donors active in civil society European Union Dominika Skubida Programme Manager, Civil Society and Democratisation 2 Donors active in civil society Soros Foundation Keti Khutsishvili Executive Director 3 Donors active in civil society Dutch Embassy Lela Lomia Policy Officer 4 Donors active in civil society SIDA Stella Ahlin Marceta Kakha Khimshiashvili 2nd Secretary,National Programme Officer CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 93 No. Stakeholder Type Organizational Affiliation Name Title 5 Government of Georgia Parliament of Georgia Nino Goguadze Former MP 6 Government of Georgia Tbilisi Mayor's office Aleksandre Togonidze Director, Tbilisi Development Fund 7 Government of Georgia EU and NATO Integration Center Tamar Medulashvili Former Deputy Director 8 Government of Georgia Government Administration Saba Jikidze Former head of Srategic Communications Division 9 Government of Georgia Ozurgeti self￾government Giorgi Ghurjumelidze Nana Tadumadze Deputy Chair of Sakrebulo Chief of Staff, Mayor’s Office 10 Government of Georgia Transport Department, MoESD Erekle Kezherashvili Deputy Head of Transport Department 11 Government of Georgia Public Defender’s Office Eka Skhiladze Deputy Ombudsman 12 Other USAID IPs in civil society MTAG Lia Chakhunashvili Chief of Party 13 Other USAID IPs in civil society GGI Levan Samadashvili Chief of Party 14 Other USAID IPs in civil society MT Marina Ushveridze Chief of Party 15 ACCESS key partners CTC Irina Khantadze Tamar Tatishvili Executive Director NMP Program Manager 16 ACCESS key partners CiDA Salome Zurabishvili Etuna Tsotniashvili Executive Director Communications Manager 17 ACCESS key partners R-CSN Pikria Gelashvili Director, Welfare and Development Center 18 ACCESS private sector partners GPI Holdings Tina Stambolishvili Director of Communications 19 ACCESS private sector partners Crystal Maia Kobalia Head of Environmental and Social Division 20 ACCESS private sector partners Efes Georgia Nikoloz Kundzakishvili Corporate Affairs Director 21 Media Batumelebi Mzia Amaglobeli Editor-in-Chief 22 Media Radio Marneuli Kamila Mamedova Director and Founder 23 Media Samkhretis Karibche Gulo Kokhodze Director CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 94 No. Stakeholder Type Organizational Affiliation Name Title 24 NCCE NCCE Nina Khatiskatsi Executive Director 25 ACCESS Grantee-COG 1 Orbeliani Foundation Keti Khachidze Executive Director (former) 26 ACCESS Grantee-COG 2 Gavigudet Tinatin Maghadani Founder 27 ACCESS Grantee-COG 3 New Thinking Institute (NTI) Kamran Afandyev Founder 28 ACCESS Grantee-COG 4 Urban Lab Giorgi Babunashvili Founder 29 ACCESS Grantee-PPAG 1 GYLA Vakhushti Menabde Program Director 30 ACCESS Grantee-PPAG 2 EPRC Nino Evgenidze Executive Director 31 ACCESS Grantee-PPAG 3 Batomi Nino Inaishvili 32 ACCESS Grantee-PPAG 4 CRRC Koba Turmanidze Director 33 ACCESS Grantee-RRG 1 Georgian Journalists Ethics Charter Mariam Gogosashvili Executive Director 34 ACCESS Grantee-RRG 2 Akaki Akhalaia Akaki Akhalaia Individual 35 ACCESS EWMI Tarama Gurchiani Deputy Chief of Party 36 ACCESS EWMI Manana Tatishvili Monitoring and Evaluation and Public Outreach Specialist 37 ACCESS EWMI Ana Drakic Delina Fico Current and Former ACCESS Home Office Directors 38 ACCESS EWMI Ana Tsurtsumia ACCESS Capacity Development Manager 39 ACCESS EWMI Nodar Tangiashvili ACCESS Public Policy and Accountability Adviser 40 USAID EWMI Tamuna Karosanidze ACCESS Chief of Party 41 USAID USAID/Georgia Adam Schmidt Director, Office of Democracy, Human Rights and Governance 42 USAID USAID/Georgia Ketevan Bakradze Democracy and Governance Team Leader ACCESS COR 43 ACCESS USAID/Georgia Lina Panteleeva CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 95 ACCESS PMP SUMMARY Table 8. ACCESS PMP Indicator Summary EXPECTED RESULT MEASURE DATA SOURCE COMMENTS ON USE USAID/Georgia CDCS Intermediate Result 1.1: A more informed and engaged citizenry Program Purpose: Increased Civic Engagement, Including Outside of Tbilisi (Aligned with sub-purpose 1 of TAG) Number of public policies introduced, adopted, repealed, changed or implemented consistent with citizen input Grantee records, policy documents, laws and regulations Measured consistently from project inception to date. Program Purpose: Increased Civic Engagement, Including Outside of Tbilisi (Aligned with sub-purpose 1 of TAG) USAID CSO Sustainability Index: Overall Score USAID’s CSO Sustainability Index Measured consistently from project inception to date. Program Purpose: Increased Civic Engagement, Including Outside of Tbilisi (Aligned with sub-purpose 1 of TAG) Change in percentage of population exhibiting civic engagement behaviors in regions where ACCESS works USAID DRG Indicators Survey Not collected after Year 3, deleted from PMP in Year 6 ACCESS Objective 1: Citizens more aware of and involved in CSO activities CSOs utilize volunteers in carrying out their activities CSOs seek citizen support for and involvement in their activities CSOs regularly consult with constituents to identify problems and solutions Percentage increase of citizens reporting having volunteered or participated in CSO activity in the last two years Grantee Records Project Records Not collected years 4 & 5. Year 6, redefined as "Percentage increase in number of citizens taking part in project assisted CSO activities and other project￾initiated citizen engagement activities" CSOs seek citizen support for and involvement in their activities CSOs regularly consult with constituents to identify problems and solutions Percentage increase of citizens being contacted by NGOs to be informed or be invited to events USAID DRG Indicators Survey Not collected after Year 3, deleted from PMP in Year 6 CSOs utilize volunteers in carrying out their activities CSOs seek citizen support for and involvement in their activities CSOs regularly consult with constituents to identify problems and solutions Number of volunteers recruited and used by project assisted CSOs and other project-supported activities Grantee Records Project Records Measured consistently from project inception to date. CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 96 EXPECTED RESULT MEASURE DATA SOURCE COMMENTS ON USE CSOs receive funding from more diverse sources Number of crowdfunding initiatives by the project supported CSOs and civic groups Grantee Records Project Records Added in Year 6, not collected prior CSOs regularly consult with constituents and informal civil society to identify problems and solutions. CSOs seek citizen support for & involvement in activities. Number of the project supported civic activities designed through a co￾creation process Grantee Records Project Records Added in Year 6, not collected prior ACCESS Objective 2: CSO capacity and sustainability improved CSO governance and management strengthened Leadership skills of senior and mid-level CSO managers enhanced Percent change in CSO Organizational Capacity Score CTC Records – OACA score at the beginning of the grant program and at the completion of CTC TA for grantees Measured consistently from project inception to date. CSOs receive funding from more diverse sources Percentage of ACCESS￾supported CSOs that report new funding sources Grantee Records Measured consistently from project inception to date. CSOs partner with private sector to promote common public causes and increase sustainable CSR initiatives. CSOs receive funding from more diverse sources Number of the project initiated and supported private sector engagement activities Grantee Records Project Records Added in Year 6, not collected prior ACCESS Objective 3: CSOs monitor and influence government policies and processes Civil society engagement with government increased Dimension of USAID CSO Sustainability Index: Advocacy USAID CSO Sustainability Index Not applicable in Year 4 Issues identified and advocated for by civil society are included in government policies and legislation A number of advocacy campaigns led by target CSOs leading to concrete policy or legislative changes Grantee Records Measured consistently from project inception to date. Think tanks produce high quality research of relevance to policy-making Percentage of think tank policy research studies revised as a result of ACCESS-facilitated peer review Grantee Records, Peer Review Analyses, Policy Research Studies Not applicable in Year 6 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 97 EXPECTED RESULT MEASURE DATA SOURCE COMMENTS ON USE ACCESS Objective 4: Local ownership of CCEs as politically neutral gathering spaces CCEs increasingly attract more diverse funding Percentage of CCE operational expenses funded by non-USAID funds CCE Records Measured consistently from project inception to date. CCEs serve as a politically neutral gathering place open to all groups Percentage increase of citizens who are correctly aware of Centers for Civic Engagement USAID DRG Indicators Survey Not collected after Year 3, deleted from PMP in Year 6 CCEs serve as a politically neutral gathering place open to all groups Number of CCEs operating independently from the project funding CCE Records, Project Records Added in Year 6, not collected prior CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 98 ANNEX D: DISCLOSURE CONFLICTS OF INTEREST Name Tamara Sartania Title Local Evaluation Specialist Organization NORC Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) GS-10F-0033M USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) ACCESS Georgia I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 30 April, 2021 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 99 Name George Gotua Title Local Evaluation Expert Organization Independent Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) GS-10F-0033M USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) EWMI ACCESS I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 01.02.2021 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 100 Name Orysia Lutsevych Title Civil Society Consultant Organization NORC Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) GS-10F-0033M USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) ACCESS Georgia I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 1 May, 2021 CONTRACT No. GS-10F-0033M / ORDER No. 7200AA18M00016 / DRG-LER II TASKING N040 USAID.GOV GEORGIA ACCESS PERFORMANCE EVALUATION: FINAL REPORT | 101 Name Zumrat K. Salmorbekova Title Evaluation Specialist and Reviewer Organization National Opinion Research Center Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) GS-10F-0033M / 7200AA18M00016 Tasking N040 USAID Project(s) Evaluated (Include project name(s), implementername(s) and award number(s), if applicable) Georgia ACCESS Performance Evaluation I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interestmay include, but are not limited to: 1.Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) arebeing evaluated. 2. Financial interest that is direct, or issignificant though indirect, in the implementing organization(s) whoseprojects are being evaluated or in the outcome of the evaluation. 3.Current or previous direct or significant though indirect experience with the project(s) beingevaluated, including involvement in the project design or previous iterations of the project. 4.Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5.Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6.Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietaryinformation of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date June 18, 2021 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523