Evaluation Report: Mid-term Performance Evaluation of Policy, Advocacy and Civil Society Development in Georgia November 2012 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by International Business & Technical Consultants, Inc. (IBTCI) under the Evaluation Services IQC, REQ￾114-12-000071. MID-TERM PERFORMANCE EVALUATION – POLICY, ADVOCACY AND CIVIL SOCIETY DEVELOPMENT IN GEORGIA Prepared by: David Sip Zehra Kacapor-Dzihic Marine Egutia November 10, 2012 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Table of Contents Acronyms ..................................................................................................................................................... a Acknowledgements ..................................................................................................................................... b Executive Summary ...................................................................................................................................... i I. Introduction ............................................................................................................................................. 1 II. Evaluation Purpose and Evaluation Questions ................................................................................... 1 III. Evaluation Design, Methods, and Limitations ................................................................................... 3 IV. Findings, Conclusions, and Recommendations .................................................................................. 5 Evaluation Question #1 ......................................................................................................................... 5 Evaluation Question #2 ....................................................................................................................... 15 Evaluation Question #3 ....................................................................................................................... 17 Evaluation Question #4 ....................................................................................................................... 20 Evaluation Question #5 ....................................................................................................................... 23 ANNEX I.: Disclosure of Conflict of Interest Forms ............................................................................... 26 ANNEX II: Evaluation Statement of Work (SoW) from the RFTOP ................................................... 29 ANNEX III: Evaluation Methodology, Data Collection Plan, and Limitations .................................... 33 ANNEX IV. : Data Collection Instruments-Interview Guides/Mini-Surveys/Focus Groups ........... 39 ANNEX V: Bibliography of Documents Reviewed .................................................................................. 56 ANNEX VI: In-Country Schedule of Meetings ......................................................................................... 57 ANNEX VII: List of G-PAC Interviews ..................................................................................................... 60 ANNEX VIII: Mini-Survey and Focus Group Reports ............................................................................ 62 ACRONYMS ACT ACT Market Research & Consulting CEE Central and Eastern Europe CSI Civil Society Institute CTC Center for Training and Consultancy CSOs/NGOs Civil Society Organizations/Non-governmental Organizations DG Democracy and Governance ET Evaluation Team EQ Evaluation Question EWMI East-West Management Institute FGs Focus Groups G-PAC Policy, Advocacy, and Civil Society Development in Georgia IBTCI International Business & Technical Consultants, Inc. ICNL International Center for Not-for-Profit Law ISU Ilia State University KIIs Key Informant Interviews MPA Master’s in Public Administration MTE Mid-Term Evaluation RFTOP Request for Task Order Proposals SOW Statement of Work TTs Think Tanks TTF Think tank Fund of the Open Society Institute/Budapest TTSGs Think tank Support Grants USAID United States Agency for International Development USG United States Government a ACKNOWLEDGEMENTS The Evaluation Team for this report consisted of David Sip, Marine Egutia, and Zehra Kacapor￾Dzihic. Together, the team designed and conducted data collection and analysis. The team would like to thank the staff at USAID/Georgia, for their support and for making this a very interesting and challenging assignment. Special thanks are extended to the personnel from the CSOs, think tanks, and partner organizations whose views helped form a more complete understanding of the context in which G-PAC was designed and implemented. The team would like to thank the EWMI staff in Georgia, headed by Mr. Fron Nahzi and Ms. Tamuna Karosanidze, for their always quick responses to our many questions and requests for assistance in facilitating the evaluation. Finally, the team expresses gratitude to Maia Bibileishvili and Giorgi Sirbiladze for their dependable logistical support and good humor. b EXECUTIVE SUMMARY Introduction International Business and Technical Consultants, Inc. (IBTCI) submits this report to USAID/Georgia as a deliverable under AID-114-TO-12-0008 G-PAC Mid-Term Project Performance Evaluation. The report was prepared following a desk review, field work and data analysis undertaken by the Evaluation Team (ET) consisting of a team leader, a subject matter expert, and an international evaluation specialist who guided a local firm that was responsible for conducting focus groups and mini-surveys. The period of project performance covered by the evaluation is from the start of the project in October 2010 through the initiation of this evaluation in August 2012. Background of Project Policy, Advocacy, and Civil Society Development in Georgia (G-PAC) is a four-year $13.3 million initiative funded by the United States Agency for International Development (USAID) to strengthen civil society’s role in advocating for and influencing the development and effective implementation of public policy reforms in Georgia. G-PAC is implemented by the New York-based East-West Management Institute (EWMI) with an office and staff in Tbilisi. G-PAC provides support to CSOs and think tanks (TTs) by providing them with comprehensive technical assistance and grants to pursue and address the most pressing issues in Georgia such as healthcare, education, environmental sustainability, unemployment, children’s rights, elections, and property rights. The program was at its midpoint after completing two years of implementation when this evaluation was conducted. Purpose of Evaluation; Key Questions; and Audience The purposes of the mid-term performance evaluation are to: 1) analyze the progress of the G-PAC program towards achieving results in the project’s objectives; 2) offer recommendations for continued project implementation; and 3) provide feedback to stakeholders involved in civil society advocacy activities in Georgia, USAID/Georgia, other USG agencies, other donors, the implementing partner, CSOs, think tanks (TT) and universities. The evaluation sought to answer the following questions: 1. How effective are G-PAC’s interventions to strengthen advocacy and watchdog roles of targeted Tbilisi-based and regional CSOs? 2. How effective are networks of CSOs supported under the G-PAC program? To what extent did project interventions result in strengthening their connections with each other, media, political parties, and constituencies? 3. How effective are G-PAC’s interventions to strengthen research capacity of targeted think tanks? 4. What are major challenges facing the newly established MPA program at ISU both in terms of academic and administrative work? 5. To what extent does G-PAC’s CSO Enabling Environment Initiative, as well as other interventions, contribute to financial sustainability of CSOs? What are the external factors affecting G-PAC’s work in this area? Evaluation Methods IBTCI’s ET employed a “mixed methods” approach to optimize the analysis, reach sound conclusions, respond to the five above evaluation questions posited in the Scope of Work (SOW), and verify findings. These methods included document review, key informant interviews, mini￾surveys, focus group discussions, and briefing meetings with USAID and EWMI Program staff. i The G-PAC Mid-term Performance Evaluation has the following characteristics: • It examined the two-year G-PAC project, not prior USAID civil society activities. • It is principally qualitative but uses quantitative data from surveys to support the arguments. It does not repeat quantitative information already provided in project reports and documents, and uses this information as basis for interviews and analysis of the findings from the field. • The evaluation design incorporates data from the desk review, fieldwork (key informant interviews, mini surveys, focus groups), and evaluators’ professional views and experience. This diverse set of data and sources enabled triangulation of data and the provision of sound assessments and recommendations. The triangulation of the data ensures objectivity, and balanced conclusions and recommendations drawn from multiple sources. • The evaluation is both retrospective and prospective. • The evaluation did not focus on other donor or other USG civil society activities. • Recommendations mainly refer to the current approved project Work Plan and findings of this project process evaluation. Recommendations offered vis-à-vis any necessary course-correction or modifications of the project for the further implementation of the project are based on information gleaned and/or technical expertise possessed by the ET. There were two overall limitations in the evaluation: 1) the evaluation occurred during the pre￾election period of the parliamentary election so CSOs were not able to provide the ET with requisite attention; and 2) the length of the fieldwork as prescribed in the SOW was limited resulting in rushed work products. Major Findings, Conclusions, and Recommendations Evaluation Question #1: How effective are G-PAC’s interventions to strengthen advocacy and watchdog roles of targeted Tbilisi-based and regional CSOs? Major Findings: G-PAC’s interventions to strengthen advocacy and watchdog roles of targeted Tbilisi-based and regional CSOs are effective. There are some interventions that are more appreciated than others, such as mentoring and individual consultations, but the whole package of interventions received, when at their best, mean that CSOs and TTs are receiving a complete package of training and technical assistance that addresses program and organizational development. Evidence indicates that CSOs have a better understanding of advocacy and what it means to engage in advocacy activities, and they are better equipped with knowledge, skills, behaviors, and relationships necessary to conduct more effective advocacy. Most all of the CSOs and TTs who have received G-PAC technical assistance have achieved positive outcomes with their advocacy campaigns. G-PAC is meeting or exceeding its PMP goals. Conclusions: • G-PAC demonstrates it is possible to influence policy-making if recommendations made by CSOs and TTs are reasonable, realistic, objective, and address priority needs of target communities. • G-PAC’s interventions increased the visibility and reputation of CSOs and expanded and strengthened their social networks. Recommendations: • G-PAC should encourage CSOs to be more integrated and connected to communities to develop sustainable cross-sector cooperation, including with the private sector. ii • G-PAC should encourage CSOs to become better aware of the distribution of responsibilities and available resources between central and local governments and to make their advocacy campaigns more realistic. • G-PAC should encourage effective mechanisms for sharing information and experiences, and for promoting possible cooperation between the following: different G-PAC project components so key trainers and mentors have a comprehensive understanding of the program goals and objectives, and accordingly, their specific roles in achieving them; other projects working on the development of civil society, like Project Harmony and MSI who are working with civil society, local and central governments and using similar approaches for their project design and implementation; and CSOs that are not the part of the G-PAC program to amplify project results. Evaluation Question #2: How effective are networks of CSOs supported under the G-PAC program? To what extent did project interventions result in strengthening their connections with each other, media, political parties, and constituencies? Findings: The CSO networks supported under the G-PAC program are proving to be effective in terms of conducting successful advocacy campaigns, but are less effective in terms of internal management and having clear roles, responsibilities, and expectations of their members. CSOs and TTs believe in the strength and successes coalitions can bring, and critical connections with decision-makers and others are being established and maintained; but CSOs are skeptical about coalitions and are hesitant to join them. G-PAC is meeting its PMP goals. Conclusion: • Despite existing skepticism for joining and working in coalitions, CSOs and TTs increasingly understand the importance of networking and coalition-building in advocacy and policy making. Recommendations: • G-PAC should consider a focus on establishing public-private partnerships. • G-PAC needs to continue to support coalitions through a needs-based capacity-building and grants program. Evaluation Question #3: How effective are G-PAC’s interventions to strengthen research capacity of targeted think tanks? Findings: G-PAC’s interventions to strengthen the research capacity of targeted TTs are demonstrating some effectiveness and achieving positive results, but working with TTs remains challenging. There are examples of the think tanks conducting successful research, using their research for advocacy purposes, or conducting coalition advocacy with the research, but their level of engagement in the G-PAC Program and acceptance of their need for training and development remains a challenge. G-PAC is meeting their PMP goals. Conclusions: • Sector-based policy research is still in short supply in Georgia because very few research organizations have deep knowledge of research methods and research design. • There is no peer-review practice for research and the provision of feedback which makes it difficult to measure the quality of research products. iii Recommendation: • Prioritize funding to TTs who produce information by conducting original field research; encourage partnerships and coalitions between research producing TTs with advocacy organizations; introduce a peer-review practice in Georgia for quality control of research products; introduce an open door grant competition to allow thinks tanks to address timely burning policy issues – much like the Act Now Grants. Evaluation Question #4: What are major challenges facing the newly established MPA program at ISU both in terms of academic and administrative work? Findings: The major challenges facing the newly established MPA program at ISU are to ensure the following: that adequate support and resources exist among the faculty and staff of the university to make the program sustainable; that there is steady, gradual, and sustainable growth of the MPA program; and that adequate support for the MPA program among Georgia’s public and private sector employers exists. Conclusions: • There is an enthusiastic new leadership in place at Ilia State University (ISU) that is willing to learn and engage constructively to overcome the challenges of developing the Masters in Public Administration (MPA) Program. • ISU’s leadership has a strategic plan in place to guide the development of the program and is pursuing the plan with vigor. • ISU’s MPA Program has received noteworthy attention from Georgian public sector officials, and student applications increased from the first selected cohort of students to the newly selected in-coming class. Recommendations: • Emphasize a pace of gradual, sustainable growth for ISU’s MPA Program following the strategic plan; emphasize broad participation and buy-in to the next strategic plan to ensure that adequate resources are committed to the program. • Encourage ISU to promote the program to public and private sector employers in Georgia, and continue to develop long-term relationships between the ISU Program and Georgia’s public and private sectors. • Consider an activity or program that encourages professor and student exchanges, and the sharing of practices and knowledge between universities. Evaluation Question #5: To what extent does G-PAC’s CSO Enabling Environment Initiative, as well as other interventions, contribute to financial sustainability of CSOs? What are the external factors affecting G-PAC’s work in this area? Findings: G-PAC’s Enabling Environment Initiative is contributing to the financial sustainability of CSOs and TTs, but at a very gradual, if incremental, pace. Appropriate and thorough analyses of laws and regulations have taken place and have been well-vetted with stakeholders. Both legislative and non￾legislative issues potentially impeding the ability of CSOs to operate effectively have been identified and prioritized. At least one amendment to the tax law has been proposed and passed by Parliament, but there are many more changes to pursue. Non-legislative issues also need continued pursuit, and relationships with the private sector should be maintained and built upon. G-PAC did not meet its PMP goals for Year One, but has set more realistic goals for Year Two and is meeting those goals. iv v Conclusions: • Some of the legal enabling environment issues have been resolved through legislative amendments proposed by G-PAC grantee CSOs, but there still remains little incentive by the private sector to contribute to charitable organizations and Georgian CSOs continue to rely heavily on foreign grants. • Establishing and maintaining relationships between CSOs and the private sector will be important as civil society evolves in Georgia. The private sector will need to better understand the benefit of contributing to and cooperating with CSOs. Recommendations: • G-PAC should pursue activities leading to the successful resolution of the remaining top 10 legal tax issues identified as priorities, and the identified key non-legislative issues. • G-PAC should consider developing a standing committee of key CSOs and key Ministry of Finance staff for long-term, continued dialogue and relationship-building for purposes of drafting legislative changes. Select Best Practices of the Project ¾ Innovations in Applying for Grants: G-PAC has remained consistently flexible in the grant application process to encourage applications from a wide range of CSOs. The use of Skype to conduct application reviews and allowing video submissions are creative means for encouraging CSOs of varied means to participate. ¾ Mentoring and Consultations: Mentoring and tailored consultations have proven perhaps the most effective means of supporting incremental development of CSOs. ¾ Study Tours: The use of study tours and other activities to expose CSOs to other ideas, models, and examples is critical to their growth and development. ¾ Encouraging Coalitions between TTs and Advocacy Organizations: The grant program to encourage these types of organizations to work together builds better advocacy initiatives. ¾ Examples of Unlikely Coalition Partners: The Insurance Association of Georgia’s (IAG) advocacy work is an example of a public-private partnership that could be built on and used as a model of cooperation between sectors. Additionally, IAG has partnered with local CSOs, including pensioner groups, to conduct research for future projects. I. INTRODUCTION International Business and Technical Consultants, Inc. (IBTCI)1 submits this evaluation report to USAID/Georgia as a deliverable under AID-114-TO-12-0008 following the G-PAC Mid-Term Project Performance Evaluation Scope of Work (SOW) (Annex II). The evaluation report was prepared following a project desk review, field work and data analysis undertaken by the Evaluation Team (ET) consisting of the team leader, a subject matter expert, and an international evaluation specialist who guided a local firm that was responsible for conducting focus groups and mini￾surveys. The period of project performance covered by the evaluation is from the start of the project in October 2010 through the initiation of this evaluation in August 2012. II. EVALUATION PURPOSE AND EVALUATION QUESTIONS The purposes of the G-PAC mid-term performance evaluation are to: 1. Analyze the progress of the G-PAC program towards achieving results in the program's four objectives and two components. 2. Offer recommendations vis-à-vis any necessary course-corrections or modifications of the program for the further implementation of the project. 3. Provide feedback to stakeholders involved in civil society advocacy activities in Georgia: USAID/Georgia, other USG agencies, other donors, the implementing partner, and CSOs, think tanks, and universities. The USAID/Georgia mission and its Democracy and Governance Office requested the evaluation to understand existing constraints and shortcomings that might hinder project implementation. The results of the evaluation will be especially useful for the implementing partner to receive an independent view about project progress. The results will also be useful for other donors and organizations working on developing civil society in Georgia, and civil society representatives themselves. Finally, the results will serve as evidence for the FY 2012 Performance Plan and Report (PPR) development scheduled for the fall 2012. The G-PAC Project is being evaluated based on the following questions. Evaluation Questions 1. How effective are G-PAC’s interventions to strengthen advocacy and watchdog roles of targeted Tbilisi-based and regional CSOs? 2. How effective are networks of CSOs supported under the G-PAC program? To what extent did project interventions result in strengthening their connections with each other, media, political parties, and constituencies? 3. How effective are G-PAC’s interventions to strengthen research capacity of targeted think tanks? 4. What are major challenges facing the newly established MPA program at ISU both in terms of academic and administrative work? 5. To what extent does G-PAC’s CSO Enabling Environment Initiative, as well as other interventions, contribute to financial sustainability of CSOs? What are the external factors affecting G-PAC’s work in this area? Project Background and Context Civil society in Georgia began to emerge in the late 1980s and witnessed vibrant activism in the second half of the 1990s. However, many observers have pointed to a decline in the impact of the civic sector following the departure into politics of influential civic activists and CSO leaders who participated in the peaceful demonstrations of 2003’s Rose Revolution.2 1 Please see Annex I for Conflict of Interest Statements. 2 Nations-In-Transit Georgia 2012 Freedom House 1 There is now concern that the overall visibility and political influence of civil society organizations is diminishing. Civil Society Organizations (CSOs), think tanks (TTs) in particular, are losing credibility in the eyes of both the government and citizenry. This greatly hampers their ability to engage the government constructively on key policy issues. Moreover, the absence of substantive public debate and discussion about key social, economic, and political issues exacerbates the lack of public trust and confidence in state institutions and in CSOs. Although some CSOs press for more transparency and the increased participation of civil society, cooperation with the government is not actively sought out and most CSOs limit their involvement to watchdog activities. Financial instability remains a major concern of Georgia’s civil society organizations. International funding and partnering are available, but are monopolized by the country’s most established and successful groups. Local funding is still scarce, and while business entities provide funds for charity, they refrain from funding civic activities.3 Evaluation survey results indicate that 39% of CSOs find it challenging to access resources from donors.4 According to the Democracy and Governance Assessment of Georgia cited in the SOW for this evaluation, “Civil society is dominated by capital-based advocacy and watchdog NGOs, much as it has been since the 1990s. Despite considerable experience with donor-funded projects and training, the policy analysis capacity of these NGOs remains limited and has to be bolstered by external experts. These NGOs excel at critiquing government actions, but constructive recommendations based on evidence and cost-benefit analysis are rare.”5 USAID designed Policy, Advocacy and Civil Society Development in Georgia (G-PAC) as the United States Government’s (USG’s) primary program to strengthen civil society in Georgia and to increase the efficacy of civic leaders and organizations. The project aims to strengthen civil society’s role in developing public policy reforms and advocating for their effective implementation. G-PAC beneficiaries include Georgian TTs, CSOs, universities, government, and the public. G-PAC Project Goal: The G-PAC project goal is to strengthen civil society’s role in advocating for, influencing the development and implementation of, and monitoring effective public policy reforms in Georgia. G-PAC Project Objectives 1. Strengthen the advocacy and watch-dog roles of civil society organizations (CSOs) as advocates for an empowered citizenry; 2. Improve public policy discourse, research, and analysis leading to the development of effective public policies; 3. Enhance the institutional capacity of universities in the areas of public policy, public administration, and political science; and 4. Develop strong networks among CSOs throughout Georgia, and strengthen their ties to media, political parties, and their constituencies. The activities G-PAC pursues to accomplish the four project objectives are organized under two components staffed full time by a combination of Georgian nationals and foreign nationals. Component One: G-PAC works to strengthen the civic engagement of CSOs in effective advocacy and watch-dog activities, improve public policy analysis, and foster more cohesive CSO networks. The project envisions that this empowered engagement secures better access to and dialogue with political parties, local constituencies, and the media. This component offers comprehensive technical assistance and grants that benefit Georgian TTs, local and regional CSOs, and networks/coalitions engaged in policy analysis and advocacy. Component Two: G-PAC is collaborating with Ilia State University (ISU) to establish a pilot MPA Program that operates in conjunction with other central and regional universities. The ISU MPA 3 Op Cit 4 G-PAC Grantees Survey Report, ACT, October 2012 see Annex IV 5 Democracy and Governance Assessment of Georgia 2 program is to serve as a focal point for public policy research and an incubator for the next generation of Georgian policy-makers and policy analysts. III. EVALUATION DESIGN, METHODS, AND LIMITATIONS The G-PAC Mid-term Performance Evaluation has the following characteristics (see Annex III for Evaluation Methodology and Data Collection): • It examined the two-year G-PAC project, not prior USAID civil society activities. • It is principally qualitative but uses quantitative data from surveys for support. It does not repeat quantitative information already provided in project reports and documents, and uses this information as basis for interviews and analysis of the findings from the field. • The evaluation design incorporates data from the desk review, fieldwork (key informant interviews, mini surveys, focus groups), and evaluators’ professional views and experience. This diverse set of data and sources enabled triangulation of data and the provision of sound assessments and recommendations. The triangulation of the data ensures objectivity, and balanced conclusions and recommendations drawn from multiple sources. • The evaluation is both retrospective and prospective6 . • The evaluation did not focus on other donor or other USG civil society activities. • Recommendations mainly refer to the current approved project Work Plan and findings of this project process evaluation. Recommendations offered vis-à-vis any necessary course-correction or modifications of the project for the further implementation of the project are based on information gleaned and/or technical expertise possessed by the ET. Methodology and Data Collection Plan This evaluation was conducted in accordance with the USAID Evaluation Policy7 so that reliable data would be produced to support evidence-based findings, be a sound basis for analysis that would lead to conclusions and recommendations that are useful for assessing the progress to date while providing support for recommendations. Upon analysis of the project logic, the needs and expectations from the MTE by USAID and the implementing partner, “mixed methods” were employed to optimize the potential of the analysis and to reach sound conclusions. These methods included key informant interviews, mini-surveys, focus group discussions, as well as a document review, and briefing meetings with USAID and project staff (See Annex IV for Data Collection Instruments). This approach allowed for the verification of the findings through triangulation. The ET defines triangulation as a method used during the evaluation process to check and establish validity of conclusions by analyzing the responses to evaluation questions from multiple perspectives, such as interviews, surveys, and documents to confirm findings. This enables evaluators to ensure the validity of conclusions, i.e. that the findings of assessment are true (accurately reflecting the situation) and certain (supported by the evidence). The evaluation was carried out in four phases: 1. Inception phase and the document review: The methodology and evaluation instruments were developed during this phase. The ET reviewed the program quarterly reports, as well as the annual work plans and PMPs. Additional documents reviewed included the EWMI/G-PAC 6 Retrospective and Prospective – these terms describe the action of reviewing and taking into consideration the history of the project vis-à-vis the requirements of the mid-term evaluation (retrospective) and the application of that knowledge and understanding towards the analysis and assessment of the program going forward (prospective), in the eyes of the ET. Based on an acknowledged understanding of the genre (civil society development) and a learned appreciation of G-PAC, this type of appraisal aligns itself to the recommendations requested in the SOW for the assignment. 7 The evaluation will meet or exceed the USAID 2011 Evaluation Policy, USAID’s Performance Monitoring & Evaluation TIPS: Data Quality Standards, relevant chapters of the USAID Automated Directive System (ADS), the performance standards outlined in the RFTOP Task Order SOW, as well as with the Federal Policy for the Protection of Human Subjects or "Common Rule" [ADS Chapter 200 - http://www.usaid.gov/policy/ads/200/200mbe.pdf]. The IBTCI team is also apprised of the July, 2012 supplement to ADS 203 on current good practice in preparing evaluation reports and the main deliverables expected in evaluation reports. 3 Grants Report, CSO Advocacy Capacity Assessment Report, the University Assessment Report, and the Report on Citizens Attitudes Towards CSOs and Civic Activism in Georgia. Reviewing these documents helped define the key stakeholders to be interviewed, as well as questions that were to be asked in the surveys and key informant interviews (See Annex V for a bibliography). 2. Fieldwork phase: The fieldwork included travel to and within Georgia undertaken by the ET over 14 working days in September 2012 (see Annex VI for the In-Country Schedule of Meetings). The ET conducted interviews, mini-surveys and focus groups with a range of stakeholders from the civil society sector, the G-PAC Partners in Tbilisi and in the field, and similar. • Key Informant Interviews. Thirty-five formal, in-depth, albeit semi-structured interviews based on a pre-decided set of guiding questions were carried out with a wide variety of stakeholders including USAID and EWMI, CSO grantees, TTs, and universities. In addition to interviews in Tbilisi, the ET travelled to Gori, Kutaisi, and Rustavi to conduct interviews with Society Biliki, the Club of Invalids Gori, the Association of Young Economists Georgia, Georgian Young Lawyers Association, New Civic Thinking, and CiDA. A complete list of all G-PAC interviews can be found in Annex VII. • Mini Surveys. The two mini-surveys, implemented by IBTCI’s subcontractor, ACT, were based on a carefully designed set of instruments that allowed respondents to provide information and share views and opinions while remaining anonymous. The surveys were tested and adapted to the needs of the research by the local survey firm. The purpose of mini surveys for this evaluation was to reach a majority of partners and beneficiaries of the G-PAC project, particularly those with whom the ET was not able to meet personally during the limited field phase. The mini-survey with grantees was conducted by phone, while the mini-survey in Kutaisi was conducted through face-to-face interviews. Both mini survey reports can be found in Annex VIII. The timing of the surveys coincided with the peak of the pre-election period and the tense political moment in Georgia, affecting the extent to which the civil society organisations could afford time to participate in the evaluation process. o The Mini-Survey of Grantees was based on a list of grantees provided by G-PAC. From that list, 75 grantees were identified and of them 28 were able to provide responses, giving a response rate of 37.3%. o The respondents of the Mini-Survey for Kutaisi community were selected from a list provided by the G-PAC partner, Young Economists Association of Georgia, implementing the Project “Protect the rights of property owners residing near the Zestaponi-Samtredia highway construction site”. Because an in-depth survey including the “universe” of beneficiaries of the Project was not feasible, the ET decided to select three communities in the Kutaisi region and respective project beneficiaries in these three communities as a sample. The communities were Kvachakhi (60 respondents), Geguti (53 respondents) and Mukhiania (23 respondents), making up the total sample of 136 persons who were reached by the evaluation. The mini-survey had a response rate of 75%. • Focus group discussions. Six focus groups were conducted during the course of the field phase using a set of guides for each stakeholder group. The focus group questions were designed to be open￾ended to acquire ideas and opinions that are wider than the scope of the prepared questions. The ET conducted three including Society Biliki and the Club of Invalids Gori, AYEG, GYLA, and New Civic Thinking in Kutaisi, and Two Interns in Kutaisi. ACT conducted three focus groups including MPA students; Mentors; and CSOs active in the Batumi region. The selection of participants for the focus groups conducted by ACT was based on the need to reach out to a broad range of G-PAC beneficiaries program that the ET could not reach due to limited time. (See Annex VIII for the report on the focus groups conducted by ACT). 3. Analysis phase: The ET conducted ongoing analysis of the information and data collected. They discussed and compared notes at regular intervals throughout the evaluation. Conclusions 4 and recommendations were based on triangulation of the document review, responses to the mini-surveys, focus group analysis, and discussions and interviews with stakeholders. 4. Writing the evaluation report: The evaluation report was based on the data gathered in the field and revised based on comments received from USAID. Limitations to the Evaluation Design and Plan and How They Were Managed There were two overall limitations: 1) the evaluation occurred during the pre-election period of the parliamentary election so CSOs were not able to provide the ET with requisite attention; and 2) the length of the fieldwork as prescribed in the SOW was limited resulting in rushed work products. Other limitations and weaknesses are as follows: Key Informant Interviews o Limitation/Weakness: Although interviewees were carefully selected based on the desk review and USAID and G-PAC confirmation of our the list of interviewees, there is bias in some of the findings by the very nature of the relationship between grantees and the program, e.g., all CSOs and TTs interviewed had positive things to say about G-PAC trainings. o How it was managed: The ET interviewed 35 key informants within the allotted evaluation period to increase confidence in the validity of the evaluation findings. The data collected from the interviews was triangulated with data collected through desk review, focus groups and surveys. Focus Groups o Limitation/Weaknesses: The main weakness of the focus group effort was the limited availability of potential focus group participants to attend the focus groups. o How it was managed: The weakness was managed by ensuring that both IBTCI and ACT had sufficient flexibility to adapt the timing and venue of the focus groups to the availability of participants. Mini-Surveys o Limitation/Weakness: inherent weaknesses of the mini-survey design are primarily the selection and argumentation of reasons why a certain sample was selected. Closely linked to that is potential risk that sample selected may not be representative of the wider group. o How it was managed: IBTCI managed the potential risk of small or inappropriate survey samples by deciding that all of the grantees that have participated in G-PAC (from the project the start until July 2012) would receive the survey. However, during the survey process, only 37.3% of grantees responded, a relatively low response rate. This remained the single most important limitation of the research process within the G-PAC evaluation. The potential limitation of the survey conducted in the Kutaisi region was the response rate of persons included in the sample. ACT managed this potential risk by visiting each target community and conducting face-to-face interviews. IV. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS The report is organized by evaluation question and for each question there is a summary of findings that provides an overall answer to the question. Following the summary of findings, there is analysis based on particular findings about activities for that question. Following the findings and analysis, conclusions and recommendations are provided. Evaluation Question #1 How effective are G-PAC’s interventions to strengthen advocacy and watchdog roles of targeted Tbilisi-based and regional CSOs? Summary of Findings & Answer to Evaluation Question #1 G-PAC’s interventions to strengthen advocacy and watchdog roles of targeted Tbilisi-based and regional CSOs are effective. There are some interventions that are more appreciated than others, such as mentoring and individual consultations, but the whole package of interventions received, when at their best, mean that CSOs and TTs are receiving a complete package of training and 5 technical assistance that addresses program and organizational development. Evidence indicates that CSOs have a better understanding of advocacy and what it means to engage in advocacy activities, and they are better equipped with knowledge, skills, behaviors, and relationships necessary to conduct more effective advocacy. Most all of the CSOs and TTs who have received G-PAC technical assistance have achieved positive outcomes with their advocacy campaigns. G-PAC is meeting or exceeding its PMP goals. Background and Activities Conducted During Year 1, G-PAC provided technical assistance and funding to the Association of Young Economists of Georgia (AYEG) to conduct the CSO Advocacy Capacity Assessment in Georgia based on the USAID Advocacy Index methodology. AYEG conducted face-to-face interviews with 100 CSOs and facilitated focus groups with citizens throughout Georgia. G-PAC’s training program, as explained in detail below responded to the needs of a broad range of Georgian CSOs, both national and regional, identified through the assessments process. G-PAC’s technical assistance for Year 2 evolved based on continual assessments of capacity development needs of G-PAC beneficiaries through the following mechanisms: (i) findings of the Year 1 Think Tanks Assessment and CSO Advocacy Capacity Assessment; (ii) suggestions for capacity development assistance identified by CSOs in their application forms for G-PAC grants; (iii) the pre-training assessments carried out in preparation of Year 2 training activities and the subsequent post-training participant evaluations of these activities; (iv) feedback from international and Georgian mentors; and (v) findings from the national public opinion poll on CSOs.8 The comprehensive package of interventions, including workshop-style trainings, tailored consultations, mentoring, and grant assistance was provided to all awardees and was based on the needs of each CSO and think tank. Findings and Analysis – Training Workshops9 During Year 1 and Year 2 the following trainings were provided: • Mid‐Level Advocacy Skills Training for 59 representatives from 45 CSOs • Advanced Advocacy Skills Training fir 39 representatives from 33 Tbilisi‐based CSOs • Trainings on the three main thematic areas of Effective CSO Management including: o Research Toolkit for CSOs delivered to 17 representatives from 17 CSO; aimed to strengthen and build basic research capacity of CSOs and enable them to more effectively plan and incorporate research within their ongoing projects. o PR/ Public Outreach skills delivered to 14 representatives from 14 CSOs; aimed to demonstrate the significance of the PR process for supporting public dialogue among major actors and highlight the importance of effective communication with target audiences. o Negotiation skills to 16 representatives from 16 CSOs to improve participants’ skills to negotiate successfully while advocating for their causes.10 G-PAC anticipated training 90 CSO/think tank members for each of the first two years of the project. After two years, G-PAC reported training 231 CSO members, exceeding their goal of training 180 members.11 CSOs interviewed individually and in focus groups recognized that trainings were needs-based and useful. The lessons learned shared in the trainings were also helpful and relevant to their respective organization’s advocacy campaign. Also, CSOs found trainings and consultations useful because they now have the capacity to design and conduct surveys, which they did not have prior to G-PAC. 8 G-PAC Year Two Work Plan, September 1, 2011-August 31, 2012 9 The sources of information for the findings are included in footnotes categorized by document review, key informant interviews, focus groups, or surveys 10 G-PAC CSO Capacity Building Assessment Year One and Year Two, Partners-Georgia 11 G-PAC PMP Year One and Two, 2010-2012 6 At that time they had to hire consultants or experts to conduct.12 Seventy-one percent of CSOs interviewed mentioned that they have better skills to conduct research, 64% cited that numbers of produced research increased and 71% said the quality of their products have improved.13 However, some indicated that training modules did not reflect current needs and suggested they be updated.14 CSOs now believe they have more capacity to influence the policy making process evidenced by the survey results showing the following: 64% of the interviewed CSOs think that their advocacy initiatives are better designed; 79% are sure that advocacy messages are concise, objective and clear; and 82% cited that their advocacy activities brought some concrete results. Some of these concrete results include: construction of ramps to provide access to buildings for people with physical disabilities; re-registration of property to protect owner’s rights; the addition of a community library and potable water filtration systems; and specific changes in local ordinances or legislation.15 Even after the training, some think tank representatives believed that they do not need formal trainings because “they passed this stage.” Others were not against having trainings but indicated that being trained is not a high priority for them.16 All CSOs provided positive feedback about G-PAC’s “Policy Analysis” trainings. For the Think Thanks (TTs) it was very interesting to learn of the policy making practice in Eastern European countries. Furthermore, they mentioned that Mr. Krassen Stanchev (Chairman of the Board of the Institute for Market Economics in Sofia), one of the international trainers, was more effective as a mentor than the trainer that was provided.17 CSOs expressed their gratitude to G-PAC for teaching and coaching them to use social media for advocacy purposes; to develop web-pages for their efforts;18 and even to re-brand their organization - 64% of interviewed CSOs think that contributed in improved visibility of their organization, stating that after G-PAC interventions, they are now more known and respected in society.19 “We were also assisting them in the conduct of an advocacy campaign, writing press releases, proofreading tests and in networking with the local government. We were providing assistance on every level.” Findings and Analysis – Tailored Consultations In addition to the trainings, Partners-Georgia provided tailored consultations to the CSOs. G-PAC selected 10 regional CSOs in different parts of Georgia to participate in advanced on-the-job and online consultations in four subject areas: PR and Public Outreach (on-the-job and online consultations), Research Toolkit for CSOs (on-the-job consultations), Negotiations (on-the-job consultations) and Report Writing (online consultations). According to stakeholder interviews, the trainings followed by tailored consultations proved to be very useful to the CSOs overall, a sentiment underscored in a focus group discussion by the Batumi￾based CSOs. 20 Specifically, CSOs mentioned that their project interventions have been improving because of the addition of the consultation component. Moreover, according to those interviewed, the introduction of consultations increased the effectiveness of the training workshops,21 namely the information provided on how to use social media for advocacy purposes. Also, interviewees noted that the consultations built upon previous trainings so it was essential that the same people (and organizations) attended the training and the subsequent consultations. 12 Key Informant Interviews, September 2012 13 G-PAC Grantees Survey Report, ACT, October 2012; During KIIs, it was learned that many stakeholders conducted surveys and interpret surveys as their improved research skills. 14 Batumi CSO Focus Group Report, G-PAC Grantee Survey, ACT, September/October 2012 15 G-PAC Grantees Survey Report, ACT; Key Informant Interviews, Document Reviews, September 2012 16 Key Informant Interviews, September 2012 17 Key Informant Interviews, September 2012 18 Key Informant Interviews, September 2012 19 G-PAC Grantees Survey Report, ACT, 2012 20 Batumi CSO Focus Group Report, ACT, September/October 2012 21 Key Informant Interviews, September, 2012 7 Findings and Analysis – Mentoring for Think Tanks and CSOs G-PAC engaged 11 Georgian experts who provided on-site and long distance mentoring to 16 G￾PAC CSO grantees on a wide variety of issues to advance the implementation of their advocacy campaigns. Mentors provided CSOs with on-the-job and long-distance training, advice, and information in relation to the advocacy initiatives supported by G-PAC awards. Mentor assistance included the following issues: advocacy tactics to maximize impact, constituency support, and effective advocacy messaging using IT tools, media presentation skills, reviewing press releases, and preparing for meetings with decision-makers. Additionally, leading United States (US) and Central and Eastern European (CEE) experts provided mentoring to seven Georgian TTs and assisted in elaborating their research and advocacy plans, identifying areas for improving their project proposals, addressing management challenges, developing plans to build think tank sustainability beyond foreign funding, and fostering strategies for increasing public interest in their work. The Center for American Progress (CAP), the Brookings Institute and the Heritage Foundation, each provided mentoring services to the Georgian TTs, and provided assistance to them for policy reform strategies.22 For example, EPRC was mentored by professors from Columbia University in New York, who provided, in part, review, comment, and recommendations on draft papers on economics and health care issues; and the Liberal Academy of Tbilisi was mentored by a senior staff of the Brookings Institute, who also provided critique of draft research papers.23 Stakeholder interviews revealed that having a mentor was a preferred capacity-building intervention to provide learning opportunities and build connections and future partnerships and that this G￾PAC intervention was effective.24 G-PAC anticipated providing mentoring services to a total of 20 CSOs during year one of the program, and 30 CSOs during year two. The progress to date indicates 23 CSOs received mentoring during year one, and 13 for year two.25 TTs consider G-PAC, along with other donor organization and clients (government, business, and advocacy organizations), contributors to the creation of their research and/or advocacy agenda. However, TTs also realize that G-PAC grants increased the amount and quality of research they produced and accordingly their visibility (please see grants program below). According to mentors who participated in focus groups, the assistance they provided was needed not only with those substantive issues identified in their advocacy campaigns, but with other organizational issue as well.26 Mentors provided additional information and resources to the CSOs in best ways to work with the media; reviewing media releases or research topic ideas; communicating with their constituencies; establishing new contacts, and developing resources and funds to pursue efforts.27 “Trainings and consultations conducted by “Partners Georgia” were great. Our concrete problems were discussed and recommendations were given” (Organization “Borjgali”.) It was observed that there may be limited awareness of the different G-PAC project activities among different actors, which may have made for challenges in providing appropriate support to the CSOs. For instance: o CSO mentors had a limited understanding of the training programs CSOs were provided by G￾PAC and trainers had similar lack of understanding regarding the services that the mentors were providing; o Network facilitators had little or no information on how the training program was designed and based on which criteria mentors had identified for CSOs.28 Representatives of the Civic 22 G-PAC Year Two Work Plan, September 1, 2011-August 31, 2012 23 Key Informant Interviews, September 2012 24 Mentor Focus Group Research Report, pages 20-28, ACT September/October 2012 25 G-PAC PMP Year One and Year Two, 2010-2012 26 Op cit 27 Key Informant Interviews, September 2012 28 Key Informant Interviews, September 2012 8 Development Institute mentioned that three times they published their policy papers and after a two-to-three week period a government ministry published its strategy widely using CDI’s information and approach, but did not give the organization credit for the work. 29 Although the ET understands the frustration this may cause the CSOs, it is important to learn that government officials are using the research and positions that are being promoted by the CSOs. At some level, this is an example of successful persuasion and advocacy in that the government not only accepts but furthers the position of the CSO. Findings and Analysis – Study Tours G-PAC organized study tours for its grantees (both CSOs and TTs) to Slovakia and Estonia to enable the participants to learn from experiences in similar conditions; explore avenues for further cooperation between Georgian and CSOs from other countries; and meet with host country’s past and current public officials to discuss their experience in advancing social-economic reforms.30 According to stakeholders interviewed, the study tours were beneficial because they exposed the participants to new ideas and creative solutions to common problems.31 For Slovakia, those interviewed said it was an “eye opening” opportunity for them. Especially interesting was meeting with Slovak CSOs and comparing similar conditions to those existing in Georgia.32 According to the CSOs of special note was learning about the Slovak models of CSO sustainability. TTs found the study tour very useful, because they too met CSOs and had an opportunity for experience sharing and building future partnerships. EPRC mentioned that the study tour to Slovakia appeared extremely useful for the organization since they had an opportunity to identify a Slovak partner Institute of Economic and Social Studies (IESS) and together they began implementing the “Price of the State” project. In addition, representative of IESS became a mentor for EPRC.33 Findings and Analysis – Grant Making Programs Citizens Advocacy Grants (CAGs): CAGs support issue-based advocacy initiatives undertaken by Georgian CSOs at local and regional levels, with priority given to CSOs based in the regions. Grantees implement creative participatory activities, and engage citizens in advocating for policy reforms. As of August 2012 40 CAG Grants were awarded for a total value of $721, 233.34 According to G-PAC PMPs, the overall grant-making aspect of the program is exceeding their goals of 40 grants awarded per year.35 Act Now Grants (ANG): ANGs offer support for strategic, targeted advocacy initiatives that respond to highly important and unexpected political, economic, and social concerns. As of August 2012 two Act Now Grants were awarded, for a total value of $8,417.36 During the second year G-PAC introduced a two-tiered grant application process, where previously successful grantees could apply for further funds in a limited competition in one tier and other CSOs could apply in an open competition in a second tier. In this approach G-PAC rewards successful grantees by offering larger grant amounts to support the continuation and evolution of successful advocacy campaigns. Arts for Advocacy Grants (AfA): AfA grants fund was an innovative engagement of the artistic community to support advocacy causes. G-PAC supports artists who use their craft - dance, theatre, film, music, poetry and/or visual art – as a medium to carve out new approaches to challenging social issues. However, as of August 2012, only five grants were awarded, with total value $45.109.37 29 Key Informant Interviews, September 2012 30 G-PAC Eighth Quarterly Report, April-June 2011 31 Key Informant Interviews, September, 2012 32 G-PAC Eighth Quarterly Report, April-June 2011 33 Key Informant Interviews, September 2012 34 G-PAC Grants Report, 2011-2012 35 G-PAC PMP Year One and Year Two, 2010-2012 36 G-PAC Grants Report, 2011-2012 37 G-PAC Grants Report, 2011-2012 9 This grant activity did not get adequate interest from CSOs and Art people. CSOs demonstrated limited awareness of this grant.38 CSOs told the ET that G-PAC CAG brought them closer to beneficiary communities and made them more aware of their needs. Under the framework of the CAG they had to identify community priority needs for advocacy purposes. It was learned that the long and continuous discussions with the communities made CSOs more aware of the problems communities are facing. CSOs together with the communities began to identify the ways to address those problems.39 Fifty-seven percent of interviewed CSOs emphasized that through the G-PAC Grants program they managed to improve their relationship with government. Thirty-six percent of the grantees mentioned the authorities more often ask for their advice. CSOs linked these changes to their activities which brought some concrete results in the community.40 For example: • Batumi-based CSO “Step Forward” mentioned that they were pleasantly surprised by the level of cooperation they got from central and local governments during the implementation of the G-PAC funded project aiming to advocate the rights of children with hearing disabilities.41 • Batumi based CSO “Borjgalo” remarked, “Communications on the local level is much easier as local governments are closer to society. Local governments understand the importance of the issues for which CSOs are advocating, but because of limited responsibilities and resources the local governments could not respond.”42 • An excellent example of local government cooperation is from the project implemented by the Club for Gori Invalids. The advocacy campaign to make public buildings accessible for disabled people received attention from the government and adequate actions were taken.43 CSOs believe that their relationship with media is gradually improving as 86% of the surveyed CSOs mentioned that media follows their work and often invites them to the organized events. Some NGOs provided special trainings to media representative in their respective fields. For instance, CSO “Borjgali” conducted a special training for journalists for improving their understanding in the field of ecology and for explanation of definitions of basic terminology. CSO Borjgali was thrilled with the outcome of the training, since journalists prepared very interesting reports and a number of articles on this particular subject.44 However, some CSOs cited that their relationship with media remains challenging in the regions. Local TVs will not broadcast any information for free unless it is considered “newsworthy,” and/or “scandalous.” CSOs must pay for media time to send messages to their target communities because media considers this as paid advertising for CSOs. CSOs have no problem to use print media for their advocacy purposes, but there is limited circulation of newspapers and very limited interest of the population in print media.45 There were a number of reasons cited by CSOs for the low number of applications for the AfA grants. Those reasons include: low budget and too much paperwork for more experienced CSOs and difficult procurement procedures for the less experienced CSOs; and limited awareness of these grants among all CSOs. 46Thirty-nine percent of CSOs surveyed indicated that one of the main challenges they face is the inability to access current and timely information about the purposes of the ANG and AfA grant.47In addition, some of them mentioned that this very new and creative G￾PAC initiative challenged their way of thinking about grants. Most of the targeted CSOs are used to designing and implementing standard grants, and it was hard for them to imagine how art or artistic 38 Key Informant Interviews, September 2012 39 Key Informant Interviews, September 2012 40 G-PAC Grantees Survey Report, ACT, October 2012 41 G-PAC Grantees Survey Report, ACT, October 2012 42 Ibid 43 Key Informant Interviews, September 2012 44 G-PAC Grantees Survey Report, ACT, October 2012, The ET did not meet with media representatives directly. 45 Key Informant Interviews, September 2012 46 Key Informant Interviews, September 2012 47 G-PAC Grantees Survey Report, ACT, October 2012 10 expression could be used to encourage provocative public debates on pressing issues. According to some CSOs, they simply lacked the creativity required to apply for this award. Also some of them mentioned that it was very challenging for them to attract artists and people involved in the artistic community, because they had no established relationships with them.48 TTs voiced their satisfaction regarding the grants program since the grants gave them an opportunity to work on nationwide policies and to voice community interest in front of the Government.49 Interviews with TTs (EPRC, IAG, LAT) indicated that they consider themselves to be strong, visible and enjoying a high reputation even before G-PAC interventions. However, they also indicated that G-PAC made them much better and contributed a lot in their capacity development, visibility, reputation and cross-sector collaboration.50 Both CSOs and TTs welcomed the G-PAC initiative of introducing a two-tiered grant application process in Year 2. The initiative provides an opportunity for previously successful grantees to apply for further funds in a limited competition in one tier and other CSOs could apply in an open competition in a second tier. TTs commented that this is important because most activities of the advocacy campaigns could not be completed within the life of the first award, and therefore knowing that a follow up award is available motivated the TTs to bring their first award to a successful, albeit incomplete. 51 Advocacy Activity Results: As a result of the advocacy activities of G-PAC’s Citizens’ Advocacy Grant (CAG) and Partnership for Change (PfC) grantees: 1) 15 households will be compensated for property rights violations; 2) 18 households living below the poverty line received state welfare benefits; 3) 29 households living below the poverty line received legal assistance; 4) 7 amendments were made to local government budgets; 5) 2 joint CSO – government healthcare initiatives were launched to meet the needs of children with autism; and 6) Park and forest areas in and near Tbilisi were protected from deforestation and construction.52 All implemented projects demonstrate the improved ability of the CSOs to engage in, suggest and impact policy changes in their respective communities, as both observed by the ET as well as having been reported in the CSO survey.53 CSOs observed that G-PAC’s interventions improved their relationship with the government and with local communities. For example, the Club of Gori Invalids led a successful advocacy campaign to create accessible building environments so that people with physical disabilities can easily access buildings. This was an issue that was, and remains, important to the community served by the Club. The Club, which has a long-standing relationship with the local municipality and authorities, was successful in getting some building to be barrier-free, and this was, in part, due to their good relationship with local decision-makers.54 During the process of conducting advocacy activities, CSOs indicated that they became more aware of the distribution of responsibilities and available resources between central and local governments, and they increasingly understand that to influence the government they should have a deep understanding of the issue and ability to provide relevant recommendations for improvement.55 At the same time, and just as critical, CSOs indicated that they became closer to beneficiary communities and more aware of their needs. Examples demonstrated that issues that CSOs advocate 48 Key Informant Interviews, September 2012 49 Key Informant Interviews, September 2012 50 Key Informant Interviews, September 2012 51 Key Informant Interviews, September 2012 52 G-PAC Eighth Quarterly Report, April-June 2011 53 G-PAC Grantees Survey Report, October 2012 and Key Informant Interviews, September 2012 54 Interview with the Club of Gori Invalids, September, 2012 55 Key Informant Interviews, September 2012 11 for are priority needs of the communities they represent and thus, have broader support from the respective population/target groups.56 One example of a priority issue identified by a Kutaisi community involved tension between government and private citizens over land rights. According to a community survey conducted by ACT, in general, after the public awareness and education campaign on land rights began, it was found that most people knew that they had to re-register their land in order to receive just compensation for the government’s taking of it. Eventually, most did re-register. Most heard about the issue through the campaign process and for those who eventually re-registered their land, compensation was received. More than half of those surveyed found the campaign helpful in terms of creating awareness for the issue, and for ultimately receiving the compensation.57 Twenty-eight G-PAC grantees were interviewed as to their evaluation of the results achieved by their organization since they began with G-PAC support. Baseline data was not available, and interviewees were asked to rate their progress toward the objectives of the grants. The most positive result achieved by the organizations according to the survey of participants is assisting/inspiring people to address their issues. G-PAC Grantees Evaluation of the Results Achieved Yes, definitely Yes, a little Average No change No, it is worse N/A Don't know We have inspired/assisted people to take action to address their issues 79% 7% 4% - - 4% 7% The authorities have considered our suggestions 64% 7% 4% 11% - 7% 7% The government law/policies/services have improved 39% 14% - 25% 4% 14% 4% The situation of our target group has improved 61% 18% 4% 11% - 4% 4% The majority (79%) of CSOs surveyed declared that G-PAC is supporting their organizations in mitigating the risks and the challenges of their organization’s advocacy project, which is an important consideration in addressing key question #1 According to the survey, risks and challenges to the CSOs include: accessibility of resources from donors and accessibility of information from Tbilisi to the regions. Other challenges include: fluctuation in staff members; legal barriers; and lack of knowledge in using new technologies.58 The graph below summarizes the data above on mitigating the risks and challenges by G-PAC. 79% 14% 7% Yes No Don't know When asked what improvements within their organization have been made since receiving G-PAC support: 82% stated that they are more focused on what they want to achieve; 79% stated that their advocacy messages are more concise and objective; 79% stated they have a better understanding of priorities and interests of the target audiences; 82% stated that their advocacy activities resulted in concrete actions; 79% stated they have more support from both members as well as the communities in which they work; and 79% stated they have better cooperation with other CSOs.59 56 Key Informant Interviews, September 2012 57 Kutaisi Community Survey, ACT, September/October 2012 58 G-PAC Grantees Survey Report, ACT, October 2012 59 G-PAC Grantees Survey Report, ACT, October 2012 12 Conclusions Overall Conclusions • Projects implemented by G-PAC’s Grantees are demonstrating that it is possible to influence policy-making if CSO/TT recommendations are realistic, objective, timely, and competent. • There may be insufficient understanding among the CSOs about the concept of “advocacy” and the role of routinely produced high quality evidence based research in this process. • G-PAC’s interventions increased the visibility and reputation of CSOs and expanded and strengthened their social networks. However, more is to be done to improve relationships with the government, become closer to the target communities, use media and social networks for advocacy, and give sufficient attention to the relationship with the business sector. • G-PAC grantees receive capacity building benefits from the program, yet there may not be effective mechanisms for sharing information and experiences between the CSOs do and do not participate in the G-PAC Program. Technical Assistance • G-PAC’s Capacity Building Program is comprehensive in that it addresses the needs of CSOs using a variety of interventions. The design and provision of each intervention under the program is uniquely tailored to the needs of each CSO and TT. • G-PAC’s Capacity Building Program strengthened advocacy and research production skills of CSOs and TTs,60 though they prefer practical capacity building programs like: consultations, mentoring and study tours to the formal trainings. • Research often means collecting information from secondary sources; very few organizations develop original research to fill the existing information gaps. • CSOs and TTs do not have a system or criteria to measure quality of their products. • Training modules and training materials should be brought up-to-date to ensure grantees are meeting requirements of the European and US donors in report writing and proposal drafting. Grants Program • G-PAC CAG and TTSG grants appeared effective having demonstrated concrete policy results on the both local and national levels. The criteria the ET used to determine effectiveness of grants included: CSOs conducting surveys and/or research to determine priority issues; CSOs defining and framing the advocacy issue; CSOs developing a plan of action or approach to their campaign; CSOs taking concrete steps in pursuance of the goal of their advocacy campaign; CSOs attempt to develop a broad-base of support for their issue; CSOs attempt to negotiate with government officials; CSOs attempt to work with media to inform broader public and/or to persuade public, stakeholders, and decision-makers. • G-PAC ANG and AfA Grants appeared less attractive for the CSOs, because of lack of awareness and understanding of the purpose of the awards. There were a number of reasons cited by CSOs for the low number of applications for the AfA grants, which were not expressed for the other grant categories. Those reasons include: low budget and too much paperwork for more experienced CSOs and difficult procurement procedures for the less experienced CSOs; and limited awareness of these grants among all CSOs. Cross-Collaboration • Improved/strengthened relationship of CSOs and TTs with the government --TTs mainly with central government and CSOs mainly with local government. • Improved/ strengthened relationship of CSOs and TTs with media. TTs have no problem to use media for their advocacy purposes; however, CSOs still are facing problems with regional media representatives. • Improved/strengthened relationship between CSOs and respective communities. Now they are more integrated and tied into communities. 60 This conclusion is reached based on the interviews with the grantees; the ET was not charged with evaluating the actual changes in the organizational strength of the CSOs as one would do in an impact evaluation. 13 • Limited relationship between CSOs, TTs and business sectors. Only very few TTs managed to maintain good collaboration with the private sector. Recommendations Technical Assistance • G-PAC should consider placing more emphasis on strengthening, expanding and shaping mentoring and consultations, since they are more flexible and well-tailored to the needs of grantee CSOs. • G-PAC should consider placing more emphasis on funding research organizations that are producing original research. “The internship program was extremely useful, as it was about the analysis of everything we had learned and I had an opportunity to put everything into practice.” • G-PAC should consider supporting the establishment of a peer review culture in Georgia. • G-PAC should emphasize attention to institutional strengthening of the CSOs, particularly in areas leading to long-term sustainability and independence of CSOs. Such activities could include: drafting program concept papers, drafting proposals; conducting strategic planning exercises to identify organizational needs and priorities; and volunteer recruitment and leadership training. Grants Program • Through the network facilitators increase awareness of the CSOs on ANG and AfA grants and with the participation of CSOs improve or re-design this grant award program to make it more appealing to CSOs. Cross-Collaboration • Support CSOs and TTs to maintain and improve their relationship with media, government and communities and to establish relationships with the private sector. • G-PAC should encourage CSOs be more integrated and tied to communities. • G-PAC should encourage CSOs become better aware of the distribution of responsibilities and available resources between central and local governments and accordingly to make their advocacy campaigns more realistic. • G-PAC should encourage effective mechanisms for sharing information and experiences, and for promoting possible cooperation between: ƒ the different program components in order that key trainers and mentors have a comprehensive understanding of the program goals and objectives, and accordingly, their specific roles in achieving them; ƒ projects working on the development of civil society, like Project Harmony and MSI who are working with civil society, local and central governments and using similar approaches for their project design and implementation as G-PAC; and ƒ CSOs that are part of G-PAC, as well as with CSOs who are not part of the G-PAC program for the amplification of the project results. • G-PAC should encourage CSOs create a two stage policy dialogue between civil society and the government. First to increase awareness of civil society on a specific policy issue and prepare its representatives for meaningful dialogue with the government sector. Second to create a neutral space for policy discussions involving a wide group of stakeholders, including the government, where civil society could demonstrate deep knowledge of the issue and propose relevant and realistic recommendations. The closest the ET observed this taking place within G-PAC was with the consortium with LAT, TI, and EPRC, led by EPRC, to bring political parties to a public forum to discuss their policy positions on issues. Similar types of forums should be organized by CSOs that would discuss important issues where ultimately suggestions for addressing the issues would be proffered and perhaps used by political parties for their own positions, or by TTs or CSO or even government officials. Best Practices • Innovations in Applying for Grants: The G-PAC Program has remained consistently flexible in the grant application process in order to encourage applications from a wide range of CSOs. 14 The use of Skype to conduct application discussions and the use of video submissions are creative means for encouraging CSOs who have varying capabilities. • Mentoring and Consultations: Mentoring and tailored consultations have proven perhaps the most effective means of supporting incremental development of CSOs. • Study Tours: The use of study tours and other activities to expose CSOs to other ideas, models, and examples is critical to their growth and development. Evaluation Question #2 How effective are networks of CSOs supported under the G-PAC program? To what extent did project interventions result in strengthening their connections with each other, media, political parties and constituencies? Summary of Findings and Answer to Evaluation Question #2 The CSOs networks supported under the G-PAC program are proving to be effective in terms of conducting successful advocacy campaigns, but are less effective in terms of internal management and having clear roles, responsibilities, and expectations of their members. CSOs and TTs believe in the strength and successes coalitions can bring, and critical connections with decision-makers and others are being established and maintained; but CSOs are skeptical about coalitions and are hesitant to join them. G-PAC is meeting its PMP goals. Background and Activities Conducted G-PAC identified local partner organizations who managed the role of the networking facilitators. The two CSOs selected to serve as networking facilitators respectively for West and East Georgia – Association ATINATI and the Civil Development Agency (CiDA) – worked to develop coalition￾based advocacy campaigns and encourage networking and coalition-building at the regional level. The networking facilitators are responsible for (i) organizing and managing meetings with NGOs in their respective region(s); (ii) assessing the capacity of existing NGO networks in the respective region(s); (iii) assisting NGOs to establish new broad advocacy coalitions; (iv) facilitating cooperation among NGO coalitions, TTs, and mentors in developing and implementing an advocacy campaign plan; and (v) organizing “Meet Your Government Representative” and other public forums; and (vi) assisting in identifying social and economic issues at the forefront of public interest in the respective region. The G-PAC Program also has a grant program specifically encouraging CSOs to work together and form consortiums or coalitions. (More below) Findings and Analysis As of August 2012 G-PAC and its two networking facilitators, ATINATI and CiDA, conducted two meetings with Georgian regional CSOs to generate new ideas. Meetings were held both in West and East Georgia (Zugdidi and Tbilisi). G-PAC, ATINATI, and CiDA also organized meetings as part of the G-PAC “You Have the Right to Know” information campaign; five were held in total, three in West Georgia (Kutaisi, Batumi, and Zugdidi) and two in East Georgia (Telavi and Rustavi).61 Additionally, ATINATI and CiDA organized consultations for CSOs. The consultations ATINATI provided included guidance regarding G-PAC grant programs. ATINATI also organized a roundtable discussion with regional CSOs and media outlets to discuss specific mechanisms for improving cooperation and information exchange between these two sectors. CiDA’s consultations aimed to spread the knowledge and skills gained during the previous quarter’s G-PAC classroom trainings conducted by Partners-Georgia. CiDA attended all of these classroom trainings, and used Partners-Georgia’s materials to share its experience and new knowledge with other CSOs that had not attended the trainings. In May, CiDA provided advice and shared information with 12 regional 61 Interview with CiDA, September 2012 15 CSOs in Mtskheta, Gori, and Telavi. A total of 256 persons attended the meetings organized by ATINATI and CiDA.62 During stakeholder interviews, the ET learned that CSOs believe that networking increases CSOs’ influence, visibility of the burning issues being addressed by the coalitions. It is because of these outcomes that CSOs do place some value on working in coalitions.63 They also believe that because they are in a coalition, they are well positioned to build a relationship between media, government, and respective constituencies that will bring many more relationships to the table.64 Finally, CSOs believe that working in a coalition can provide more information, knowledge, and experiences for them.65 CSOs also believe, however, that there is hesitancy to join coalitions due to skepticism, and that coalitions can be much harder to work in because there may be a lack of clear roles, responsibilities, and coordination among its members.66 There are notable examples of cooperation between Tbilisi-based established thinks tanks with research capacity and regional CSOs, including IAG conducting research with regionally-based pensioner organizations. Through their experience, IAG believes it needs reliable partners in the regions who routinely provide information for their research, and in turn, regional CSOs need strong voices and connections in the Capital.67 According to a telephone survey with 28 CSO respondents, more than half believe it is much easier to advocate within the coalition and more than a third believe that results and policy changes can take place through coalitions. 54% 32% 11% 7% 7% 4% 4% 4% 0% 10% 20% 30% 40% 50% 60% It is much easier to advocate within… Results and policy changes happen… Better results are gained when CSO… In general the coalition is better In the form of civil organization Working in group is effective We are not in touch with advocating It depends on advocacy goal Grant-making Program The Partnership for Change (PfC) grant seeks to broaden Georgia’s civil society and increase its impact by facilitating linkages and coordination among advocacy CSOs, TTs, and academic institutions in Tbilisi and throughout the regions of Georgia. It also seeks to strengthen civil society’s connection with Georgian citizens by promoting direct interaction between civil society 62 G-PAC Sixth Quarterly Report, October-December 2011 63 Key Informant Interviews, September, 2012 64 Key Informant Interviews, September, 2012 65 Key Informant Interviews, September, 2012 66 Key Informant Interviews, September, 2012 67 Interview with IAG, September 2012 16 groups and their constituencies. PfC was designed for formal or ad-hoc coalitions or networks, or partnerships of at least two organizations established around an advocacy issue. To increase civil society’s impact on the public policy process, the PfC program facilitates linkages and coordination among advocacy CSOs, TTs, and academic institutions throughout Georgia. This program also promotes interaction between CSOs and their constituencies. It is designed for formal or ad-hoc coalitions, networks, or partnerships of at least two organizations established around an advocacy issue.68 As of August 2012, G-PAC awarded 12 grants to 10 Tbilisi-based CSOs advocating for issues related to education, the environment, elections, and healthcare. The total value of PfC Grants awarded: $610,001.69 According to G-PAC PMPs, the overall grant-making aspect of the program is exceeding their goals of 40 grants awarded per year.70 Conclusions • G-PAC’s interventions are strengthening CSOs networks’ role in policy-making; • Despite the existing skepticism CSOs and TTs increasingly understand the importance of the networking and coalition-building in advocacy and policy making; • Coalitions are issue-based and are created between strong research, advocacy and watchdog organizations; • Internal management of the coalitions need to be improved, in terms of the leadership, internal communication, cooperation, distribution of resources and responsibilities; • Coalitions support development of the member organizations through the natural flow of information, knowledge and experience. Recommendations • Encourage the continued development of coalitions, in terms of a needs-based capacity-building and grants program. Motivate coalitions to improve internal management in terms of leadership, delegation of responsibilities and resources, communication and coordination. • Encourage the creation of coalitions of advocacy organizations and TTs as well as coalitions of Tbilisi based thinks tanks/CSOs and regional organizations. • Develop case studies on coalition projects to demonstrate their advantages and disadvantages. Best Practice • Examples of Unlikely Coalition Partners: The Insurance Association of Georgia’s advocacy work is an example of a public-private partnership that could be built on and used as a model of cooperation between sectors. Additionally, IAG has partnered with local CSOs, including pensioner groups, to conduct research for future projects. Evaluation Question #3 How effective are G-PAC’s interventions to strengthen research capacity of targeted think tanks? Summary of Findings and Answer to Key Evaluation Question #3 G-PAC’s interventions to strengthen the research capacity of targeted TTs are demonstrating some effectiveness and are demonstrating positive results, but working with TTs remains challenging. There are examples of the think tanks conducting successful research, using their research for advocacy purposes, or conducting coalition advocacy with the research, but their level of engagement in the G-PAC Program and acceptance of their need for training and development remains a challenge. G-PAC is meeting their PMP goals. Background and Activities Conducted 68 G-PAC Third Quarterly Report, January-March 2011 69 G-PAC Grants Report, 2011-2012 70 G-PAC PMP Year One and Year Two, 2010-2012 17 G-PAC’s technical assistance to TTs aims to enhance their capacity to conduct high quality policy research; propose efficient public policy solutions; ‘package’ and ‘re-package’ their policy proposals to communicate them successfully to both the public and decision-makers; strengthen their governance structures; improve organizational management; and establish and uphold high standards of professional independence, ethics, and integrity. G-PAC, in cooperation with the Open Society Institute’s Think Tnk Fund (TTF) based in Budapest, Hungary, undertook an in-depth assessment of Georgian TTs mainly based in Tbilisi. The assessment report comprises: (i) an analysis of the current public policy processes in Georgia that includes the identification of entry points for larger public participation in policy making; (ii) an inventory of TTs’ capacity including human resources, focus areas to-date, geographic coverage, types of activities, and relations/partnerships in and outside Georgia; (iii) an analysis of the major challenges to Georgian think tank effectiveness; (iv) strategies for turning TTs into higher profile institutions with stronger policy relevance in public policy making; (v) recommendations for effective assistance activities that G-PAC could provide to these organizations, including identification of policy topics that G-PAC think tank Support Grants would support; and (vi) recommendations for coordinating the OSI’s TTF and G-PAC support for Georgian TTs in the next three years. In view of the challenges identified during the assessment, G-PAC designed the Think Tank Support Program, which through a combination of grants, mentoring, and study tours, assisted TTs: (a) identify the issues that are of vital importance to Georgians; (b) develop expertise of select TTs and researchers to conduct policy research in a relevant and sustainable way with help from Central and Eastern European TTs; (c) raise overall standards for policy research and analysis; and (d) generate government and public demand for such research products.71 Findings and Analysis Stakeholder interviews with TTs indicated they perform a variety of functions and perceive themselves in a variety of ways. Some conduct both research and advocacy; some conduct original research, while others work with secondary sources only; others conduct research and provide their research to advocacy organizations; some conduct fee-for-service work.72 In many cases TTs do not develop a research agenda. They conduct research based on the requests coming from advocacy organizations and/or government. In some cases they also identify research topics and conduct research if there is donor support. TTs are increasingly concerned about the absence of research quality control systems.73 TTs are building relationships with the government, media, civil society and some with the private sector through the G-PAC Program, and they increasingly see the importance of doing so, but still two-way external communication needs improvement to maintain continuity.74 TTs also indicated in interviews that they believe they should be considered more as G-PAC’s implementing partners, rather than grantees that need capacity strengthening. This makes it a challenge to suggest that TTs participate in training workshops, or even study tours. Some TTs mentioned that they have enough resources to organize study tours if they need. One exception to this is the mentorship program. Some TTs prefer to be a mentor to others; while some preferred to have a mentor, and benefitted from having one. The most favorable project intervention was the support grants; many of the TTs are proud of the results they achieved.75 EPRC mentioned the importance of the political parties’ project implemented by the Consortium - Economic Policy Research Center (EPRC); Transparency International (TI); Georgia; Liberal 71 G-PAC Second Quarterly Report, October-December 2010 72 Interviews with EPRC, LAT, IAG, September 2012 73 Interviews with EPRC, LAT, IAG, September 2012 74 Interviews with EPRC, LAT, IAG, September 2012 75 Interviews with EPRC, LAT, IAG, September 2012 18 Academy of Tbilisi (LAT). According to the EPRC and LAT the project created a precedent for leading Georgian political parties to present and discuss their pre-election programs to the general public to facilitate voters making an informed decision at the pools. According to the consortium members the project received wide recognition, had excellent media coverage and enjoyed the cooperation of leading political parties and high international and local interest.76 Training Workshops: G-PAC organized a 2-day Policy Analysis Workshop for Georgian TTs. The workshop was attended by 27 representatives of 15 organizations; out of these, six organizations composing the first group of participants were G-PAC grantees, while the second group was a mix of nine advanced think tank staff. According to G-PAC PMPs, training goals are being met in terms of numbers of workshops, and there are more trainees among think tank representatives than anticipated.77 The workshops accomplished the following: 9 Informed participants of the history and the role of TT and advocacy groups in Europe and around the world; 9 Disseminated examples of policy analysis and analytical techniques that have been proven effective; 9 Drew conclusions for effective management structure and practicalities for such organizations.78 Mentoring Program: The TTs interviewed are split on mentoring: some prefer to be mentors while others preferred having mentors and gained much from a relationship with a mentor.79 TTs interviewed discussed some of the assistance they’ve received from mentors, including developing a research agenda, reviewing research products, and developing public messages based on their research products, all of which are needed by the TTs.80 Study Tours: The TTs have participated in G-PAC’s study tour to Estonia, and have benefited from the opportunity to learn from their Estonian counterparts, but some TTs believe they can organize and fund their own study tours when necessary.81 Grant-making Program: Based on TT assessment findings, G-PAC designed the TT Support Grants (TTSG) program and encouraged TTSG applicants to submit projects aimed to foster informed public debate and policy-making addressing specific issues in one of the following priority areas: (1) economic development, (2) agriculture, (3) health and social assistance, and (4) education. The G￾PAC think tank assessment identified these areas as some of the most important, yet least addressed issues by Georgian civil society. The projects had to include comprehensive analysis of key issues facing Georgia, provide concrete recommendations for addressing problems, ensure a wide dissemination of findings, and conduct follow-up advocacy targeting the public and decision-makers. One example of a TT’s successful advocacy campaign involved the Insurance Association of Georgia, who conducted research on the factors influencing prescription practices and high pharmaceutical costs. Through their work, they produced recommendations for policy reforms to improve the situation, and these recommendations were endorsed and accepted by the government.82 Further, this is an example of a public-private partnership, involving both public and government interests as well as that of the private sector. The director of IAG indicated that the success of the project required the active participation of multiple sectors, and that it is a mistake for civil society to not consider working closer with the private sector in its activities.83 76 Interviews with EPRC, LAT, IAG, September 2012 77 G-PAC PMP Year One and Year Two, 2010-2012 78 G-PAC CSO Capacity Building Assessment Year One, Partners-Georgia 79 Interviews with Think Tanks, September, 2012 80 Interviews with Think Tanks, September 2012 81 Interviews with Think Tanks, September, 2012 82 Interview with IAG, September 2012 83 Interview with IAG, September 2012 19 Another example is that of the consortium led by the Economic Policy Research Center who conducted a project to increase the transparency of the Georgian electoral process by publicly promoting awareness of political parties’ electoral promises, policy proposals, and views on policies.84 As of August 2012, 13 awards were given to nine Tbilisi-based TTs with a total value of $724,487 to work on issues related to sustainable economic development, education, and health.85 According to G-PAC PMPs, the overall grant-making aspect of the program is exceeding their goals of 40 grants awarded per year.86 “It was rather useful as we learned about the public sector. We knew how the public sector worked but we learned much more during our when direct contact. I think it was quite useful.” Conclusions • G-PAC’s TTSG program demonstrated tangible results of contributing to, and influencing, the policy making process in Georgia; • Sectoral policy research production is still in short supply in Georgia and very few research organizations have deep knowledge of research methods and research design; • There is no peer-review practice to review research and provide feedback which makes it difficult to measure the quality of research products; • In many cases research is produced but not communicated properly with the government and the public; • TTs are improving their social network with peers and government sector and public; • TTs believe that they don’t need extensive capacity building and therefore, there is a danger that they will stop working on self-development. Recommendations • Emphasize funding to TTs who produce information by conducting original field research; encourage partnerships and coalitions between research producing TTs with advocacy organizations; introduce a peer-review practice in Georgia for quality control of research products; introduce an open door grant competition to allow TTs to address timely burning policy issues – much like the Act Now Grants. • Promote policy dialogue between TTs, government, civil society, and the private sector. • Promote cross-border cooperation with US and CEE counterparts under the mentoring program. Re-evaluate the TT capacity building program and propose different approaches for each one, based on the activities (research oriented, advocacy oriented or mixed) and their current capacities; consider a triage approach involving key staff, a mentor, and a training expert, working together closely providing a team-centric approach to the needed interventions. Having a training expert work closely with the mentor would develop appropriate trainings and strengthen key staff of the TTs. Best Practice • Encouraging Coalitions between TTs and Advocacy Organizations: Using grant program to encourage these types of organizations to work together builds better advocacy initiatives. Evaluation Question #4 What are the major challenges facing the newly established MPA program at ISU both in terms of academic and administrative work? Summary of Findings and Answer to Key Evaluation Question #4 The major challenges facing the newly established MPA program at ISU are to ensure the following: that adequate support and resources exist among the faculty and staff of the university to make the 84 Interview with EPRC and observation of Tbilisi-based event, September 2012 85 G-PAC Grants Report, 2011-2012 86 G-PAC PMP for Year One and Year Two, and G-PAC Grants Report, 2011-2012 20 program sustainable; that there is steady, gradual, and sustainable growth of the MPA program; and that adequate support for the MPA program among Georgia’s public and private sector employers exists. Background and Activities Conducted The establishment of a Master’s Program in Public Administration (MPA) at Ilia State University has two goals: 1) provide a new generation of well-trained public policy and public administration professionals; and 2) develop centers of excellence in Tbilisi and in the regions that can provide fundamental research and analysis to other users, such as TTs and NGOs. To achieve these goals, G-PAC partnered with Columbia University’s Picker Center for Executive Education through Columbia’s School of International and Public Affairs (SIPA). After an assessment of the institutional capacity of Ilia State University (ISU), Tbilisi State University (TSU), and the Georgia Institute of Public Affairs (GIPA), the Columbia-G-PAC assessment team recommended ISU to host the MPA program. In early 2011, EWMI, ISU and Columbia signed an MOU, outlining the terms of cooperation. Columbia then began working with ISU to develop a strategic plan for launching the MPA Program.87 After the MOU was signed, Columbia began sending faculty to ISU for consultation. In March, Professor Francisco Rivera-Batiz arrived and held a series of working meetings with Dean of the School of Graduate Studies David Aprasidze, and Head of the Office of Development and Foreign Relations, Nino Dvalidze, to discuss admission requirements, curriculum design, and other matters. He also consulted on Research Methods with ISU faculty members Nino Zhvania and Tamuna Bregvadze; on Statistics with Nikoloz Maghlaperidze and Archil Gagnidze; on Education with Tamuna Bregvadze and Sophie Gorgodze; and on Economics with David Tutberidze and David Khantadze.88 ISU’s MPA curriculum, developed jointly by Columbia University and ISU, consists of three parts: the core curriculum during the first year, which provides the necessary skill sets; the advanced curriculum during the third semester, which involves electives based on students program and policy interests; and the capstone curriculum during the final semester, which consists of an internship in the student’s area of interest and a workshop designed to give students an opportunity to apply the skills they have acquired. One semester has been completed by the first selected cohort of students who began in January 2012, and the second semester is currently on-going. This first group of students will study through December 2013, at which time they will graduate from the program. The MPA Program has a newly revised curriculum being taught by a number of professors jointly selected by ISU and Columbia University, and has a number of internships developed through relationships with G￾PAC, Columbia University, and a number of other institutions. Guest lecturers frequently enrich the program with topical presentations and discussions. University Assistance Grants (UAGs): UAGs support smaller-scale activities aimed at improving the capacity of all accredited Georgian universities in public policy, public administration, political science, and related academic programs or courses. As of August 2012 13 UAG Grants were awarded to eight universities for a total value of $ 117,861.89 According to G-PAC’s PMP, awarding 13 UAGs in the two-year period is exactly what was anticipated.90 Findings and Analysis 87 An Assessment of Three Georgian Universities, Columbia University, November 2010 88 G-PAC Third Quarterly Report, January-March 2011 89 G-PAC Grants Report, 2011-2012 90 G-PAC PMP Year One and Year Two, 2010-2012 21 The main challenges facing the MPA Program include: ensuring that adequate support and resources exist among the faculty and staff of the university to make the program sustainable; ensuring the steady and gradual, and sustainable, growth of the MPA program; and ensuring adequate support for the MPA program among Georgia’s public and private sector employers. New and enthusiastic leadership at Ilia State University are in place and are committed to addressing these challenges. The three staff together emphasized that the MPA Program is being guided by a strategic plan and they have the support of the University to pursue the strategic plan and work towards the growth and development of the program. They believe that in order for the program to experience growth the University will need additional resources for infrastructure to support larger classrooms, more computers, and a library.91 Ten MPA students from the first cohort participated in a focus group discussion and pointed to the following elements of the MPA Program that they thought most beneficial: highly qualified professors; professors’ teaching methods; team work; current reading materials; a proper balance between theoretical and practical; good management of program; visiting professors from U.S.; teaching in English; monitoring quality of research.92 The internship program was evaluated by the 10 ISU MPA student interviewees as interesting, intellectually challenging, and fun. The internship program, according to the interviewees, helped them put the knowledge acquired at the university into practice which subsequently contributed to the improvement of their skills. They believe that the internship program will enrich their CVs.93 The students also discussed that G-PAC had also funded an internship program for two students in Canada. This program was competitive, and in high demand among students. According to the interviewees those with more experience were given priority in the selection process for this internship program. They believe that the nature of this internship provides motivation for them to compete and apply for it.94 95 A stakeholder interview was also held at Gori Teaching University. These stakeholders cited a low employment rate of their graduates due in part to the minimal qualification of lecturers; the use of outdated materials for lectures; and the lack of skills to use technology to teach the courses.96 Conclusions • There is enthusiastic new leadership in place at ISU, willing to learn and willing to engage constructively in overcoming the challenges of developing the MPA Program; • ISU’s leadership has a strategic plan in place to guide the development of the MPA Program, and is pursuing the plan with vigor as indicated by stakeholders; • The MPA program has received noteworthy attention from Georgian public sector officials, and student applications increased from the first selected cohort of students to the newly selected in￾coming class, and more than doubled what the program anticipated.97 Recommendations • Emphasize continued learning opportunities for the new leadership of the MPA Program to ensure its continued success and sustainability. • Emphasize a pace of gradual, sustainable growth for the MPA Program following the strategic plan; emphasize broad participation and buy-in to the next strategic plan to help ensure adequate resources committed to the Program. 91 Interview at Ilia State University, September 2012 92 ISU MPA Students Focus Group Report, October 2012 93 ISU MPA Students Focus Group Report, October 2012 94 ISU MPA Students Focus Group Report, ACT, September/October 2012 95 ISU MPA Students Focus Group Report, ACT September/October 2012 96 Interview at Gori Teaching University, September 2012 97 G-PAC PMP for Year Two Work Plan, September 2011 22 • Encourage ISU to promote the MPA Program to public and private sector employers in Georgia, and continue to develop long-term relationships between the ISU program and Georgia’s public and private sectors. • Encourage the development of case studies. • Consider an activity or program that encourages professor and student exchanges, and the sharing of practices and knowledge between universities. • Based on the recommendation of the Rector of Gori Teaching University there should be: o An exchange of lecturers with Ilia University in order to assist improving curricula design and using modern teaching methodologies.98 o A G-PAC paid intern, who would work at the University and for G-PAC to develop a grant program that encourages consortiums of universities working together to facilitate knowledge and the sharing experiences.99 Best Practices • Internships and Study Tours: Encourage opportunities to allow for further comparative learning and analysis; • Networking with TTs: Encourage opportunities to share best practices and experiences in conducting evidence-based research. Evaluation Question #5 To what extent does G-PAC’s CSO Enabling Environment Initiative, as well as other interventions, contribute to financial sustainability of CSOs? What are the external factors affecting G-PAC’s work in this area? Summary of Findings and Answer to Evaluation Question #5 G-PAC’s Enabling Environment Initiative is contributing to the financial sustainability of CSOs and TTs, but at a very gradual, if incremental, pace. Appropriate and thorough analyses of laws and regulations have taken place and have been well-vetted with stakeholders. Both legislative and non￾legislative issues potentially impeding the ability of CSOs to operate effectively have been identified and prioritized. At least one amendment to the tax law has been proposed and passed by Parliament, but there are many more changes to pursue. Non-legislative issues also need continued pursuit, and relationships with the private sector should be maintained and built upon. G-PAC did not meet its PMP goals for Year One, but has set more realistic goals for Year Two and is meeting those goals. Background of Activities Conducted In August 2011, International Center for Not-for-Profit Law (ICNL) finalized its analysis of the Georgian Tax Code (Tax Code Analysis) identifying key issues that prevent CSOs from being financially sustainable and that impede philanthropy in Georgia. The analysis does not address all taxation problems affecting CSOs and businesses. Rather, it focuses on 10 specific problems prioritized by Georgian CSOs and businesses during their discussions with G-PAC and ICNL in March and May 2011. In addition to discussing these problems, the analysis provides concrete proposals on how each issue can be resolved.100 Once reviewed by the Civil Society Institute (CSI), the Tax Code Analysis was disseminated to Georgian CSOs and businesses for their additional input. It served as a guide on how they can proceed with proposed changes. ICNL and CSI also prepared a separate policy paper specifically addressing policy-makers, where they focused on the issues requiring legislative changes. The policy paper is shorter than the analysis and it prioritized issues differently, considering issues regarded as 98 Ibid 99 Ibid 100 Georgian Tax Code Analysis, ICNL 2011 23 particularly important from the government’s perspective, towards securing government buy-in. In total, three analyses were provided by G-PAC, which met their PMP goal for Year One.101 G-PAC and CSOs worked quickly with the Finance Ministry to develop a draft bill with amendments to the tax code, and in May 2012 the parliament approved the draft bill initiated by the Finance Ministry that freed CSOs from the requirement to withhold income tax on certain expenditures related to conferences, seminars, and publication dissemination.102 Findings and Analysis Although some progress has been made per G-PAC’s PMP, and some legislative amendments proposed by G-PAC grantees have been passed that improve the enabling environment for CSOs, there still remain significant challenges in the current tax code that will affect the operations of CSOs, e.g., key stakeholders, including the former deputy Minister of Finance, identified current tax code restrictions on economic activities of charities and the unclear rules regarding requiring CSOs to have separate accounting for economic and tax exempt activities.103 The meetings held with CSOs involved in carrying out income-generating activities and with local businesses involved in philanthropy showed that although CSOs in Georgia are allowed to carry out economic activities, they are subjected to complicated, unclear tax rules, and despite the income tax deduction provision, there are a number of legal obstacles that make charity an expensive and risky activity for businesses. These could be important contributors to the fact that most Georgian CSOs are reliant on foreign grants for over 90% of their funding, as shown in the NGO Sustainability Index of 2008.104 According to stakeholders, both the private sector and government find it attractive to contribute to service-providing CSOs, such as those CSOs advocating for the rights of the disabled, and this is likely to continue and may even grow in the coming years once tax restrictions are relaxed. On the other hand, according to stakeholders, government has little incentive, financial or otherwise, to contribute to traditional advocacy organizations due to the demands advocacy organizations may be placing on the government. This is not likely to change, even with improved and amended tax codes. The private sector may be more inclined to contribute to advocacy organizations, and there are examples within G-PAC, such as the IAG advocacy campaign, to point to as indication of this. Significant non-legislative issues have been identified through research and CSO forums that affect the ability of CSOs to operate; CTC worked on identifying the non-legislative issues that hamper CSO development in Georgia. Specifically, CTC focused on the following four questions: 1) How do people perceive CSOs and what do CSOs need to do differently to have more public support/trust? 2) How can CSOs improve their relations with government and gain more influence over policy￾making in Georgia? 3) How can CSOs improve their cooperation with donors? 4) What transparency and accountability standards do CSOs need to meet to become more professional and influential? In addition to discussing the above issues with CSOs, donors, and the media, CTC prepared four summary documents (one per topic) with specific recommendations for follow-up actions. They also organized a business-CSO forum, leading to greater cooperation between these two sectors.105 After the four topical discussions, four working groups were set up for each topic to summarize the discussion points and propose follow-up steps. G-PAC pursues these forums and activities implemented by CTC to provide an improved and sustainable environment for CSOs to operate. 101 G-PAC PMP Year One, 2010-2011 102 Key Informant Interviews, September 2012 103Key Informant Interviews, September 2012 104G-PAC Third Quarterly Report, January-March 2011 105 G-PAC Sixth Quarterly Report, October-December 2011 24 25 Conclusions • Some of the legal enabling environment issues have been resolved through legislative amendments proposed by G-PAC grantees, but there still remains little incentive by the private sector to contribute to charitable organizations and Georgian CSOs continue to rely heavily on foreign grants. • Establishing and maintaining relationships between CSOs and the private sector will be important as civil society evolves in Georgia. The private sector will need to better understand the benefit of contributing to and cooperating with CSOs. Recommendations Enabling Environment-Legislative Issues • The Ministry of Finance indicated that they have established excellent relationships with financial institutions. The same should be developed between the MoF and CSOs so that the MoF and CSOs better understand the issues and positioning of each other. This will help civil society understand current priorities of the government as it relates to the requests or demands of CSOs and to help CSOs make reasonable and realistic demands upon the government. • Consider developing a standing committee of key CSOs and key Ministry of Finance staff for long-term, continued dialogue and relationship-building for purposes of drafting and vetting legislative changes. • Pursue activities leading to the successful resolution of the remaining top 10 legal issues identified as priorities in the Georgia Tax Code Analysis. Enabling Environment-Non-Legislative Issues • Pursue activities leading to the successful resolution of identified key non-legislative issues that were identified by stakeholders during forums and roundtables organized by CSI. • Consider developing a standing committee of key CSOs and key stakeholders for continued dialogue and relationship-building for purposes of vetting and implementing priorities. Activities Leading to Sustainability • Consider a focus on establishing public-private partnerships where they make sense within the framework of the G-PAC program – look to the IAG advocacy campaign as an example of the public and private sector working together to achieve a common goal. • Consider adding a training/mentoring module on fund development, fundraising, and proposal development for CSOs. Best Practice • Cooperation with Ministry of Finance: Establishing close cooperation and long-term, continuing relationships with key Ministry of Finance staff is critical to CSOs in implementing changes to tax legislation. ANNEX I.: Disclosure of Conflict of Interest Forms 26 DISCLOSURE OF CONFLICT OF INTEREST FORM Name Marine Egutia Title Local Consultant Organization IBTCI Evaluation Position? Team Leader / Team member Evaluation Award Number (contract or other instrument, if applicable) AID-114-TO-12-00008 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) EWMI G-PAC I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. Signature: Date: 09.17.2012 27 DISCLOSURE OF CONFLICT OF INTEREST FORM Name Zehra Kacapor-Dzihic Title Evaluation Specialist Organization IBTCI Evaluation Position? Team Leader / Team member Evaluation Award Number (contract or other instrument, if applicable) AID-114-TO-12-00008 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) G-PAC I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6 P i d id t d i di id l i ti I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. Signature: Date: 9/17/12 28 ANNEX II: Evaluation Statement of Work (SoW) from the RFTOP STATEMENT OF WORK106 Mid-term Performance Evaluation of Policy, Advocacy and Civil Society Development in Georgia (G-PAC) and Georgian Media Enhance Democracy, Informed Citizenry and Accountability (G-MEDIA) Projects May 31, 2012 Summary The Statement of Work includes evaluation of USAID’s G-PAC: 1. Policy, Advocacy and Civil Society Development in Georgia (G-PAC) Project Number: AID-114-A-10-00005 Project Dates: September 1, 2010 – August 31, 2014 Project Funding: $13,278, 482.00 Implementing organization/s: East-West Management Institute (EWMI) Democracy, Informed citizenry and Accountability This mid-term project performance evaluation will be carried out in late August-September 2012 and must assess the extent to which the intended results of both projects have occurred. It must also look at the major impediments to achieving some of the results, and how the interventions can be improved to ensure achievement of projects’ end results. The results of the evaluation will be used by USAID/Georgia for making mid-course adjustments to both programs to increase their effectiveness and ensure the achievement of intended end results. G-PAC I. Purpose of G-PACs Evaluation and its Intended use The purpose of this mid-term performance evaluation is to: 1. Analyze the progress of the G-PAC program and its specific results to date in four project areas. This will be done by reviewing actual versus planned progress toward achieving key expected results, identifying accomplishments, delays, challenges and the reasons for them; 2. Offer recommendations vis-à-vis any necessary course-corrections/modifications of the program for its further successful implementation. The timeframe to be covered by the evaluation is from the start of the project in August 2010 through the initiation of this evaluation on/about August 2012. The audience for this evaluation will be USAID/Georgia mission and its DG office to understand the results as well as existing constraints and shortcomings that might hinder project implementation. The results of the evaluation will be especially useful for the implementing partner to understand the independent view about the project progress. The results will also be useful for other donors/organizations working on developing civil society in Georgia, as well as civil society representatives themselves. 106 The full Request for Task Order Proposal Scope of Work includes two mid-term evaluations IBTCI is conducting, one for this current project and the other for G-PAC. Excerpted here is information related only to G-PAC. 29 Finally, the results will serve as evidence for the FY 2012 Performance Plan and Report (PPR) development scheduled sometime in fall 2012. II. Background Georgia’s civil society sector has suffered in many ways since the Rose Revolution. The overall visibility and political influence of civil society organizations continue to diminish. Civil Society Organizations (CSOs), and think tanks in particular, are losing credibility in the eyes of both the government and citizenry. This greatly hampers their ability to engage the government constructively on key policy issues. Moreover, the absence of substantive public debate and discussion about key social, economic, and political issues exacerbates the lack of public trust and confidence in state institutions and in CSOs. “Civil society is dominated by capital-based advocacy and watchdog NGOs, much as it has been since the 1990s. Despite considerable experience with donor-funded projects and training, the policy analysis capacity of these NGOs remains limited and bolstered by external experts. These NGOs excel at critiquing government actions, but constructive recommendations based on evidence and cost-benefit analysis are rare.”107 Policy, Advocacy and Civil Society Development in Georgia (G-PAC) is the USG’s primary program to strengthen civil society in Georgia and to increase the efficacy of civic leaders and organizations. The project aims to strengthen civil society’s role in developing public policy reforms and advocating for their effective implementation. It works in several areas, namely: strengthening the advocacy and watchdog roles of civil society organizations; improving public policy discourse, research, and analysis, leading to the development and provision of viable policy options; enhancing the institutional capacity of universities in the areas of public policy, public administration, and political science; building coalitions to develop strong networks of CSOs throughout Georgia and strengthening their connections to media, political parties, and constituencies. G-PAC beneficiaries include Georgian think tanks, CSOs, universities, government, and the public. The G-PAC project involves two components: a comprehensive technical assistance and grants program benefiting Georgian think tanks, local and regional NGOs, and networks/coalitions engaged in policy analysis and advocacy; and the establishment of a Master in Public Administration (MPA) program. In Component One, G-PAC strives to strengthen the civic engagement of NGOs in effective advocacy and watch-dog activities, improve public policy analysis, and foster more cohesive networks. This engagement secures better access to and dialogue with political parties, local constituencies, and the media. Component One assists three major types of CSOs: think tanks, developed Tbilisi-based CSOs and CSO networks, and community and regionally-based CSOs. A grant program addresses each of these segments of Georgian civil society. G-PAC cooperates with two international partners in implementing Component One: the Open Society Institute Think Tank Fund (TTF) and the International Center for Not-For-Profit Law (ICNL), which has been working with G-PAC to create a better enabling environment for Georgian CSOs. 107 Democracy and Governance Assessment of Georgia 30 In Component Two, G-PAC, in partnership with Columbia University’s School of International and Public Affairs (SIPA) and the Picker Center for Executive Education, collaborates with Ilia State University (ISU) to establish a pilot MPA program, which will operate in conjunction with other central and regional universities. The MPA program at ISU will serve as a focal point for public policy research and an incubator for the next generation of Georgian policy makers and policy analysts. G-PAC’s anticipated results include: Strengthened advocacy and watchdog roles of more than 50 Tbilisi-based and regional civil society organizations (CSOs) as advocates for an empowered citizenry; Strengthened more than 10 issue-based CSO coalitions and networks throughout Georgia that advance their agenda through effective media, political parties, and constituencies; Improved public policy discourse, research, and analysis of more than 12 Georgian think tanks and academic institutions leading to the development of viable public policy options in key areas of interest to citizens, such as education, economic reform, health reform, land and property rights; A high quality Master in Public Administration program established at Ilia State University with 50 graduates by the end of G-PAC; Enhanced institutional capacity of six public and private universities in the areas of public policy, public administration, and political science; and More than 180 graduate students have gained experience in public policy analysis and advocacy. Progress to date By the end of March 2012 G-PAC accomplished the following: Grants Following an in-depth assessment of Georgian think tanks G-PAC awarded eight grants to think tanks and policy-oriented CSOs to develop and present policy options. Conducted the CSO Advocacy Capacity Assessment, G-PAC awarded 28 Citizens Advocacy small grants (CAGs) that support issue-based advocacy initiatives primarily at the local, municipal or regional level; Issued five Partnership for Change awards that support long-term advocacy initiatives that link multiple CSOs and think tanks. Introduced two new types of awards: Arts for Advocacy Grants, that support engagement in furthering advocacy causes (3 awards); and Act Now Grants that support advocacy initiatives in response to highly important and unexpected political, economic and social concerns (1 award). Training and Mentoring Engaged six Georgian experts who provided on-site and long distance technical assistance to CAG grantees. Dr. Krassen Stanchev, a leading Central and Eastern European (CEE) economist and analyst, provided mentoring to six Georgian think tanks and assisted in elaborating their research and advocacy plans. Offered issue-based trainings to CSOs and think tanks from all over the country in planning advocacy initiatives, communication skills, negotiation and presentation techniques, media strategy, 31 using new media for advocacy, engaging public, recruiting and managing volunteers, fundraising, building and managing coalitions and networks, monitoring and evaluation, innovative policy research techniques, think tank management, initiating public debate, existing global networks of policy research, etc. ICNL and its local partner Civil Society Institute (CSI), drafted an analysis of the Georgian Tax Code provisions regulating CSOs’ economic activities and business philanthropy and brought the issues affecting the financial sustainability of CSOs to the government’s attention. MPA Supported establishment of a two-year Master’s degree program in Public Administration (MPA) at ISU through a partnership with Columbia University’s SIPA and its Picker Center. A $200,000 grant has been provided to ISU for curriculum development, academic personnel recruitment, modern IT equipment, and a broad recruitment campaign. Hosted nine visiting professors to consult on a range of administrative and academic issues to the ISU MPA faculty and administration. Organized a study trip for the ISU administration and MPA faculty representatives to SIPA at Columbia University to provide them with greater insight into SIPA’s academic and administrative work. The MPA program was launched in March 2012 with 20 students selected out of 116 applicants. University Assistance Grants Through its University Assistance Grants (UAG) program, G-PAC continues to provide assistance to nine universities in developing public policy courses, organizing summer schools, and strengthening academic program management. The G-PAC Internship Program supported 38 graduate students from 11 universities and academic institutions to work for diverse governmental institutions, think thanks and CSOs throughout Georgia. Evaluation Questions and Methodology Key evaluation questions are: 1. How effective are G-PAC’s interventions to strengthen advocacy and watchdog roles of targeted Tbilisi-based and regional CSOs? 2. How effective are networks of CSOs supported under the G-PAC program? To what extent did project interventions result in strengthening their connections with each other, media, political parties, and constituencies? 3. How effective are G-PAC’s interventions to strengthen research capacity of targeted think tanks? 4. What are major challenges facing the newly established MPA program at ISU both in terms of academic and administrative work? 5. To what extent G-PAC’s CSO Enabling Environment Initiative, as well as other interventions, contribute to financial sustainability of CSOs? What are the external factors affecting G-PAC’s work in this area? 32 ANNEX III: Evaluation Methodology, Data Collection Plan, and Limitations This process evaluation will be conducted in accordance with USAID Evaluation Policy so that reliable data will be produced to support evidence based findings, be a sound basis for analysis that will lead to conclusions and recommendations that are useful for assessing the progress to date and also provide support for recommendations. Upon analysis of the project logic, the needs and expectations from the MTE by USAID, and the implementing partner, this MTE will employ “mixed methods” to optimise the potential of the analysis and to reach sound conclusions (See Annex II for the G-PAC Evaluation Matrix). In line with that, the methodology to be applied for this MTE includes qualitative and quantitative methods and instruments, such as key informant interviews, mini-surveys, focus group discussions, as well as a document review, and briefing meetings with Project staff. This approach allows for the verification of the findings through triangulation. The Evaluation Team understands triangulation as a method used during the evaluation process to check and establish validity of conclusions by analyzing the responses to evaluation questions from multiple perspectives, such as desk review, interviews, surveys, etc. Triangulation of data enables evaluators to ensure validity of conclusions, i.e. that the findings of assessment are true (accurately reflecting the situation) and certain (supported by the evidence). The evaluation is carried out in four phases: 1. Inception phase and the document review: these are used to base the development of the methodology and the evaluation instruments. 2. Fieldwork phase: comprises the travel to Georgia and within Georgia undertaken by the ET in Georgia over 14 working days. The ET will conduct interviews, mini-surveys and focus groups with a range of stakeholders from the civil society sector, the G-PAC Partners in Tbilisi and in the field, and similar. 3. The launch of research comprising of three mini–surveys, and (at least) nine focus groups, will be undertaken by a local research firm, EMC. 4. Analysis and report writing phase will include two main deliverables: the fieldwork de-brief meeting with USAID; and the evaluation report that is based on the data gathered in the field. The ET will finalize the report upon receipt of comments and inputs from USAID. Inception phase The inception phase focuses on detailed desk research and familiarisation with the project concept, vision and results thus far. This phase includes the following steps: • Initial discussions about expectations and current context with USAID, G-PAC and EWMI/Washington DC). • Preparation of a detailed work plan, timing and scope of the field mission, institutions likely to be met, survey data collection approach, sample and target regions; as well as research instruments; • Review of USAID comments and update of the work plan based on these comments. Deliverable: Evaluation Design and Work plan 33 Desk Review Document review includes: • Project documents, including project work plan and PMP; • Studies, reports and other relevant documentation prepared and commissioned by the project; • Relevant policy publications and studies from donors and other institutions. The document review and the analysis of the project interventions are used for the development of this evaluation matrix and design of the research methods to be applied in the data collection phase. During the desk phase, document review will attempt to identify examples of a targeted CSO-related effect on improvement of advocacy and watchdog roles as well as examples of targeted CSOs’ capacity building that is related to G-PAC support. During the desk review, the ET learned that the GPAC conducted a number of needs assessment for CSOs and think tanks. These documents provide the baselines for the ET to compare actual versus planned progress toward achieving expected results of strengthening advocacy and watchdog roles as defined in project documentation. These documents include but are not limited to: AYEG’s NGO Advocacy Capacity Assessment Report, the Report on Citizens Attitudes Towards CSOs and Civic Activism in Georgia, ICNL’s Tax Code Analysis, and PARTNER’s Year One Training Impact Assessment. Other reports will be considered as well, including the USAID’s CSO Sustainability Index, the NDI Poll, and Freedom House Nations-in-Transit. Deliverable: Survey questionnaires, Interview and focus groups’ guides Data Collection Phase The purpose of the data collection phase for the MTE is to systematically collect the information required to identify findings through answers to the evaluation questions. This data and the information from the previous phases will be used for conclusions and recommendations. The data collection phase will include coordination with the G-PAC team, contacts with stakeholders and counterparts, and the application of data collection methods. The main data collection methods to be applied, within this phase, are the desk review, interviews with stakeholders, focus groups, mini surveys, and the follow up interviews with the G-PAC project team. In this phase, the triangulation strategy based on the interviews will serve to question and/or verify the preliminary findings from the desk phase. In the interviews with the different stakeholders, their assessment of the project will be sought. While the individual assessments may be biased in various ways, triangulation will reveal a pattern of interest to the overall assessment of the evaluators. Key Informant Interviews. Formal, in depth, albeit semi-structured interviews based on a pre￾decided set of guiding questions will be carried out with the following respondents: 1. Implementation leaders such as EWMI component leaders; 2. Subject Matter Specialists such as technical assistance advisors (including short term specialists not presently in country); 34 3. Managers of partner organizations, Government of Georgia, policymakers and University staff who have been supported. Approximately 53 interviews will be held with representatives of different institutions and organizations. These interviews will be specialized, however the questions will be those submitted with this Work Plan to ensure parity and equity in the evaluation. Whenever possible, two members of the Evaluation Team will be present at interviews for the purpose of both note taking and also to compare impressions/conclusions after the interview. In addition to the interviews in Tbilisi, the ET will travel to Gori, Rustavi, Telavi, Kutaisi, and Batumi to conduct interviews with representatives of the civil society organisations who participate in G-PAC. Mini Surveys. A mini-survey is an important tool to complement narrative data collected during the desk review, field visits and the interviews. It will be based on a carefully designed set of methodological instruments that will allow respondents to provide information, share views and opinions while remaining anonymous. The purpose of the mini-surveys for this evaluation is to reach out to a majority of partners and beneficiaries of the G-PAC project, particularly those with whom the ET will not meet personally during the limited field phase. The survey respondents will be contacted either by phone or face-to-face. The 3 mini-surveys that will be conducted for this evaluation include: • Mini survey with grantees of the G-PAC project. A sample of grantees of approximately 140 respondents, i.e. all grantees that received G-PAC grants. This survey will reach out to all grantees of the G-PAC project, including both the think tanks and CSOs. The Survey will also aim at getting feedback from CSOs participating in G-PAC supported networks/coalitions • Community Opinion Poll. One community survey will take place in Kutaisi region where Georgian Association of Young Economists has conducted an advocacy campaign Protect Property Owners rights. The purpose of the survey is to better understand the extent to which the community residents were involved, participated, or otherwise were engaged in the overall effort, and what they knew and understood to be the purpose of the campaign. We want to learn whether they believe the campaign was beneficial to them, and whether they understand that by taking appropriate action, positive changes can be made to identify local issues. Total of 136 residents of communities of Kvachakhi (60 respondents), Geguti (53 respondents) and Mukhiania (23 respondents), who were beneficiaries of the campaign will be included in the sample. Focus Group discussions. A maximum of nine focus groups will be organized in order to allow participants to share opinions and ideas in the course of a discussion on a specified theme. The focus groups will be based on a standard methodology used to gather information that is beyond the scope of quantitative research that will be conducted through mini surveys. Focus group questions are designed as open-ended; in order to get ideas and opinions that are wider than the scope of prepared questions. Focus group guides have been developed for each of the target groups, and will 35 assist systematic analysis of the data collected. Selection of focus group participants will be performed by the ET and the G-PAC team. Ideally, focus groups will have five – twelve participants, to ensure that everyone has a chance to present their views, and will be organised in a neutral site, so that the participants feel comfortable. A small stipend for participants is provided for attendance. ACT is responsible for conducting the focus groups in with Mentors, Think Tanks, and CSOs in Tbilisi; also CSOs in Zugdidi and Svaneti. ACT will ensure that each focus group includes a facilitator and an observer, to make sure all viewpoints and ideas are recorded and that notes are taken. The ET team will conduct focus groups with CSOs and Interns in Gori, Rustavi, Telavi, Kutaisi, and Batumi. Each focus group conducted by ACT will be followed by a narrative report and transcripts of the focus group discussions, produced as part of the record. Note: The G-PAC Project conducted an Impact assessment of its Training programme in spring 2012. The data from this assessment will be used for the MTE. Therefore, no mini survey will be conducted with trainees of the Project. Throughout the data collection phase, data collection methodologies will be adjusted and modified in response to the constraints and opportunities encountered, in order to maintain and improve the quality of the evaluation. All collected data and work products, including but not limited to interview notes, surveys and focus groups reports and notes will be submitted to USAID. Deliverable: Survey results and initial findings Analysis and Reporting The analysis of received information, data and perceptions of the project will be an ongoing activity of the ET. The team members will discuss and compare notes at regular intervals during the data gathering and document review phase. The evaluation questions as stated in the RFTOP will provide for a fixed reference and guide for the analysis, conclusions and recommendations. The findings at each phase of this in country visit, will be noted and, as analysis progresses, these notes will be used to triangulate evidence for conclusions. Formulation of conclusions and recommendations will be based on the triangulation of collected documentation and its review; responses to the survey, focus group analysis, discussions and interviews with a broad range of stakeholders. The consultants’ experience in other CSO projects or evaluations within this sphere will also be relevant. The notes on gender implications, and also differences in responses by gender, will be cross tabulated and reported. Deliverable: MTE report including conclusions, and recommendations. Sampling Through mini surveys, the ET will reach out to as many as 140 representatives of beneficiaries, grantees and CSOs. However, it should be noted that compared to the number of individuals who have participated and benefited from G-PAC interventions during the two-year period, the Assessment sample size is not large enough to statistically generalize the findings to the rest of the beneficiary population. The sample does, however, offer a general pattern of opinions and 36 perspectives that, when triangulated, offer evaluative “impressions.” Individuals and organizations to be included in the sample will be limited by circumstances on the ground. Geographic coverage is relevant to this evaluation. Because G-PAC is a national project, the ET we will be traveling to Gori, Rustavi, Telavi, Kutaisi, and Batumi to interview recipients of G-PAC project’s interventions. EMC will organize focus groups in Tbilisi and Zugdidi/Svaneti regions. While these locations do not deliver a 100% perspective, they should enable the Evaluation Team to glean a sufficient understanding, when combined with data collected from surveys, polls and questionnaires implemented in areas not physically visited, to present valid conclusions and relevant recommendations. In order to ensure that the survey findings are based on inputs from all stakeholders and partners of G-PAC, the ET will aim at ensuring that the sample for the survey is full list of grantees. Survey data analysis will include both analyses of the entire sample, but also cross tabulations in order to explore views of different types of grantees, regions vs. Tbilisi, men vs. women, etc. Final survey report will also include the cross-tabulated data, and will be presented as an annex to the main Evaluation report. Weaknesses and Limitations of the Evaluation Design and How They Will be Managed There are two overall weaknesses and limitations for this evaluation having to do with the timing of the evaluation because of the upcoming parliamentary elections and the length of the fieldwork for the evaluation. The following list provides additional details as to how limitation factors could impact the evaluation. - This evaluation is taking place during the pre-election period of the upcoming parliamentary elections. It is not an opportune time because CSOs are primarily focused on the pre￾election processes. To overcome this, IBTCI is casting a very wide net to interview as many CSOs as it possibly can to compensate for the inevitability that some CSOs may not be able to meet with IBTCI. - The length of the fieldwork as prescribed in the RFTOP SOW is very limited. Fourteen working days in-country is an insufficient amount of time to be able to cover the breadth of the country and project the participants. G-PAC is a large project with numerous areas of intervention and wide range of partners and more time would be beneficial to ensure more complete coverage. To mitigate this, to complement the meetings and interviews conducted by the ET, IBTCI has partnered with the local firm, EMC, to conduct a range of focus groups and mini-surveys that the evaluation team would not be able to do on its own. Following are limitations and weaknesses of the evaluation methodologies and how IBTCI will manage them. ¾ Document Reviews: o Limitation/Weakness: Information may be inapplicable, disorganized, unavailable, or out of date; biased because of selective survival of information; incomplete or inaccurate; time consuming to collect, review, and analyze many documents. o How it will be managed: IBTCI will provide USAID with a supplemental list of the relevant types of documents required to conduct the desk review, in addition to the illustrative list provided in the RFTOP. When 37 the material is received, IBTCI’s project director will cull through the documentation to ensure that it is organized, current, and complete. If any gaps are found, the project director will communicate this with USAID to determine how to fill the gaps and/or alternatives to collecting the necessary information. Besides, the ET will collect and analyze all reports and data that will emerge as relevant once in the country. In order to ensure that all files are managed and available at all times to the team, a Drop box file has been created and will be accessible to the team. ¾ Key Informant Interviews: o Limitation/Weakness: Findings can be biased if the informants are not carefully, or purposively, selected. When only a few people are interviewed, it may be difficult to demonstrate any general, let alone statistically significant, validity of the findings. It is difficult to prove that the interviewees are, in fact, knowledgeable and informed and that they are representative of their peers in their information and recommendations. Findings could be susceptible to interviewer bias. o How it will be managed: IBTCI will manage the potential limitations and weaknesses of the key informant interviews by working with USAID and implementing partners to confirm that all key informants selected are relevant to the projects. IBTCI will develop a purposive sample, i.e., a qualitative sample based on those respondents likely to respond to questions and therefore to contribute to findings. IBTCI will interview as many relevant key informants as necessary within the allotted evaluation period to increase confidence in the validity of the evaluation findings. Furthermore, all data received will be triangulated through ongoing desk review and situational analysis, validation of findings through interviews, focus groups and survey as well as reflection and analysis of project reports and meetings with the team. The Evaluation Expert will provide an objective, scientifically rigorous set of interview guides and findings to prevent interviewer bias. ¾ Focus Groups: o Limitation/Weakness: The moderator has less control over the data produced than in either quantitative methods or even in KIIs. The moderator has to allow participants to talk to each other, ask questions and express doubts and opinions, while having very little control over the interaction other than generally keeping participants focused on the topic. By its nature focus group research is open ended and cannot be entirely predetermined. o How it will be managed: The focus group moderator will come from the professional evaluation firm that has been subcontracted by IBTCI. This will increase the level of confidence and rigor in ensuring that the focus groups stay on topic without dictating the direction of the conversation so not to jeopardize the organize nature of a focus group. ¾ Mini-Surveys: o Limitation/Weakness: The findings of mini surveys will provide quantitative findings that will allow for generalizations more so than those obtained from large surveys. This is especially true when probability sampling is not used. In many instances the small sample size does not permit an elaborate statistical analysis. Many policy- and decision-makers may consider findings from mini-surveys unreliable because of their small sample size. o How it will be managed: The number of the target groups that participate in the projects is small allowing the evaluation to survey each of them. This will reduce the chance of generalizations from a small population to a larger population.   38 ANNEX IV. : Data Collection Instruments-Interview Guides/Mini￾Surveys/Focus Groups Survey – G-PAC Grantees 1. Name of organisation: 2. Person answering: 3. Role of Person in the institution and main responsibilities: 4. What is the type of grant you received from G-PAC? o Think tank grant o Citizen Advocacy grant o Partnership for Change grant o University Assistance Grant o Arts for Advocacy grant o Act now o Other, _____ 5. How long has the organisation been active? a. Less than one year b. Up to five years c. More than five years 6. Geographical coverage a. National b. Regional c. Municipality 7. Was your experience with the grant process? o Positive. Explain o Negative. Explain o I don’t know 8. What are your project’s main achievements thus far? Please, elaborate on each achievement ____________ 9. What are the main challenges? Multiple response a. Accessibility to resources from government institutions b. Accessibility to resources from donors c. Accessibility to information d. Lack of resources of my institution e. Fluctuation in staff f. Political challenges g. Lack of technical equipment h. Other, _______ 10. What external factors influence your work? a. Political situation in the country b. Socio-economic factors and poverty c. Our region is isolated d. Other, _____ 11. Do you get support from G-PAC in mitigating the risks and challenges of your Project? 39 a. Yes b. No c. Don’t know 12. If yes, what type of support have you received? Multiple answer • Grant support • Technical equipment support • Advocacy support • Training and capacity building for improved institutional capacities • Trainings for professional growth (thematic trainings) • Other, ___ 13. Please grade the improvements as compared to the situation before the G-PAC support. Tick the relevant box. Question Yes, definitely Yes, a little No change No, it is worse Comment We are more focussed on what we want to achieve We have more professional expertise in our institution thanks to trainings we attended Advocacy initiatives are now better designed Our advocacy messages are more concise and objective We have skills to conduct better research on a topic of interest Better understanding of priorities and interests of the target audience The number of research studies increased the quality of our studies improved Our advocacy activities result in concrete actions We have a better strategy We have more competent staff We have better leadership We have more members/clients/supporters We have more funding sources We have more moral support from the community We have better relationship 40 with the authorities The authorities are asking us for advice more often We receive more funding from the authorities We have better cooperation with other NGOs We are more well known and respected in society Our mission and vision are more clear We have better capacities for strategic planning and management We improved our monitoring protocols and systems We have improved human resources protocols and systems 14. Please grade the results achieved by your organisation since the start of the G-PAC support Question Yes, definitely Yes, a little No change No, it is worse Comment We have inspired/assisted people to take action to address their issues The authorities have considered our suggestions The government law/policies/services have improved The situation of our target group has improved Other (explain) 15. Do you receive mentoring support for your CSO/think tank from GPAC? a. Yes b. No c. I don’t know 16. Is the mentoring support beneficial? a. Yes b. No c. I don’t know 41 17. What are the improvements that your organization had as a result of mentoring support? Multiple response a. We improved our technical skills for advocacy that result in more effective advocacy campaigns b. We extended our network c. We increased the number of advocacy activities d. We have better results of our advocacy initiatives e. We strengthened our institutional capacities f. We increased our funding base thanks to their support to our proposal writing process and contacts g. Other,__________ 18. What is your experience working with the media? a. They follow our work b. We send them press material and invite them to our events c. They do not follow our work d. They make negative campaigns about my organization e. Other,_____ f. Don’t know 19. How often do you work with the media to further your advocacy goals? a. Very often b. Often c. Not so often d. Rarely e. Never f. Don’t know 20. Have the results with working with the media been satisfactory? Beneficial? a. Yes, to a great extent b. Yes c. Not so much d. No, they have been adversary e. Don’t know 21. What have been the main challenges when working with public officials? _________________ 22. From your experience, which is the best scenario of working with public officials? What are the best practices? ________________ 23. When you approach public officials and the media, what types of communication tools do you use? Multiple answers a. Policy papers b. Policy briefs c. Studies d. Other,_____ 24. Do you contact them by… (multiple response) 42 a. email, b. phone, c. face-to-face d. by fax e. through media f. Other,____ 25. What communication skills did you improve thanks to the G-PAC’s intervention skills? Multiple response a. Increased number of interactions using different formats, b. Quality of studies, policy briefs and other advocacy products c. Number of studies or products 26. Which G-PAC intervention has had the most positive impact on your capacity to conduct advocacy? Multiple response • Capacity building, including training, mentoring, and consultations • Grant • Support to building alliances and networks • Other,_____ Related to Networks/Coalitions 27. To what extent did G-PAC’s interventions (training, mentoring, grants, etc) contribute in increase in activity and quality of advocacy of the networks/coalition of CSOs? a. To a large extent b. To some extent c. Average d. Not so much e. Not at all 28. What is the level of attribution of G-PAC to creating issue-based networks/coalitions? a. Very high, they are primary supporter b. High, they support the networks, but there are others too c. Average, this is not their primary focus d. Not so high e. Not at all 29. Is it better to advocate as an individual CSO alone, or within a network/coalition, and why? a. Better results are gained when CSO alone advocates b. It is much easier to advocate within a coalition c. Results and policy changes happen when coalition advocates d. Other, ______ e. Don’t know 30. What challenges do you face when working within networks/coalitions? Multiple response a. Selecting right network members, b. internal division of responsibilities within the network; c. ensuring accountability of all members; d. interaction with each other in external environment; e. decision-making processes; f. internal organization of the work of the networks; 43 g. selection of network leaders and delegation of roles and responsibilities h. Other, ______ i. Don’t know 31. Does working within a network of diverse CSOs in the coalition increase individual members’ social network? a. Yes, to a great extent b. Yes c. Not so much d. Not at all e. Don’t know 32. How would you assess: Very positive Positive Average Not very positive This community is not familiar with G￾PAC I don’t know Overall perception of G-PAC in the CSO community Overall perception of G-PAC in local communities 33. What are the strengths of the partnership between your organisation and G-PAC? ------------------ 34. What are the weaknesses of the partnership between your organisation and G-PAC? ------------------------- 35. What improvements should be made at this point of project implementation? ----------------------------- 36. Overall, how would you assess your cooperation with G-PAC? o Very good o Good o Average o Not so good o Very bad 46. Other comments. 44 Interview guide for Grantees (CSOs and Think Tanks) Name of organisation: Person answering: Role of Person in the organization and main responsibilities: General questions 1. Tell us briefly about your organisation and the target group 2. How long have you cooperated with G-PAC Project? 3. How would you describe the partnership with G-PAC Project? 4. How many and what types of grants have you received from G-PAC? What have been the results of the support by/cooperation with the G-PAC? 5. What types of capacity building interventions did you receive? Did they respond to your organization’s Advocacy Capacity strengthening needs? 6. What are changes does your organisation encounter as result of the capacity building? Please give examples a. increased organization’s research production capacity (increased # and quality of research)? b. peer review of scientific research before it is disseminated c. improved communication skills -- summarise research results in a user friendly manner and use research communications skills to present research results concisely and in accessible language for policy makers and other target audience d. encouraging the use of research evidence by policy-makers 7. Do you think that decision makers in the country usually give formal consideration to any evidence identified and recommended by your organization for policy making? 8. Has your organization established arrangements (formal and informal) with policy makers to identify the relevance of what you do and what they need? Please give examples on how this has changed over the years? 9. What are the main factors motivating the government to use the research evidence? 10. Do you think that G-PAC Package of Interventions increased your social network (Increased interaction with Public officials, citizens, peers, media, experts etc. on a policy related issues)? Please give examples on how this has changed over the years? 11. What is your experience of using the Media in Advocacy process? Please give examples on how this has changed over the years? 12. Has your organisation managed to empower people or organisations to take action? Give examples on how this has changed over the years. 13. Has your organisation managed to build alliances with other stakeholders? Please give examples on how this has changed over the years. How sustainable is your work? 14. Please describe what are your major internal achievements (in terms of capacity: human resources, procedures, budgets, and similar) since 2010? 15. What would happen if the support from the G-PAC was faced out? What would happen if your organisation had to close down? 16. Have you designed any strategic plan for financial sustainability? Other questions 17. In your opinion, how is G-PAC doing overall? How are GPAC activities integrated in the society as a whole? 18. What are the recommended areas of improvement? 19. Do you have any further suggestions, comments or ideas? 45 Interview Guide – MPA programme Person answering: Role of Person in the organization and main responsibilities: 1. How would you describe the partnership with G-PAC Project? 2. How many students have enrolled this year/last year? 3. Do you believe that Ilia University is attracting a better quality student for the MPA program, and do you believe the numbers of applicants are increasing? 4. Do you believe you are meeting the students expectations, and how? 5. Is the faculty able to attract high quality professors from the market? (Competitive, salaries, etc). 6. How would you assess your performance as per administrative and organizational structures? What are the main challenges in your performance? 7. How flexible are your administrative and organizational structures in adapting to changing external conditions? 8. Have you built good relationships with the public sector, and do you know the expectations of the public sector? 9. How would you describe the partnership with G-PAC/Columbia University? What have been the main results of this cooperation? Challenges? 10. Do you believe that the G-PAC program supports Ilia University to be a stronger university in the market – through reputation, curricular development, additional seminars by outside professors, etc.? G-PAC related questions 11. What are the overall impacts of G-PAC at this point in the life of the project? 12. What are the strengths of the partnership between your organisation and G-PAC? 13. What are the weaknesses of the partnership between your organisation and G-PAC? 14. What have been the most successful and impactful interventions of this partnership so far? 15. What improvements should be made at this point of project implementation? Other questions Do you have any further suggestions, comments or ideas? 46 Interview guide: External partners Name of organisation: Person answering: Role of Person in the institution and main responsibilities: A. General questions 1. Tell us briefly about your institution 2. Have you cooperated with G-PAC? 3. If yes, how long? 4. Please describe your relationship with the G-PAC? 5. How long time have you interacted with the G-PAC? 6. What is your impression of their aims? 7. Which CSOs/think tanks are you also supporting? 8. To what extent has the project been able build relationships and alliances for change? 9. To what extent has the project been able to interact constructively and influence public authorities? Outcome/impact 1. How do you view the developments of civil society in your country? Present and future role? 2. Are civil society organisations able to influence/contribute to change? In what areas are they succeeding (democracy, human rights, social justice, gender equality, environment/climate protection etc)? 3. What is your impression of the major donors of civil society development? How do they differ? 4. Have you any impression of the specific contributions made by the CSOs/think tanks? Examples? Relevance 5. What are the most important enabling and impeding factors for a vibrant civil society in Georgia? 6. To you knowledge, how does the G-PAC support differ from other donors support to civil society? 7. What do you consider to be the most important task or role of civil society at present? Effectiveness 8. What is according to your opinion the best way to achieve the overall goals of a vibrant civil society in the country? 9. What types of organisations are able to make a difference? What kind of methods work best? 10. In order to achieve change one often talks about reaching a “critical mass”. What does this critical mass imply in your country? Is the society close to reach a critical mass? 11. In general, do donors align their support to the civil society? What are the driving/restraining forces for alignment/non alignment? Sustainability 12. In what way do you consider the achievements made by civil society sustainable, what are the main risks if donor would stop it support to civil society? 13. What would be the risks of the civil society organisations to become dependent on the own state (governmental funding)? 14. Will there be possibilities to institutionalise the support (into public structures)? 15. What are the different preconditions between the different type of organisations (advocacy/social service or member based/expert) with regards of financial and institutional sustainability? 47 Part V - discussions with the G-PAC representatives Person(s) answering: The intent of this focussed conversation is to gain an understanding of 1. The efficiency and added value of the G-PAC contribution in relation to partners and programs. (We want to give G-PAC representatives an opportunity to reflect on their work, their relationship with their partners and to experience this reflection as both useful in making an assessment of their work and in providing direction and impetus for the future.) 2. The relevance, effectiveness (impact and outcome) and the sustainability of the support provided since 2010 The added value and efficiency 1. It would be useful to have your organisation’s perspective on the efficiency of this type of program support. Is it you view that the approach of supporting Georgian CSOs in advocacy activities at regional and national levels is an efficient and useful approach? How so? What changes would improve efficient delivery of the support? Is there a risk of not achieving local ownership and drive? 2. What exactly do you do apart from providing funding to the CSOs/Think tanks/MPA? Give examples. 3. What would happen to the CSOs/Think tanks if you withdrew the finding? The other support? 4. Does this approach add value to Sida’s operations in the development of civil society? How? Relevance 1. To what extent does your work with CSOs/think tanks and their work in particular, contribute to the overall democratization in Georgia? 2. To what extent does your work with CSOs/think tanks and their work in particular, contribute to development of a vibrant civil society in the county? 3. To what extent does your work with CSOs/think tanks and their work in particular, contribute to achieving the desired changes? Effectiveness There are three levels to the questioning here (effectiveness of your organization, effectiveness of your partner and effectiveness of the joint program • How would you describe your relationship with your partners? How many are mainly interested in funding? How many use your skills and experience in designing and delivering programs? How many are interested in your networking facilitation? 4. How has the capacity of you partner organisations developed as a result of you work with them? 5. How would you describe your strategic thinking and strategic planning? Do your strategic processes provide a good framework for your work with CSOs/think tanks? How does it differ from that of other donors? 6. To what extent do you assist national partner organizations in developing their skills and knowledge? How do you approach issues of value and attitudes? Describe your programs and approaches. 7. How do you manage risks? Sustainability 1. To what extent is the work you are doing with CSOs sustainable? In your answers, please give consideration to the state of CSOs o management and administrative capacity, o ability to keep updated and work strategically and flexible o ability to interact with and influence authorities, o ability to mobilise participation and support from grass roots o ability to mobilise a diverse funding base 48 Impact and outcome 1. Has the capacity of F CSOs/think tanks been developed? How? 2. Has civil society been strengthened as a result of the program? In what way? 3. To what extent does your work with CSOs/think tanks, and their work in particular, contribute to change in processes or policies at a local or national level? 4. To what extent does your work with CSOs/think tanks and their work in particular, contribute to democracy and rights? 49 Focus group with Think tanks/CSOs Name of organisation or forum: list think tanks/CSOs Person(s) answering: list positions in society or organizations The intent of this focused conversation is to gain an understanding of 1. The efficiency and added value of the G-PAC contribution in relation to partners and projects. (We want to give partners an opportunity to reflect on their work, their relationship with G-PAC and to experience this reflection as both useful in making an assessment of their work and in providing direction and impetus for the future.) 2. The relevance, effectiveness (outcome) and the sustainability of the support provided since 2010 3. What are the most important changes you encounter in the work of think tanks/Advocacy-watch dog CSOs? 4. What are the most important results of your work? 5. What helped you reach these changes? 6. What is the role of G-PAC in this process? 7. What is the role of policy makers/government in this process? Now let’s discuss the general context for the work of CSOs/think tanks in Georgia 1. What it Georgian Civil Society? How do you define it? 2. How has Georgian Civil Society changed over the past ten years? 3. Is Georgian Civil Society getting better or worse? Why? 4. Who are today’s leaders of Georgian Civil Society? 5. What more can be done to support the development of GCS? Regionally vs. Tbilisi? 6. What more can the Government of Georgian do to support the development of GCS? 7. What more can international donors do to support the development of GCS? 50 Focus group with University Students 1. Why did you select Ilia University over other university MPA programs? Explain 2. Did you have experience working in the public sector before joining the MPA program? Yes or No 3. Do you find that the courses of the MPA program help you in your daily work performance, and increase your work performance? Yes or No, Explain 4. Do you believe that the MPA program will increase your chances of earning promotions through your work and to advance your career? Yes or No, Explain 5. What are the main skills and knowledge gained through the MPA program that will help you in your professional advancement? 6. How accessible are the professors to help answer questions or help you with your work, and how flexible are professors in reflecting comments and feedback from students and incorporating it into their class work? 7. Does the university MPA program assist with work placement? 8. How is the balance between theory and practice in the course work? 9. Please, rate the extent is the internship program useful in: Extremely helpful helpful Moderately helpful Not so helpful Not at all Don’t know Putting your theoretical knowledge into practice Building future relationships with employers in finding new employment Other 10. What are your suggestions and ideas for improvement of the MPA? 11. Other comments 51 Focus group with mentors List of mentors: The intent of this focused conversation is to gain an understanding of scope of work of mentors, clarity of responsibilities, efficiency and effectiveness of this kind of intervention. Also, the focus group will explore the lessons learnt and best practices from the process so far. 1. What are your main duties within the role of Mentor? 2. Did you get terms of reference from GPAC and/or you change/adapt your roles based on identified needs of your beneficiary organisation? (focus the discussion on the balance btw. institutional strengthening/organisational development and technical capacities for advocacy) 3. How many organisations do you mentor? 4. What are the main activities that you conduct in your role of mentor? What methodologies do you apply? 5. How frequently do you visit/mentor your beneficiary organisation? 6. What are the main challenges? 7. What changes have you seen in your beneficiary organisations that are the result of your support? 8. Do you perceive any improvement in quality of advocacy initiatives organised by your beneficiary organisation? In what way – please, give examples. 9. Did the number of these advocacy activities increase in the last two years? How? Motivation to do mentoring 10. Why are you a mentor? What motivates you to do this job? 11. How are you benefiting by being mentors? Do you gain new knowledge? New skills? New experiences? 12. Do you get support from G-PAC in areas where you lack extensive information/skills for supporting your beneficiary organisation (focus the discussion to what support mentors get in situations when the beneficiary organisation asks for some assistance that is beyond their knowledge/experience) 13. What are the areas of improvement in the mentors’ scope of work? 14. What are the areas of improvement in the GPAC Programme as a whole? 52 Community Survey Questions – Kutaisi Municipality Advocacy Campaign of Roads/Land Registration Campaign by AYEG-Kutaisi Branch Tell us about yourself 1. Gender o Female o Male 2. Age: o 18 to 29 yrs. o 30-39 yrs. o 40-49 yrs. o 50-59 yrs. o 60-69 yrs. o over 70 3. Your current employment status o Self employed in farming – livestock and agriculture o Self employed in own business or professional activity unrelated to farming o Intermittently employed or works from time to time o Permanently employed – state or public sector o Permanently employed – private sector o Unemployed – seeking for an employment in the last month o Unemployed – not seeking for an employment in the last month o Pensioner o Student o Unfit or of limited fitness for work o Other o No answer, refused 4. Level of education o No education o Elementary o Secondary o Tertiary education o Masters/PhD o No answer 5. Did you know that you needed to re-register your land? o Yes o No o Do not know 53 6. Why was your land not re-registered? Multiple response a. Lack of financial resources b. Lack of knowledge that it should be registered c. The land was registered in some other person's name, so it was difficult to restore ownership d. Other, ______ 7. Why was it important to you to re-register land at this time? _________________ 8. How did you hear about the land re-registration procedures? Sources? a. T.V. b. Newspapers, c. Neighbors, d. Community meeting e. Other people f. Georgian Association of Young Economists g. Other, _______ 9. Did you encounter problems during registration process a. I did not have enough money to pay registration b. I did not have proper documents to prove my ownership c. I did not know where to go d. Other, _______ 10. Did the Georgian Association of Young Economists help you through the land registration process? a. Yes b. No 11. If yes, in what way did they help you? • Through consultation • They helped me to collect papers • They provided legal counseling • Other, ________ 12. If yes, are you satisfied with the way in which they helped you? • Yes, very much • Yes, to certain extent • Not very satisfied • Not at all satisfied 13. If no, who helped you? _______ 14. What was the result of the campaign to register land? a. We registered our land b. We got compensation for highway construction c. We got partial compensation 54 55 d. No results of the campaign 15. Was the campaign helpful to you? a. Yes, to a great extent b. Yes, to a certain extent c. Not really d. Not at all 16. If not, what is the reason? ______________________ 17. Without the campaign, would you have achieved the same results? a. Yes b. No c. Maybe d. Do not know 18. What did you learn from the campaign? Multiple response a. New things about land registration b. New things about highway construction compensation c. New things about the way in which land registry office operates d. Other, ______ 19. In what way the process of land registration with help of Georgian Association of Young Economists improved your life? Multiple response a. I now have registered property b. I know have more financial resources thanks to compensation I got for highway construction c. I now am not worrying about my land papers d. Now, I know more about the process and can help others e. Other,________ 20. What did you like about the campaign? _______ 21. What did you dislike about the campaign? __________ 22. Did you know that the campaign was supported by USAID/G-PAC Program? a. Yes b. No ANNEX V: Bibliography of Documents Reviewed # Title Source 1 USAID/EWMI/G-PAC Scope of Work EWMI/G-PAC 2 EWMI/G-PAC Year One Work Plan and 6 Annexes EWMI/G-PAC 3 EWMI/G-PAC Year One M&E Plan EWMI/G-PAC 4 EWMI/G-PAC Year Two Work Plan and 4 Annexes EWMI/G-PAC 5 EWMI/G-PAC Year Two M&E Plan EWMI/G-PAC 6 EWMI/G-PAC Quarterly Reports and Annexes 2010 - 2012 EWMI/G-PAC 7 EWMI/G-PAC Grants/Grantees Report 2012 EWMI/G-PAC 8 EWMI/G-PAC Success Stories EWMI/G-PAC 9 University Assessment Report by Columbia University EWMI/G-PAC 10 ISU MPA Program Strategic Plan 2012-2015 EWMI/G-PAC 11 MPA Program Visiting Experts Database EWMI/G-PAC 12 EWMI/G-PAC Press Kit on Citizens’ Advocacy Grants 2011 EWMI/G-PAC 13 EWMI/G-PAC Fact Sheets (9) EWMI/G-PAC 14 EWMI/G-PAC News Issue #4 Oct. 2011-March 2012 USAID/DC 15 USAID CSO Sustainability for Georgia 2011 USAID/DC 16 USAID DG Bi-Weekly Report Issue #10 July 1-19, 2012 USAID/DC 17 USAID DG Bi-Weekly Report Issue #12 Aug 6-19, 2012 USAID/DC 18 Nations-in-Transit Report Georgia 2012 Freedom House 19 Freedom in the World Report Georgia 2012 Freedom House 20 Georgia’s Television Landscape TI August 2012 TI and IREX 21 Mapping Digital Media Georgia June 2012 OSF 22 OSCE PA Press Release on Pre-Election Visit Aug 2012 OSCE 23 USAID Evaluation Policy January 2011 USAID 24 USAID Evaluation Policy Checklist 2011 USAID 25 AYLG’s NGO/CSO Advocacy Capacity Assessment 26 Report on Citizens Attitudes Towards CSOs and Civic Activism in Georgia 27 ICNL’s Tax Code Analysis 28 PARTNER’s Year One Training Impact Assessment 56 ANNEX VI: In-Country Schedule of Meetings Estimated Timeline In-Country for G-PAC Wednesday, September 5, 2012 • EWMI –, via telephone conference call Thursday, September 6 • USAID Friday, September 7 • USAID Wednesday, September 12 • Columbia University/SIPA Thursday, September 13 • Depart WDC, 10:00 PM Friday, September 14 • Traveling to Tbilisi Saturday, September 15 • Work with Zehra and Marine on Work Plan Draft Sunday, September 16 • EWMI • Meeting with Zehra and Marine • Meeting with Maia and Marine Monday, September 17 • Economic Policy Research Center Media Event at Marriott Hotel • Economic Policy Research Center (EPRC) (TT) • Young Economists Association of Georgia (TT) • Prepare for USAID Meeting • Skype Call with IBTCI Tuesday, September 18 • USAID • PARTNERS • Ilia State University Wednesday, September 19 • Day Trip to Gori • Society Biliki, and Club of Gori Invalids, for Focus Group 57 • Intern at Gori Municipality • Gori State Teaching University Thursday, September 20 • Marine and Zehra • Civic Development Institute 2:00 PM • Liberal Academy of Tbilisi 6:30 PM Friday, September 21 • Caucasus Research and Resource Council 11:30 PM • Insurance Association of Georgia 1:00 PM • OSFG 2:30 PM Saturday, September 22 • CSI –11:00 AM • MoF –12:30 PM • Meet Marine to Debrief Sunday, September 23 (Zehra Departs Tbilisi) • Prep for USAID out-brief • Prep for Final Report • Catch Up with Notes • Clean Up Files in Dropbox Monday, September 24 • Day Trip to Kutaisi • Meet in Kutaisi, Young Lawyers Association and Young Economists Association and New Civic Thinking and Interns of the Mayor’s Office and of Young Economists respectively and Local Democracy Agency (LDA) and Kutaisi City Hall respectively. (2 Meetings: CSOs and Interns) • Return to Tbilisi • Prepare for USAID Pre-Out-Brief Tuesday, September 25 • Pre-Out-Brief for USAID – 10:00 AM • EWMI Wednesday, September 26 • Official Out-Brief to USAID • ISFED Thursday, September 27 • Move Hotels • EWMI • Green Alternative 58 59 Friday, September 28 • Day Trip to Rustavi • In Rustavi Meet with CiDA Return to Tbilisi • Project Harmony Saturday, September 29 • Work with Marine on Final Draft Assignments Sunday, September 30 • Work with Marine on Final Draft Monday, October 1 • Work with Marine on Final Draft Tuesday, October 2 • Depart Tbilisi Wednesday, October 3 Thursday, October 4 Friday, October 5 • Return to Tbilisi Saturday, October 6 • Depart Tbilisi for WDC via Munich Sunday, October 7 Possible Meetings for Consultants in Poti and Batumi • Meet in Poti, Human Rights and Social Justice Research Center – EMC TBD • Meet in Batumi, Step Forward, the Institute for Democracy (2 Meetings: CSO and Intern) ANNEX VII: List of G-PAC Interviews # Date Entity Contact Title Place of Interview City 1 9.05.2012 EWMI/G-PAC CoP – via Telephone Conference Call IBTCI Offices Vienna, VA 2 9.05.2012 EWMI Vice President – via Telephone Conference Call IBTCI Offices Vienna, VA 3 9.06.2012 USAID Evaluation COR – via Telephone Conference Call IBTCI Offices Vienna, VA 6 9.06.2012 USAID DG Officer/POC – via Telephone Conference Call IBTCI Offices Vienna, VA 4 9.07.2012 USAID Democracy and Regional Specialist State Department Washington, DC 5 9.07.2012 USAID Democracy and Regional Specialist State Department Washington, DC 7 9.12.2012 Columbia Uni Prof. Director of ISU MPA Program – via Telephone Call IBTCI Offices Vienna, VA 8 9.12.2012 Columbia Uni Prof. Director of ISU MPA Program – via Telephone Call IBTCI Offices Vienna, VA 9 9.16.2012 EWMI Director of Civil Society Programs Courtyard Marriott Tbilisi 10 9.17.2012 EPRC CSO Representatives for Media Event Marriott Tbilisi 11 9.17.2012 EPRC CSO Representatives EPRC Office Tbilisi 12 9.17.2012 Young Econ. Ge. Representative AYEG Office Tbilisi 13 9.18.2012 PARTNERS Ge Representatives PARTNERS Office Tbilisi 14 9/18/2012 Ilia State Univers. Representatives Ilia University Tbilisi 15 9.19.2012 Society Biliki Representative for Focus Group Club Invalid Office Gori 16 9.19.2012 Club Invalids Gori Representative for Focus Group Club Invalid Office Gori 17 9.19.2012 Intern Gori Munic. Intern from Gori Municipality Club Invalid Office Gori 18 9.19.2012 Gori State Univers. Representatives Gori State University Gori 19 9.20.2012 Civic Dev. Instit. Representatives CDI Office Tbilisi 20 9.20.2012 Liberal Acad. Tbil. Representatives LAT Office Tbilisi 21 9.21.2012 Caucasus RRC Representatives CRRC Office Tbilisi 22 9.21.2012 Ins. Ass. Ge. Representatives IAG Office Tbilisi 23 9.21.2012 OSFG Representatives OSFG Office Tbilisi 24 9.22.2012 CSI Representatives Courtyard Marriott Tbilisi 25 9.22.2012 MoF Representatives Courtyard Marriott Tbilisi 26 9.24.2012 New Civ Thinking Representatives AYEG’s Office Kutaisi 27 9.24.2012 GYLA Representatives AYEG’s Office Kutaisi 28 9.24.2012 AYEG Representatives AYEG’s Office Kutaisi 29 9.24.2012 Intern Intern AYEG’s Office Kutaisi 30 9.24.2012 Intern Intern AYEG’s Office Kutaisi 31 9.24.2012 Intern Intern AYEG’s Office Kutaisi 32 9.24.2012 Intern Intern AYEG’s Office Kutaisi 33 9.25.2012 EWMI Representative Courtyard Marriott Tbilisi 34 9.25.2012 EWMI Representative Courtyard Marriott Tbilisi 35 9.26.2012 USAID Out-Briefing with Mission USAID/Georgia Tbilisi 60 61 36 9.26.2012 ISFED Representative ISFED’s Office Tbilisi 37 9.27.2012 EWMI Representative EWMI’s Office Tbilisi 38 9.27.2012 Green Alternative Representatives Tbilisi Marriott Tbilisi 39 9.28.2012 CiDA Representative CiDA’s Office Rustavi 40 9.28.2012 Project Harmony Representative PH’s Office Tbilisi ANNEX VIII: Mini-Survey and Focus Group Reports 62 Kutaisi Community Survey Technical Report Prepared by ACT Research For IBTCI October, 2012 Tbilisi. Georgia ACT Research / October, 2012 2/5 Content 1. INTRODUCTION .................................................................. 3 2. SURVEY METHODOLOGY....................................................... 3 QUANTITATIVE RESEARCH – FACE TO FACE INTERVIEWS ................................................................ 3 NOTES ON THE PROCESS ................................................................................................................... 4 3. LIMITATIONS/OBSTACLES OF THE SURVEY ................................ 5 4. REPORTING AND DELIVERABLES ............................................. 5 ACT Research / October, 2012 3/5 1. Introduction The given document is the technical report on conducting Kutaisi Community Survey. The report is prepared by ACT Research for IBTCI. The survey aimed at evaluation of effectiveness of Advocacy Campaign of Roads/Land Registration. Association of Young Economists of Georgia (AYEG) has been defending the interests of families living near the Zestaponi - Samtredia highway construction zone. In autumn 2011, AYEG organized informational meetings with families affected by the construction, and provided them with free legal consultations about how to register their properties, calculate appropriate monetary compensation for damages, and request this compensation from the construction company and the government. AYEG then monitored the compensation that the state provided to eligible families. The detailed methodology of the project is described below in the relevant chapters of the report. 2. Survey Methodology Quantitative Research – Face to Face Interviews Instruments The final instrument for the survey was provided by the client in English. The instrument was translated in Georgian by ACT translators and had undergone several corrections, e.g. skip questions identified and minor adaption of the text into Georgian was done. The methodology of conducting the survey was suggested by client. The survey was done via face to face interviews. Target Groups and Sample Target group of the survey were the beneficiaries of G-PAC program. The list was provided to ACT by the client. ACT Research / October, 2012 4/5 ACT has corrected the list in order to facilitate the fieldworks process. Notes on the process Training of Field Personnel Fieldwork staff training was conducted according to existing ACT procedures. Trainings were performed by the Project Manager and Field Manager. Interviewers were instructed on survey theme (objective and tasks) and Survey instruments. Experienced interviewers were involved in implementation of the field work. Fieldworks The fieldworks were done during 3 days, 28th and 29th of September, 2012 and 3rd of October, 2012. The table below shows the statistics of the survey conduction: Type of Database Number of Unique Attempts Made Number of conducted Interviews Methodology Beneficiaries 132 102 Face to Face Interviews In 73 cases, the interviews were conducted with the people that were given in the list, in other cases; the interviews were conducted with other informed persons of the family, like family members and heirs. All the changes of persons in the list were preliminary agreed with the project manager. The statuses of each attempts of contacting the respondents are described in table below: Status Beneficiaries Interview was conducted 74 Interview was conducted with other informed person 28 Lives in other city/could not find the address in other city 19 Could not find the person with such name 5 Has died 2 Lives abroad 4 ACT Research / October, 2012 5/5 Fieldwork Quality Control Almost 30% of the face to face interviews were controlled by the Quality control department of ACT. The quality control was conducted via special mini questionnaire made based upon the basic questionnaire of the survey. No serious violations occurred during the fieldwork quality control process. 3. Limitations/Obstacles of the survey The list provided to ACT did not contain any numbers or addresses of the respondents, the interviewers had to search for targeted respondents using snowball methodology that made the fieldworks process harder. Some of the respondents were included in the list more than one time, in such cases, only one interview was conducted with such respondents. Due to lack of time and tight timeline, the piloting of the final instruments provided by the client was not considered. 4. Reporting and Deliverables After conducting the research ACT provided the client with the following deliverables: • Survey data files with original data SPSS and MS Excel format; • Survey Frequency tables in MS Excel format; • Summary report in English (12th of October) • Technical report G-PAC Grantees Survey Summary Report Prepared by ACT Research For IBTCI October, 2012 Tbilisi. Georgia ACT Research / October, 2012 2/19 Content 1. INTRODUCTION .................................................................. 3 2. SURVEY METHODOLOGY....................................................... 3 QUANTITATIVE RESEARCH – TELEPHONE INTERVIEWS .................................................................... 3 NOTES ON THE PROCESS ................................................................................................................... 4 3. LIMITATIONS/OBSTACLES OF THE SURVEY ................................ 5 4. SUMMARY REPORT .............................................................. 6 4.1. GRANTEES PROFILE .................................................................................................................. 6 4.2. ACHIEVEMENTS AND CHALLENGES ............................................................................................ 7 4.3. G-PAC SUPPORT ..................................................................................................................... 10 4.4. G-PAC IMPACT ....................................................................................................................... 14 4.5. RELATIONS WITH MEDIA AND PUBLIC OFFICIALS ..................................................................... 15 4.6. NETWORKS AND COALITIONS ................................................................................................. 16 4.7. OVERALL ASSESSMENT OF G-PAC ............................................................................................ 18 4.8. RESUME .................................................................................................................................. 19 ACT Research / October, 2012 3/19 1. Introduction The given document is the technical report on conducting G-PAC Survey. The report is prepared by ACT Research for IBTCI. The main goal of the survey was to evaluate the satisfaction, effectiveness and support of the G-PAC program. G-PAC helps organizations to improve their public policy analysis and form cohesive networks in order to garner better access to the media, political parties, and community constituencies. Through a partnership with the Think Tank Fund of the Open Society Institute, the G-PAC Think Tank Support Program will strengthen the capacity of think tanks and policy-oriented NGOs to develop and present policy options. The Citizens’ Advocacy Assistance Program will provide a combination of demand-driven technical assistance and small grants. The Partnership for Change – Networking and Advocacy Program will strengthen NGO networks’ advocacy skills and ability to connect with the media, think tanks, and citizens. G-PAC will strive to improve the enabling environment for the civil society sector by partnering with the International Center for Not-for-Profit Law (ICNL) and the Civil Society Institute (CSI). The detailed methodology of the project is described below in the relevant chapters of the report. 2. Survey Methodology Quantitative Research – Telephone Interviews Instruments The final instrument for the survey was provided by the client in English. The instrument was translated in Georgian by ACT translators and had undergone several corrections, e.g. skip questions identified and minor adaption of the text into Georgian was done. The methodology of conducting the survey was suggested by client. Quantitative research methodology was chosen for the survey, telephone interviews were used as method of data collection. Telephone interviews enabled ACT to reach the respondents on the list in very short time period and fit the tight timeline of the survey. ACT Research / October, 2012 4/19 Target Groups and Sample Target group of the survey were the beneficiaries of G-PAC program. The list was provided to ACT by the client. Please refer to Annex # 1 for the detailed list. ACT has corrected the list in order to facilitate the fieldworks process. Notes on the process Training of Field Personnel Fieldwork staff training was conducted according to existing ACT procedures. Company has developed the module for general training, which is compulsory for every interviewer. Training module is developed by research specialists, field work department, trainer and company HR. After passing the general training module interviewers study: • General communication skills; • Techniques of interviewing; • Main methods and techniques of the research Training module also includes practical parts. Every candidate has testing and after successful completion of the test is provided with the status of interviewer. Before starting specific survey interviewers are selected according to their experience. Trainings for the specific survey and the instrument were performed by the Project Manager and Field Manager. Interviewers were instructed on survey theme (objective and tasks) and Survey instruments. During the fieldworks project and field manager were actively involved in the process, in order to eliminate any technical problems and make fast decisions. Fieldworks The fieldworks were done during 3 days, 29th of September, 2012 and 2nd and 3rd of October, 2012. The table below shows the statistics of the survey conduction: Type of Database Number of Unique Attempts Made Number of conducted Interviews Methodology Grantees 78 28 Telephone Interviews ACT Research / October, 2012 5/19 In total there were up to 63 attempts were made in scope of the G-PAC survey, since some persons in the list were duplicated. For the duplicated entries the interviewer asked the respondent whether they would agree to fill out different questionnaires regarding different grants they received, but unfortunately the telephone interview already took around 15 minutes, so no one agreed to participate in survey twice or three times. ACT conducted only 28 telephone interviews with those beneficiaries who satisfied the selection quota and agreed to take part in the survey. The statuses of each attempt of contacting the respondents are described in table below: Status G-PAC Grantees Interview was conducted 28 Number Busy 3 Does not satisfy selection quota 4 Duplicated Entry 15 Foreigner Respondent 2 Interview was terminated 3 No answer 5 Number is not registered 1 Number is out of coverage area 6 Refusal 5 Technical problem/Impossible to contact 4 Just started the project, can not evaluate 1 Out of country 1 Fieldwork Quality Control Almost 40% of the telephone interviews were controlled by the Project manager and the field manager of the survey. No serious violations occurred during the fieldwork quality control process. 3. Limitations/Obstacles of the survey ACT worked hard in order to achieve maximum quantity of the interviews, but unfortunately considering the duplicates and other database errors, there were only 28 interviews conducted. Due to lack of time and tight timeline, the piloting of the final instrument provided by the client was not considered. ACT Research / October, 2012 6/19 4. Summary Report In the chapter below is described the results of telephone interviews, the whole target segment consisted of 78 respondents, although the interviews were done only with 28 respondents. The results are described based on opinion of respondents who participated in the survey and does not necessarily represent the opinions of whole target segment. 4.1. Grantees Profile More than half of the organizations have been involved in their business for more than five years (See chart #1). Chart # 1. Activity in the business (N=28) ACT Research / October, 2012 7/19 Half of the grantees organizations operate regionally (see the chart # 2) Chart # 2. Geographical Coverage (N=28) 4.2. Achievements and challenges As for the achievements that the organizations have reached thus far, they are intensely diverse and all the stated answers are connected to the specifics of the organization, please see the detailed table # 1 below; the number of the answers to the question exceeds the number of the interviewed respondents, since multiple responses were acceptable to the question: Table # 1. Achievements of the project (N=39) activation of the society 1 person was bought an equipment, which filters water 1 person cars drive by other ways in villages 1 person performance was done 1 person theme was regarding general problems 1 person protection of interests of socially insecure persons 1 person was created organization of social security 1 person infrastructures rehabilitation 1 person brought requests to local government 1 person projecting of forest`s rules/monitoring about ecology 1 person protection of interests of population who use forest 1 person ACT Research / October, 2012 8/19 monthly social aid 1 person help to homeless people 1 person sewerage does not flow in Tbilisi sea any more 1 person parks property program was studied 1 person all owners got legal compensation 1 person informing private owners 1 person helping in property registration 1 person library in Khurvaleti refugees settlement 1 person the film was set about problems of refugee children 1 person state security included our work in their works 1 person spread the information about the elections and majority leaders’ programs 1 person Electrical station of Tkibuli is not used daily 1 person we had meetings with refugees and with insurance companies 1 person communication between government and population was held in high￾mountains villages 1 person potable water is supplied by schedule now 1 person bringing new regulations, which consider population defending 1 person ip-organizations had strategic plan about the local self-governance , which needed money from local settlements 1 person the attention was increased towards the NGO in Guria 1 person financed research 1 person people were informed 1 person assistance 1 person collaboration with agencies 1 person we made inquiries and wrote a report about this 1 person construction of airport was stopped temporary near Kolkheti park 2 person offering secure alternatives 1 person ministry accredited our program 1 person media outlets are more active 1 person The main challenges that the organizations face nowadays are mostly the accessibility of the resources from the donors and accessibility of the information. Government institutions also seem to have restrictions to accessibility of the resources for the organizations. The statements with low percentages are grouped in ‘Other’ and include: • Fluctuation in staff • Financial terms • Protection of human rights • Lack of communications • The low does not operate as necessary • New technologies ACT Research / October, 2012 9/19 The number of the answers to the question exceeds the number of the interviewed respondents, since multiple responses were acceptable to the question. The chart # 3 shows the distribution of the percentages of the challenges: Chart # 3. Challenges (N=51) Except the challenges that the organizations face, other external factors also influence the work of the survey participants. Most important factor named by the respondents is the political situation of the country, socio – economic factors and poverty also seem to play a role for the half of the interviewed organizations. The statements with the low percentages are grouped in ‘Other’ and include: • Lack of information; • Communication with donor organizations; • Specifics of the region. The number of the answers to the question exceeds the number of the interviewed respondents, since multiple responses were acceptable to the question (See chart # 4). ACT Research / October, 2012 10/19 Chart # 4. Influencing Factors (N=44) 4.3. G-Pac support Vast majority of the respondents declared that G-Pac is supporting their organizations in mitigating the risks and the challenges of their project (see chart # 5). Chart # 5. Mitigating the risks and challenges by G-Pac (N=28) As for the support that the organizations have named as the support from G-Pac includes thematic trainings, grant itself and the capacity building. Other statements with low percentages are grouped in ‘Other’ and include: • Advocacy support ACT Research / October, 2012 11/19 • International Mentors program • Net working The number of the answers to the question exceeds the number of the interviewed respondents, since multiple responses were acceptable to the question (See chart # 6). Chart # 6. Support of G-Pac (N=61) Respondents of the survey were asked to evaluate the improvements of their organization as compared to the situation before G-Pac support. The improvement that is evaluated by most of the organizations as extremely positive is connected to the advocacy activities, according to the respondents their advocacy activities result in concrete actions, also they are more focused on what they want to achieve. As for the negative changes, only few respondents evaluated statements as regressive, the authorities are asking us for advice more often (N=1) and we receive more funding from the authorities (N=3); the positive improvements are highlighted in pink and negative improvements are highlighted in grey in the table below (see table #2). ACT Research / October, 2012 12/19 Table # 2. The evaluation of the improvements Yes, definitely Yes, a little No change No, it is worse N/A Don’t' know We are more focused on what we want to achieve 82% 11% 4% - 4% - We have more professional expertise in our institution thanks to trainings we attended 75% 4% 14% - 7% - Advocacy initiatives are now better designed 64% 21% 11% - 4% - Our advocacy messages are more concise and objective 79% 11% 4% - 4% 4% We have skills to conduct better research on a topic of interest 71% 11% 4% - 4% 11% Better understanding of priorities and interests of the target audience 79% 11% 7% - 4% - The number of research studies increased 64% 18% 4% - 14% - the quality of our studies improved 71% 14% 4% - 11% - Our advocacy activities result in concrete actions 82% 11% 4% - 4% - We have a better strategy 75% 18% - 4% 4% We have more competent staff 61% 11% 21% - 4% 4% We have better leadership 71% 11% 11% - 4% 4% We have more members/clients/supporters 79% 11% 4% - 4% 4% We have more funding sources* 57% 25% 7% - 4% 4% We have more moral support from the community 79% 7% 4% - 4% 7% We have better relationship with the authorities* 57% 14% 14% - 7% 4% The authorities are asking us for advice more often * 36% 11% 32% 4% 11% 4% We receive more funding from the authorities * 4% 4% 50% 11% 18% 11% We have better cooperation with other NGOs 79% 11% 4% - 4% 4% We are more well-known and respected in society 64% 11% 14% - 4% 7% ACT Research / October, 2012 13/19 Yes, definitely Yes, a little No change No, it is worse N/A Don’t' know Our mission and vision are more clear 71% 7% 14% - 4% 4% We have better capacities for strategic planning and management 54% 29% 11% - 4% 4% We improved our monitoring protocols and systems 57% 25% 7% - 7% 4% We have improved human resources protocols and systems 46% 29% 14% - 7% 4% (* indicates that the respondents did not reply to the question with the answers offered by the interviewer and gave the other answer, like funding is necessary; we wrote letters but did not receive feedback, others took away materials prepared by us. Such answers does not exceed 4%, N=1) Respondents also evaluated the results achieved by their organization since the start of G-Pac support. The most positive result achieved by the organizations according to the survey participants is assisting/inspiring people to address their issues. The field that has not changed since the G-Pac support is the government law, policies and services; the positive improvements are highlighted in pink and negative improvements are highlighted in grey in the table below (see table # 3). Table # 3. The evaluation of the results achieved Yes, definitely Yes, a little Average No change No, it is worse N/A Don't know We have inspired/assisted people to take action to address their issues 79% 7% 4% - - 4% 7% The authorities have considered our suggestions 64% 7% 4% 11% - 7% 7% The government law/policies/services have improved 39% 14% - 25% 4% 14% 4% The situation of our target group has improved 61% 18% 4% 11% - 4% 4% ACT Research / October, 2012 14/19 4.4. G-Pac impact According to the survey results, thanks to G-Pac intervention skills, respondents have increased the number of the interactions, the quality of advocacy documents have raised. Grant seems to have the most positive impact on the advocacy conduction skills, according to the survey participants’ grant and capacity building, training, mentoring and consultations had facilitated the quality of the advocacy processes; the number of the answers to the question exceeds the number of the interviewed respondents, since multiple responses were acceptable to the question (see charts #7 and #8). Chart # 7. Improved skills via G-Pac (N=35) Chart # 8. Most positive impact of G-Pac (N=48) ACT Research / October, 2012 15/19 4.5. Relations with media and public officials Vast majority of the respondents think that the media follows their work, the media is also often invited to the organized events, and the organizations also stated that they quite often collaborate with the media in order to further their advocacy goals; the number of the answers to the question exceeds the number of the interviewed respondents, since multiple responses were acceptable to the question (See charts # 9 and # 10). Most of the survey participants are also quite satisfied with their relations with media. Chart #9. Relations with Media (N=37) ACT Research / October, 2012 16/19 Chart #10. Relations with Media (N=37) According to the organizations, the main challenges while communicating with media representatives are unwillingness to communicate, distrustfulness and prolonged processes. As for communicating with public officials, according to the survey participants, the best ways to communicate with them is having frequent contact and cooperation. The most used tools of communication are official documents, like policy papers and briefs and studies. Types of communication with public officials include phone, email and face to face contact almost equally. 4.6. Networks and Coalitions According to the survey results, more than half of the respondents think that it is much easier to advocate within the coalition and relevantly, results and policy can be different when coalition is advocating; The number of the answers to the question exceeds the number of the interviewed respondents, since multiple responses were acceptable to the question (see chart #11). ACT Research / October, 2012 17/19 Chart # 11. Choice between individual or coalition (N=31) As it appeared, respondents think that it is almost equally important to divide the responsibilities, select the right leader and the members while working in network, since these are the conditions that seem to be challenging for the organizations. Respondents also agreed that individual members’ social network is increased while working within a network of diverse CSOs in the coalition. Overall assessment of the G-Pac perception in the CSO community is more positive than the perception of G-Pac in local communities (see chart #12). Chart # 12. G-Pac in different fields (N=28) ACT Research / October, 2012 18/19 4.7. Overall assessment of G-Pac Survey participants think that the strongest sides of the G-Pac and organizations partnership is the support that the organizations get, also convenient relations bring some comfort to the partnership. It is also worth mentioning that the respondents could not think of any exceptional weaknesses of the partnership. The respondents’ opinions were also quite diverse while thinking about the improvements that could be done in the project implementation at the moment, according to the survey participants, better monitoring should be done in the future. Overall, the G-Pac grantees assess the cooperation only with the positive evaluations, all 28 respondents think that the cooperation is good, among these 28, 24 thinks that the cooperation is very good (see chart #13). Chart # 13. Overall assessment of cooperation (N=28) ACT Research / October, 2012 19/19 4.8. Resume According to the survey results, the respondents had various and diverse opinions regarding the achievements of their organizations thus far, all the achievements that were named by the organizations were very specific to the organization profile, although, it should be noted that all of them could name a concrete achievement. Since the start of G-Pac, organizations are more focused on what they want to achieve and they also think, that they have motivated and inspired people to address their needs, these can positively speak regarding the motivation of the organization, and itself the raised motivation can lead to more achieved goals. As it can be seen from the survey results, G-Pac has only positive evaluation from the grant receivers, grantees found difficult to name any weak sides of the partnership. Capacity building, training, mentoring and consultations are assessed as the most impressive and effective inputs of the G-Pac program. Evaluation of projects implemented by G-PAC in Georgia Report of qualitative research Version: 1.0 Prepared for IBTCI by ACT Research October, 2012 Tbilisi, Georgia Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 2/ 30 Contents 1. Report №3 – Batumi CSOs _____________________________________________________ 3 1.1 Evaluation of the trainings funded by G-PAC ___________________________________ 3 1.2 Preventive and propitious factors in project implementation _______________________ 6 1.3 Evaluation of cooperation with G-PAC ________________________________________ 8 1.4 Evaluation of civil society __________________________________________________ 10 1.5 Recommendations regarding how to assist civil society development ________________________________________________________________ 12 2. Resume ____________________________________________________________________ 15 3. Annex 1 - “Step forward” ______________________________________________________ 16 4. Annex 2 - “Borjgali” __________________________________________________________ 21 Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 3/ 30 1. Report №3 – Batumi CSOs Notes on the process For the organization of focus group discussion in Batumi, the client provided us with the list of CSOs of Batumi. In the list there were only two CSOs: “Step forward” and “Borjgali”. In the process of the recruitment it was revealed that only two representatives from “Step forward” were participated in G-Pac project and 4 representatives from “Borjgali”. In order to get deeper information, it was agreed with the client to conduct one mini focus group discussion with “Borjgali” representatives and one dual interview with “step forward” representatives. On dual interview with “step forward” representatives one respondent (project coordinator) could not attend on interview, so an in-depth interview was conducted with “step forward” director. 1.1 Evaluation of the trainings funded by G-PAC The given chapter presents evaluation of projects by representatives of Batumi CSOs “Step forward” and “Borjgali”. Evaluations were mainly made on the projects completed at the given moment. Organization „Step forward“ Main activity of organization “Step Forward” is working on problems of children with hearing disabilities. The project of “Step Forward” funded by G-PAC aimed to advocate the rights of children with hearing disabilities. Specific goals of the project were: (1) Proving the significance of the given issue (conducting a survey). (2) Establishing an initiative group of parents. As stated by the head of organization, main goal of the project was successfully achieved – representatives of local government and and the respective ministries saw how significant are the problems of children with hearing disabilities. They expressed a will to work on this issue; moreover, concrete steps were made apart from expressing the will. Namely, post￾operation rehabilitation program came into force. As stated by the respondent, local authorities clearly saw the economic effect of funding the Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 4/ 30 prevention. “It should be noted that at the meetings there was fixed the will of the state to work on this subject. The Minister of Healthcare said that they must carry out this project in the future. It was an unexpected result to us. Other ministries also expressed their willingness to help including the Ministry of Education. It was a good result. We are preparing concrete recommendations already. The main result is that they saw a real problem that it is a problematic issue which needs to be solved immediately and not to be delayed. The result was also that we showed the Minister of Healthcare how much a problem is when a child is provided with a hearing aid and then the rehabilitation is not financed. This question was also raised by the chairman of the union of the deaf. The program financing the postoperative rehabilitation has already started. It was not done at all before. This result was reached not only with our participation but we played a certain role in these results.” [Organization „Step forward“] As stated by the respondent, the following project of G-PAC which is ongoing at the moment implies elaboration of concrete action plan and concrete recommendations of experts. Besides, early diagnostics which was available in Tbilisi only will be established in maternity homes. This process will begin from cities and will cover regions afterwards. “The purpose of this project is to carry on the work started by us. Then it was a work group. Now there must be put an action plan with experts’ involvement but we must give the experts a three-year plan in concrete steps in more details.” [Organization „Step forward“] Organization “Borjgali” Borjgali’s main field of activity covers eco-migrants. Organization also works on issues like education, development of volunteering and environment protection. Organization received 2 grants from G-PAC – one of it is completed and the other one is ongoing. Both projects refer to eco-migrants. Main objective of the projects were to improve socio-legal condition of eco-migrants. As stated by the head of organization this issue is very problematic in Georgia. As official statistics say, 40 000 people are evicted only from Adjara. Additionally, there are 35000 families damaged as a result of natural disasters, 11000 of which live under the risk of losing residential place and physical destruction. Despite such statistics, as the respondents declare, the state did not pay attention to this issue. Moreover, definition of eco-migrants was not defined on legislative level. Concrete goals of first project were: Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 5/ 30 • to study the given topic comprehensively • To identify the problems and prioritize them. Main accent was made on Adjara highlands in the first project. Group of project carried out detailed registration of families living in villages according to which problems of eviction were analyzed. Apart from studying the problems, legislative pack with concrete pattern and recommendation has been elaborated. The pack was sent to ministries. Besides, web-page was created where all kind of problems and information are accessible. A documentary movie was directed within the frames of project. Aim of ongoing project is to study the current condition of re-settled eco￾migrants. Re-settlement zones, 7-8 municipalities are studies at the given moment. Representatives of Borjgali expressed their opinions regarding the completed project. They believe that the project was rather successful. Success was conditioned by the following concrete results: • Issue of eco-migrants appeared in the report of Public Defender and separate chapter was dedicated to it • Local government and Mayor’s office significantly increased finances granted for eco-migrants (sum was increased up to 15 million GEL which is relatively much than sum granted for eco-migrants for the last 10 years) • Several families were evicted from dangerous zones and re-settled in specially constructed houses • Conducting effective information campaign – creation of web-page, activities carried out via social networks, preparing media reports, documentary movie (up to 30 reports were prepared on central TV channels, movie about eco-migrant children is to be directed). • Signing a memorandum with Employment Agency of Adjara according to which the agency will assist eco-migrants to attend different courses and find jobs. • Settling problems in newly re-settled villages (for instance, opening Georgian sector in school, envisaging a sum for ambulance building in budget). Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 6/ 30 “Relationship with Autonomous Republic of Adjara was simpler. This had a result in terms of financial support. Even one GEL was not envisaged for eco￾migrants from state. We cannot say that it is all our merit but we kept Autonomous Republic under permanent stress and 9 million GEL was appointed for building social residential houses and eco-migrants from the most dangerous areas were re-settled in these houses. Two houses are built in Batumi and 2 buildings are under reconstruction in Keda and Shuakhevi – up to 300 families will be ensured with homes. Besides, Mayor’s office envisaged some money for eco-migrants.“ [Organization „Borjgali“] As the respondents believe making an accent of media campaign was very important as increase of awareness regarding the issue assisted a will of government to cooperate. Special trainings were held in order to improve the qualification of journalists and professional coverage to be prepared. “We held trainings for journalists as well because media representatives made very wrong accents while preparing their reports. Terminology, demonstrating problems in a right way, making them visible for society – all these issues were very complicated for society. It is complicated in every field by particularly in ecology. We had to explain definitions of basic terminology, there are categories to separate eco-migrant from a victim of natural disaster. After trainings, the journalists prepared very interesting reports and number of articles also increased.“ [Organization „Borjgali“] As the respondents declare, they still function as recommendation providers for journalists. “Not only one activity has been covered on First Channel and we often get calls regarding recommendations how to cover this or that accident.” [Organization „Borjgali“] 1.2 Preventive and propitious factors in project implementation As stated by the head of organization „Step forward“ project implementation was assisted by several factors, namely: Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 7/ 30 • Appointing a research consultant by G-PAC who held the trainings and consultation for project group. As the respondent declares, they had not had any research experience before this project. It was qualified and competent survey that plaid a decisive role in displaying the instancy of issue. “It was effective. If we had not had a G-PAC consultant we would have had difficulties. Mentoring was very good. The consultant gave us training and kept helping on-line.” [organization „Step forward“] • Good communication from authorities which was based on high level of trust towards the organization (they were trusted as qualified organization working on these issues). “There were no problems from authority circles. They knew us. We had normal relations and they trusted us. When they needed information they applied to us because they knew that we had a data base and considered us to be a reliable source of information.” [organization „Step forward“] Obstacles, preventive external factors in project implementation named by the head of organization: • Frequent change of the authorities (because of this, they had to re-prove the importance of issue to new authorities) • Post-election period (namely, decision making process is prevented) “One of the difficulties was a frequent change in the authorities. When a new man is put in charge explanation must be given again. There was a case when a scheduled meeting was to be postponed because of the resignation of the minister… Because of the post election period it is impossible to make a decision at the current stage. Everything has been uncertain so far until the situation is put in order.” [Organization „Step forward“] Small budget of the project was also named as an obstacle. As the respondent declared, qualitative and quantitative components of the research could not be implemented within the given budget, only quantitative research was conducted. However, it was also noted that the research was successfully Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 8/ 30 conducted thanks to a consultant and despite small budget, it was still very effective. One of the obstacles were different financial accounts (“USAID had different demands”) though as the respondent declares, representatives of G-PAC made everything clear regarding this issue and it did not have any negative impact on project implementation. As stated by the representatives of Borjgali, a minor project funded by EU which was carried out earlier enabled the project team to analyze external and internal risks of project implementation. As the respondents state, in fact there were no internal obstacles and external obstacles mainly referred to relationship with state authorities. As they believe, relationship with local government was well-regulated while there were obstacles in communicating with central government as they were rather rough in relationship. By the evaluation of the respondents, relationship with local government was simplified by the fact that they are closer to society, they are members of coordinating group. However, it was also noted that many decisions depend on central government which is very problematic and prevents timely solution of important issues. “Communication on local level is much easier as they are closer to society.” [Organization „Borjgali“] “The problem is that many things that could improve the condition of eco￾migrants are decided by executive government. As for the things that can be decided by local government, I can’t say perfectly, but they do their best to fulfill them.” [Organization „Borjgali“] 1.3 Evaluation of cooperation with G-PAC As stated by the representatives of “Step forward” and “Borjgali” cooperation with G-PAC was very comfortable and based on partnership. There were no obstacles or problems from G-PAC. The respondents think that such positive attitude was conditioned by the factors such as: • Comfortable situation (representatives of G-PAC made reactions on questions and concrete problems of grantee organization in timely manner) Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 9/ 30 • Professional consultants (consultant worked with project team in online regime and helped them to implement the research in qualified manner) • Qualified trainings from Partners Georgia • Assisting to gain European experience (visit in Slovakia) “The collaboration with G-PAC was very positive. I am very content. I did not know this organization at all. I am very happy and comfortable with working with them. They helped me greatly in many things. I did not think that by such a low-budget project I would be able to change the attitude to this problem.” [Organization „Step forward“] “A very positive moment was attaching a consultant, their timely reaction to our question. We are also very content with trainings. Trainings were conducted by “Partners Georgia”. The consultants were great. Our concrete problems were discussed and recommendations were given.” [Organization „Step forward“] “Our relationship is different from relationship of donor and grantee. This is our second project and we did not have a feeling that this organization is a donor. It is more partnership than relationship of donor and grantee. In partnership I mean cooperation, sharing information.” [Organization „Borjgali“] “Study visit was carried out in Slovakia and we were engaged in this process as well. They helped us to gain European experience and it can be said that G-Pac is more a partner than a donor.” [Organization „Borjgali“] As the head of “Step forward” declares, cooperation with G-PAC made a positive impact on them in terms of changing their perspectives. Namely, organization members had different understanding of their role before – they took too much responsibility on them. Cooperation with G-PAC made them see that main thing is to make influence on government, make the government carry out concrete activities regarding sensitive issues for society. Creation of working groups appeared to be effective way which increased the participation of different institutions (government, local government, society). This approach enabled the organization representatives to improve their skills in writing project proposals - they won a very big competition of EU). A good thing was that these projects (including even those which were not financed) changed our vision. If we were taking responsibilities on many Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 10/ 30 things before, these demands of G-PAC showed us that we must act on the authorities, make them do. Our task is to force the government to do.” [Organization „Step forward“] 1.4 Evaluation of civil society Awareness of civil society The respondents think that a notion of civil society is incorrectly interpreted. NGO sector is often equated with civil society and the government perceives everything in it. As “Borjgali” representatives say, “Everything is civil society minus government”. Civil society today Representative of organization “Step forward” thinks that our civil society is more refined compared to neighbor post Soviet Union countries. Georgian civil society has been developing step by step for the last 10 years. Though there are many things to be done in this connection. This opinion is shared by Borjgali representatives. However, they add that some kind of stagnation, prevention of development dynamics is noticed lately. The respondents think that this is caused by the state policy. They believe that the government made an accent of the development of state and not the society. “dynamics of development stopped in recent period. Growth of civil consciousness is crucial. State policy was the reason of stopping the development as the state made an accent of development of state and not society. These two might be the same for someone but not for me. Let’s develop state institutions and the society will develop on its own but the thing is that society is in shadow and not under development.” [Organization „Borjgali“] Example of civil society development named by the head of organization “Step forward” is pre-election surveys according to which there is grown interest of citizens towards the programs presented by political parties. Another example of society development is that as it appears despite intense social background people are worried about humans right as well. Interest towards citizens’ constitutional rights has also increased. Engagement of population in interesting topics is gradually growing. The respondent illustrates this fact by reaction of parents representing Union of the Deaf in Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 11/ 30 connection with dismissal of indecent chairman. “The investigations made in the period of elections showed that people are interested in programs; their interest in them is obviously increased. Though people are worrying about social problems but it turned out that human rights are also their problem. For example human rights are very important for 60 % of the listeners of radio Freedom. [Organization „Step forward“] “In Adzharia a chairman of the deaf union was an official of the old ОБХСС (department of struggling against the theft of socialist property). The parents of the union raised a question. They said: if we are the union, we are entitled to decide who will be the chairman of our union. This example shows that civil society is gradually growing. [Organization „Step forward“] The representatives of “Borjgali” also talked about the development of civil society compared to the last 10 years. They believe that there is some progress though there is still much to do in this direction. Capital city is in better position compared to regions. Nowadays students have more opportunities to increase their civil knowledge (trainings are held) and be actively engaged in civil life (there are centers for volunteers). As the respondents note, their activity was limited to participating in political rallies while their area of activity is broader now. Approach towards working in NGO has also changed. As the respondents declare getting in NGO sector for the last 10 years was perceived as kind of springboard for appearing in government – that has somehow changed. “Activity of NGOs was expressed by taking part is some kind of rallies. But now, if we speak about the example of Batumi, all NGOs that are formed in the city carry out activities directed mainly to civil society development.” [organization „Borjgali“] “We implemented a project on volunteering last year – this project was the first one conducted in Batumi and one of the exceptional in Georgia. More than 100 students and pupils of senior year were recruited and trained. Since then, they are actively engaged in different activities and are not occupied only with political rallies. They were just in classroom before the trainings and their area of activity has widened – they started looking for new trainings and exchange programs.” [organization „Borjgali“] Chairman of organization “Step forward” also talked about the role of NGOs. As the respondent believes, population is not well aware of activities conducted by NGOs – this is proved by G-PAC’s survey. Attitudes towards NGOs are mainly negative – they have image of “grant-consumers”. The Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 12/ 30 respondent thinks that actions need to be made in this direction both by NGOs and whole society. Leaders of civil society As for civil society leaders, representative from “Step forward” named Zviad Koridze from media, though as the respondent believes it is hard to name apolitical, neutral leaders today. “From the leaders I recall Zviad Koridze – from media. “Association of young jurists of Georgia”, but it was very unpleasant to me when I heard that sums were transferred for party purpose. I used to rely on their conclusions. Gela Nikolaishvili used to protect prisoners’ rights but on the previous day he protected the Asaval-Dasavali and I was shocked. I could not recall a neutral person.” [Organization „Step forward“] As the respondents from “Borjgali” think, leaders could not be born from today’s society. Besides, there is lack of young leaders. “There is a lack of young leaders In general. I don’t mean political leaders, I speak about public leaders. They should become leaders in their field of activity, for instance in science. Society, masses of people need leaders to whom to follow. Society does not create leaders anymore and respectively, it stopped development.” [Organization „Borjgali“] 1.5 Recommendations regarding how to assist civil society development As the respondents believe, concrete, effective actions need to be conducted by the side of government, donors and public organizations in order to assist the development of civil society. Government should not take vertical decisions. It is important that representatives of society were engaged in decision making process. As the respondents believe the government needs to change accents – “We are building, not I”. This will increase the contribution of society in building our country. As the NGO representatives declare, the government needs to work in the following directions: • Establishing right educational system, introducing correct reforms. Namely, civil education system needs to be refined in terms of school and universities. The respondents believe that school has lost its main function Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 13/ 30 nowadays – it does not bring up a citizen. Graduates from school do not have a benchmark and proper education. “I don’t see school’s function nowadays. Giving an education is not a sole function of school – but bringing up and developing a citizen. Schools do not bring up a citizen nowadays but a person without anything - no education or orientation.” [Organization „Borjgali“] • Strengthening a cult of family as family bears a citizen and patriot. “mental changes must take place at home. Family forms the priority and citizen. Notion of family has faded as well. Family is a main cell of any society. Government should try to create the cult of family which ensures the sustainability of country.” [Organization „Borjgali“] • Solving legislative problems. Namely, regulation of law on social entrepreneurship (it will help NGOs to widen their area of activity and they will not be so depended on foreign donors). Besides, tax relief for Georgian funds (as the respondents state, Georgian funds are not established due to legislative complications. Such legislative amendments will assist them). “Other types of legislative warranties need to be created for this purpose. NGOs should not be depended only on foreign donors. Mentality of “grant￾consuming” will be permanent. On the given stage, there is no other source rather than grants; social entrepreneurship is regulated in such a complex way that you may get stuck in any case. Besides, Georgian funds are not established; there are legislative obstacles, taxation and so on. Certain privileges should be given in terms of tax policy in order to develop NGO sector.” • Legislative amendments regarding volunteering. “In general, some amendments should be made in legislation even in terms of volunteering activity. Volunteering is one of the most active and important form of civil society development. There are rather serious taxation obstacles while employing a volunteer. “ As for donors, it is important that they should carry out the following activities: • Assisting with surveys. Besides, monitoring of reforms and human right protection Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 14/ 30 • Strengthening self-government, public organizations with mentoring, studying, trainings so grantee organizations will be able to conduct more effective management of projects. • Assisting advocacy. As the head of “Step forward” thinks, assisting advocacy will help to obtain concrete positive outcomes. • Strengthening organizations in terms of management so they will not be depended only on grants. • Better fitting of priorities of donors to priorities of country. “As for donors, we, NGOs have to fit to them and very important problem may be left unnoticed. It is very difficult for foreigners to see the existing problems in Georgia in a way as we see them.” [Organization „Borjgali“] NGOs should work in the following directions: • Freeing them from sympathies towards political parties – NGO sector should permanently be up to the notch. There should be criticism and permanent observation under any government. • Development of management - this way they will not be depended on grants and will not stop functioning after the grant is completed. • Strengthening relationship among each other and intensifying cooperation on problematic issues- by doing so they will assist opening the closed society. Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 15/ 30 2. Resume As the qualitative research shows, representatives of NGOs from Batumi funded by G-PAC evaluate outcomes of the projects implemented by them as well as cooperation with G-PAC. They think that successful implementation of the projects was assisted by G-PAC’s cooperation and partnership. Communication with G-PAC, consultants appointed to the projects, trainings were positively evaluated. As for obstacles related to project implementation, these factors were named: frequent change of authorities, pre-election and post-election period, complications in relationship with central government namely, rough communications (it is different in terms of local government authorities; relationship with them has been relatively more effective and fruitful. This fact was named as one of the factors that assisted the project implementation). As the respondents believe, civil society is more developed now than it was 10 years ago. This is expressed in grown interest of society towards human rights protection, improvement the level of knowledge regarding citizens’ rights. However, it was also noted that there is a problem of some kind of stagnation, termination of development dynamics which is conditioned by rough state policy, weakened accents and assistance on society development. Lack of society leaders was also named as one of the factors that prevent the development of today’s civil society. NGO representatives place the most important role in the development of civil society to the government though setting right priorities from donors and strengthening management of public organizations are also very important. Creating the proper environment for the development of civil society is the most important thing that the government can do. Namely, it implies elaborating/refining legislative regulations (including regulation of law on social entrepreneurship), tax relief to assist establishment of Georgian funds, implementation educational reforms in schools and universities to increase the level of civil education. Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 16/ 30 3. Annex 1 - “Step forward” Notes of the in-depth interview with Manana Inaishvili Basic activity of the organization “The organization was founded in 2003. It is a regional organization the main goal of which is to increase the accessibility of healthcare for people, to protect rights of primary healthcare, awareness of people about programs. One of the main orientations in Adzharia is to support children and youth with hearing deficiency… G-PAC projects just concern it. We started work on this subject with an EU project. It is already the third grant.” About G-PAC grant “We received the G-PAC grant last year. With the means of this grant we managed to carry out a research. G-PAC attached us a consultant which helped us greatly in conducting a professional research. We reached very interesting results which we submitted to the state structures and the local government. “ “The above mentioned grant concerned advocating rights of children with hearing deficiency. The research performed a role of evidences. The concretetask of the project was to confirm that this subject was urgent. We also worked in the other direction, created an initiative group of parents. This group involved state structures, media, non-governmental structures, parents. We had had contacts with parents before but by the help of this group we were able to work for common aims, to make parents participate in these processes. As a result the parents became more active. They said; if you are taking care of our children so much, we will not be passive. Our aim was to show the parents ways of solving their children’s problems and together with the authority’s representatives to involve them in planning of the programs.” What was a concrete result of the project and to what extent was it reached “The concrete result of the project was to carry out a research and to create an initiative group. It should be noted that at the meetings there was fixed the will of the state to work on this subject. The Minister of Healthcare said that they must carry out this project in the future. It was an unexpected result to Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 17/ 30 us. Other ministries also expressed their willingness to help including the Ministry of Education. It was a good result. We are preparing concrete recommendations already. The main result is that they saw a real problem that it is a problematic issue which needs to be solved immediately and not to be delayed. The result was also that we showed the Minister of Healthcare how much a problem is when a child is provided with a hearing aid and then the rehabilitation is not financed. This question was also raised by the chairman of the union of the deaf. The program financing the postoperative rehabilitation has already started. It was not done at all before. This result was reached not only with our participation but we played a certain role in these results. The second G-PAC project started in August. The purpose of this project is to carry on the work started by us. Then it was a work group. Now there must be put an action plan with experts’ involvement but we must give the experts a three-year plan in concrete steps in more details. In August we worked intensively. The Minister promised us to finance screening. It has not been easy. The Ministry is not allowed to acquire apparatus, it can only finance service. The apparatus must be acquired, it is expensive, and service is cheap. We have got our apparatus which we obtained early from the EU. This apparatus is meant for early diagnostics in maternity hospitals. This was only carried out in Tbilisi and nowhere else. In Adzharia only we work on this subject. We wish this work to be performed in cities and regions at the same time, though it can’t be done at this stage. Nothing will be gained unless the town council is met. They are very much interested in this subject.” Difficulties at the implementation stage of the project “The project was low-budget; hence we could not carry out a complex research – we wanted to perform a qualitative and then a quantitative research. We were unable to do this because of lack of the budget. We only did a quantitative research. It was effective. If we had not had a G-PAC consultant we would have had difficulties. Mentoring was very good. The consultant gave us training and kept helping on-line…” “…There were no problems from authority circles. They knew us. We had normal relations and they trusted us. When they needed information they applied to us because they knew that we had a data base and considered us to be a reliable source of information. One of the difficulties was a frequent change in the authorities. When a new man is put in charge explanation must Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 18/ 30 be given again. There was a case when a scheduled meeting was to be postponed because of the resignation of the minister.” “…Because of the post election period it is impossible to make a decision at the current stage. Everything has been uncertain so far until the situation is put in order.” The biggest success of the project “The biggest success of the project is that we made them see how important is early diagnostics, how much economically effective is for the state to spend money at this stage and not on the further assistance. I hope to make new comers believe that the prevention early diagnostics is important. It is also important to socialize such people so that they will not feel being rejected during the whole life.” Assessment of collaboration with G-PAC “The collaboration with G-PAC was very positive. I am very content. I did not know this organization at all. I am very happy and comfortable with working with them. They helped me greatly in many things. I did not think that by such a low-budget project I would be able to change the attitude to this problem. One of the difficulties can be named a different financial accountability (USAID had different demands), though we were clarified in it from the very start. A very positive moment was attaching a consultant, their timely reaction to our question. We are also very content with trainings. Trainings were conducted by “Partners Georgia”. The consultants were great. Our concrete problems were discussed and recommendations were given. A good thing was that these projects (including even those which were not financed) changed our vision. If we were taking responsibilities on many things before, these demands of G-PAC showed us that we must act on the authorities, make them do. Our task is to force the government to do. Now we have received a new grant from the EU and I felt how much effect the projects of G-PAC had on me. A contest for receiving a grant was big. We got a high assessment. These projects had really a positive influence on me. There must be a process of planning and the participation. If earlier we were told it Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 19/ 30 is good if we do this, the creation of a work group made me see that they were becoming the participants of this process themselves and were saying that they would do it. So we worked together with the mayor’s office. Actually we wrote the project together.” Assessment of civil society “The research of G-PAC showed that people are not aware of NGOs’ doings. They have a negative attitude towards them – a “grant eaters” image. Perhaps the society must work more in this direction. The investigations made in the period of elections showed that people are interested in programs; their interest in them is obviously increased. Though people are worrying about social problems but it turned out that human rights are also their problem. For example human rights are very important for 60 % of the listeners of radio Freedom. Comparing with neighboring countries our civil society is ahead. The neighboring countries are still the Soviet Union. It means that we are not going on the circle; we are going ahead. For example there are often mentioned constitutional rights that were never mentioned earlier. In Adzharia a chairman of the deaf union was an official of the old ОБХСС (department of struggling against the theft of socialist property). The parents of the union raised a question. They said: if we are the union, we are entitled to decide who will be the chairman of our union. This example shows that civil society is gradually growing.” What must be done to develop of civil society “People must feel themselves that their participation is important in building of the country. I do not want political notes but earlier I heard: I am deciding and I am not asking people. Now I hear again: “I will do it” and I don’t hear a phrase: “We will do it”. I am recalling Roland Raigan’s words. He was saying: “We will settle problems together. Population must feel that you are a participator. There should be criticism, self-government must become more active and such vertical decisions must not be made. Public organizations must be free from party sympathies. There must be criticism and constant vigilance. We must not relax our vigilance by the Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 20/ 30 motive that we are building a country now and have no time to waste otherwise we will get the same result that we have got. The main thing is mentality. For example I demand checks principally and not by a motive of winning. From donors it is very important to help with researches at the country level. It’s also important to strengthen monitoring of reforms. For example reforms were changed many times in healthcare and a lot of money was lost. From donors it is important to carry out researches and monitoring of reforms and human rights violation at the country level, it’s also important to strengthen the self-government and public organizations. I think the right direction is advocating. If advocating is continued, we might get concrete positive results. I often hear, why the new are not financed by international organizations, they are giving to their people, though it is not so. We did not know anybody in the EU. It’s perhaps good if they will strengthen mentors, mentoring, and trainings. It’s good to strengthen organizations so that they will manage their projects more effectively. Accordingly demands must be high. You are given money because they want a result from you.” Leaders of civil society “From the leaders I recall Zviad Koridze – from media. “Association of young jurists of Georgia”, but it was very unpleasant to me when I heard that sums were transferred for party purpose. I used to rely on their conclusions. Gela Nikolaishvili used to protect prisoners’ rights but on the previous day he protected the Asaval-Dasavali and I was shocked. I could not recall a neutral person.” Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 21/ 30 4. Annex 2 - “Borjgali” Table 1. List of people invited and participating in the survey (Batumi CSOs) Organization name Name Location Position Organization “borjgali” Ilia Guchmanidze Batumi Chair of organization “Borjgali” Organization “borjgali” Davit Bagrationi Batumi Executive director of organization “Borjgali” Organization “borjgali” Adel Saladze Batumi Eco-Omburdsman Organization “borjgali” TsialaKatamid ze Batumi Media-Manager Notes of mini focus group (representatives of “Borjgali”) About organization “We are organization “Borjgali”. Main area of our activity is Adjara region, though we have some projects outside the region. Generally, our field of activity includes displaced people including eco-migrants and main direction of our activity is youth sector. As for directions of youth sector, these are: education, development of voluntary activity, etc. These are our main activities and we take some actions in environment protection field as well. These are three main directions on which we actively work.” Project goals and objectives “We have two projects: completed and ongoing. This is the issue which was hidden and forgotten for years – this is a problem of eco-migrants. Main objective of the project was to improve socio-economic condition of eco￾migrants on the bases of effective state regulations in eco-migration field. No Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 22/ 30 one has worked on this project recently, though it was very topical. 40 000 people have been evicted from only Adjara during these years. This is an official statistics, though the number of evicted people is bigger. Additionally, there are 35 000 families damaged from natural disasters and 11 000 of them live under permanent risk – they need to be evicted. Eco-migrants are people who are evicted as a result of natural disasters or need to be evicted immediately. The problem is that number of eco-migrants in Georgia exceed 100 000 and despite this fact, government did not pay any attention to these people. Even their definition, the respective term is not defined in terms of legislative level. There is no law that could protect this people. Whole accent is made on IDPs who of course are important though both IDPs and eco￾migrants are displaced. Reason of displacing is different but they are displaced. There are social guarantees, legislative base in case of IDPs while eco-migrants are left without any regulations. Our goal was to initiate legislative amendments and establish structural politics, regulations regarding eviction need to be created even on two sheets of paper. This is a very complicated issue how migration should take place. As this topic was not studied, our primary goal was to study the issue comprehensively – how processes are carried out, what are the main problems i.e. identifying problems and making priorities from them. As for the first project, main accent was made on Adjara highlands – majority of eco-migrants are from Adjara highlands. In fact, we visited all the villages, registered the families, what are the problems in eviction process and analysis was prepared based on the obtained data. The second project enabled us to study the current condition of re-settled people. We visited areas of main resettlements – 7-8 municipality, we were in Tsalka and all main zones. Firs project paid attention to eviction problems and the other one specifics of re-settlement. Apart from studying the problems, pack of recommendations has been elaborated. We have worked on very serious and comprehensive legislative pack with its concrete pattern. For instance, our pattern of how to define the definition of eco-migrants. Experts were engaged and we provided the government both with analysis and recommendations regarding how to improve the situation and solve the problems. Systematized base in which the problems were reported was created within the second project. We created a web-page where all problems, information can be obtained. Problems in this field are easily perceivable for journalists and experts by means of our web. Systematized base was created according to Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 23/ 30 villages, what are the problems of each village with the respective photo￾video materials attached. We shoot a small documentary movie which reflects the reality and we can see the situation in 35 minutes. Visualization is very important on its side. Concrete recommendations were sent not only to ministries but to self-governments regarding how to control the existing situation of eco-migrants. Problems were defined in detail and we share this information via our web. Our web-page enabled us to illustrate these problems and donor finds it easier to evaluate the process of project… …Besides, our analysis was envisaged in the report of Public Defender – it was a successful case for us as separate chapter was dedicated to eco-migrants in the report. It happened by means of information provided by us. Special report was prepared, we are waiting for it. The biggest result and success is that there are two bodies central and self-governments – it is difficult to communicate on central level, but relationship with Autonomous Republic of Adjara was simpler. This had a result in terms of financial support. Even one GEL was not envisaged for eco-migrants from state. We cannot say that it is all our merit but we kept Autonomous Republic under permanent stress and 9 million GEL was appointed for building social residential houses and eco￾migrants from the most dangerous areas were re-settled in these houses. Two houses are built in Batumi and 2 buildings are under reconstruction in Keda and Shuakhevi – up to 300 families will be ensured with homes. Besides, Mayor’s office envisaged some money for eco-migrants. We could not achieve all the results we wanted. There are global goals which mean making a law though sum granted due to our effort is up to 15 million GEL which is relatively much than money granted for eco-migrants from budget for last 10 years. Our recommendations regarding eviction imply that people should be re-settled in villages and not in social settlements. But main thing is that people who were under permanent risk of burying alive at any second, examples of which took place in Adjara are out of this risk now. However, many people still face this problem. We place a particular accent of media campaign. The more popular the issue is, the more is the interest, the more willing is the government to cooperate. You may elaborate very good recommendations but it may not be a source of political dividends for government and officials may not cooperate with you. If you become louder, they become interested in this field. We arranged press tours; approximately 30 reports have been prepared within both projects on Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 24/ 30 central TV channels. We are in the process of creating a movie regarding eco￾migrant children. These children are in a very difficult situation and we hope that this movie will manage to increase the attention towards them and society will feel more responsibility. We also created a coordinating group covering all structures of Adjara and this group is engaged in solving the eco-migrants’ problems. At the same time, there are public organizations, representatives of mass media and eco￾migrants themselves. There is some kind of forum created for cooperation. We hold discussions and debates around all issues.” Challenges in the process of project implementation “We certainly take internal and external risks into consideration while elaborating on this project. We were aware of this issue very well. We had implemented a minor project with a grant issued by EU. So, we were aware of the issue and tried to avoid risks. We knew the state policy in this direction and planned activities maximally correctly. In fact, we have not encountered with internal risks and obstacles and external obstacles mainly implied relationship with governmental structures. We did not anticipate that we would achieve this result as we knew their position. The thing that could be managed beyond the project could not be done on central level, there were more problems. In terms of communication, there were several people who created a serious obstacle.” “Communication on local level is much easier as they are closer to society and there is a department of refugees and evicted people under the authority of Ministry of Healthcare. They even come to us and become members of coordinating group. Members of some other organization may not come but state officials always come and share their opinions. The problem is that many things that could improve the condition of eco-migrants are decided by executive government. As for the things that can be decided by local government, I can’t say perfectly, but they do their best to fulfill them. Both of our projects aim to have relations with society and make different groups interested in this topic. First of all, trainings were held with people who are directly related to eco-migrants, these are head representatives of villages. We explained how and in what forms to help the victims of landslides, though they had already done this before. Trainings were about responsive mechanisms in order to make them more active and fruitful.” Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 25/ 30 “We held trainings for journalists as well because media representatives made very wrong accents while preparing their reports. Terminology, demonstrating problems in a right way, making them visible for society – all these issues were very complicated for society. It is complicated in every field by particularly in ecology. We had to explain definitions of basic terminology, there are categories to separate eco-migrant from a victim of natural disaster. After trainings, the journalists prepared very interesting reports and number of articles also increased.” “We function as a recommendation-giver institution for journalists. For instance, we recommend where to send a journalist for report, which village is more convenient in this connection. Not only one activity has been covered on First Channel and we often get calls regarding recommendations how to cover this or that accident.” Role of government “Results of cooperation with government have been achieved though there is a distinct difference between central and local governments. As we are more oriented on Adjara due to first project, we did not encounter with any problems. They were really engaged and envisaged all our recommendations. Our work made an influence of social houses and all this was reflected in budget figures. “ The biggest success of the project “We speak about regulating of process; this is not a concrete and local problem. This is an improvement of process which will be followed by general results. Improvement of policy takes time; state position needs to be changed and so on. The first achievement I can speak about is that a very good information campaign was implemented. In fact, we took this issue from ground and made it topical. I think that media campaign was successfully implemented in all directions including internet– we used social networks, we were very active on Facebook. Second big success was money that was granted and it is not a thousand or two, we speak about millions. Another success was that this issue was covered in the report of Public Defender – it is a very important achievement. In fact, this issue was re-studied, problems were identified, topic was systematized and recommendations sent. Re￾studying this issue demands much time and resources. This is the biggest achievement.” Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 26/ 30 “As for recommendations, it is a global issue. Probably we will not manage creation of legislations. There were problems in concrete villages, for instance regarding irrigation canel. We sent our recommendations that there was no ambulance or school building and they responded that these issues would be taken into consideration. Georgian sector was opened in school in 2012 and we made a great effort in it. We wrote recommendations regarding this issue and we found out that they were taken into consideration.” “We signed a memorandum with Employment Agency of Adjara according to which the agency will assist eco-migrants to attend different programs and find jobs afterwards. We just signed this memorandum and we hope that it will have a result. They have lists of people who need jobs and database of refugees. Presumably, new base of eco-migrants will be created under this memorandum.” Evaluation of cooperation with G-Pac “Our relationship is different from relationship of donor and grantee. This is our second project and we did not have a feeling that this organization is a donor. It is more partnership than relationship of donor and grantee. In partnership I mean cooperation, sharing information. We did not have any problems regarding reports and we have regular relationship. If we have any news, we tell them: let’s work on this issue. Study visit was carried out in Slovakia and we were engaged in this process as well. They helped us to gain European experience and it can be said that G-Pac is more a partner than a donor.” Evaluation of civil society “Civil society does not imply only NGO sector. If we compare the situation 10 years ago and now, there is some progress though it is not close to ideal. Not only NGO sector but Ministry of Education needs to work on this issue. If recruitment level of professional staff working in different sector is increased and they become well aware of their rights, this will help the society to develop. Current situation is the best condition for 10 years, but many things are to be done. Civil life is more active in Tbilisi compared to regions.” “I think that civil society is under development. People did not have an opportunity to carry out any other activities rather than social activity. Nowadays, students have an opportunity to engage in civil life, do something Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 27/ 30 good for society. In old times, there was no possibility to conduct trainings on different topics. We were content with practical course. There are so many other ways nowadays that even we are surprised, take information center of volunteers. Students are able to increase their civil knowledge.” “During the last 10 years, appearing in NGO sector was perceived as one of the ways to appear in government. I think it has changed recently.” “To discuss the situation 10 years ago and now briefly. Activity of NGOs was expressed by taking part is some kind of rallies. But now, if we speak about the example of Batumi, all NGOs that are formed in the city carry out activities directed mainly to civil society development. Main goal of our organization stipulated in our protocol is participation in formation of civil society and and its assistance. Organizations were engaged in political movement during the last few years and accent was made on student activities. We implemented a project on volunteering last year – this project was the first one conducted in Batumi and one of the exceptional in Georgia. More than 100 students and pupils of senior year were recruited and trained. Since then, they are actively engaged in different activities and are not occupied only with political rallies. They were just in classroom before the trainings and their area of activity has widened – they started looking for new trainings and exchange programs.” “Formation of civil society is never completed. Its main disadvantage is that it will never be completed. It is a permanent process, demands are permanently increasing and society cannot catch up with it. The main thing is that there should be dynamics of development. I don’t believe in words- completed society. As or problems, they are too much. NGO sector is often equated with civil society and this is wrong. Government perceives everything as civil society and this is wrong too. Everything is civil society minus state. It is not like this but as for development tendencies, dynamics of development stopped in recent period. Growth of civil consciousness is crucial. State policy was the reason of stopping the development as the state made an accent of development of state and not society. These two might be the same for someone but not for me. Let’s develop state institutions and the society will develop on its own but the thing is that society is in shadow and not under development.” Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 28/ 30 “Speaking about students, youth sector is a main direction. Youth is the most active segment of society. I notice that there is a lack of young leaders In general. I don’t mean political leaders, I speak about public leaders. They should become leaders in their field of activity, for instance in science. Society, masses of people need leaders to whom to follow. Society does not create leaders anymore and respectively, it stopped development.” What needs to be done for development of civil society “Activities need to be carried out in different fields and only one decision cannot change the whole situation. First of all, the state must establish right educational system. Society should be given an opportunity of self-realization. International donors make an accent that NGOs need to develop their own management in order not to be depended on donors only. Other types of legislative warranties need to be created for this purpose. NGOs should not be depended only on foreign donors. Mentality of “grant-consuming” will be permanent. On the given stage, there is no other source rather than grants; social entrepreneurship is regulated in such a complex way that you may get stuck in any case. Besides, Georgian funds are not established; there are legislative obstacles, taxation and so on. Certain privileges should be given in terms of tax policy in order to develop NGO sector. System of civil education needs to be developed; school should be given a certain function. I don’t see school’s function nowadays. Giving an education is not a sole function of school – but bringing up and developing a citizen. Schools do not bring up a citizen nowadays but a person without anything - no education or orientation. School is a main cell and mental changes must take place at home. Family forms the priority and citizen. Notion of family has faded as well. Family is a main cell of any society. Government should try to create the cult of family which ensures the sustainability of country.” “The first thing that needs to be done by the state is a reform of education. Fees in universities are too high and the result obtained by a student is too irrelevant compared to the fee. Universities should attract professional staff. Speaking on the example of my university, I can say that there are no new lecturers in Batumi. Staff is old. Another thing is that roles should be separated. State should know its place and NGO its own position. All kinds of Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 29/ 30 state activities were planned from the state and the feeling was – “whoever is not with us, is our enemy”. Hopefully this will not go on. Our criticism and idea of problems were not adequately understood. In general, some amendments should be made in legislation even in terms of volunteering activity. Volunteering is one of the most active and important form of civil society development. There are rather serious taxation obstacles while employing a volunteer. As for donors, we, NGOs have to fit to them and very important problem may be left unnoticed. It is very difficult for foreigners to see the existing problems in Georgia in a way as we see them. NGOs should not be depended only on the projects funded by donors. ”Beyond the project, we don’t have any opportunity to implement other activities. It is a problem of our own management too. Besides, when the project is over you finish working on the topic when some kind of activities, like consultations could go on without grant. This is a problem of NGOs.” “We try to solve this problem and we have other activities out of project. For instance, we provide citizens with legal consulting on which we do not have any grant. We put more resources on every project but we work on enthusiasm. We want to do more and become a role model for NGOs. Society is closed, found its field of activity and does not have any relationship with people or structures with similar activities. This mentality of closing in its own shell should be changed. They should not perceive each other as competitors. “I will not cooperate with this person because I will slip in something” – this is an attitude towards volunteers. We brought volunteers to one structure, they did not get it right and abolished some vacancies not to leave the volunteers on these positions. NGOs within one region may have a contact too but meetings are not held. If something like this happens, it is only within the project and not by good will of people. Society should become free. We often observe Europe while we have the traditions we do not recall.” “Educational system beginning with school ended with university needs to be changed for the first place. I remember the situation in Batumi University where old lecturers made us write notes. As for new staff, they hire the people who study with these notes and get excellent points. They have not received a proper education and what can they teach others. The state must perceive NGO sector as an equal partner. There are many problems like eco-migrants which are unnoticed by the state. When we offer a recommendation pack on this issue on which we have been working for 3 years, you should take it and Evaluation of the projects implemented by G-PAC and G-Media in Georgia Report of qualitative research/ October, 2012 30/ 30 look at us as partners. 300 NGOs are registered in Batumi and only 15 of them function in reality. They see the propriety, get funding and then stop.” “NGOs are mainly used as a springboard for getting in government. Such facts took place in Adjara and I hope that it won’t happen again.” “Development of civil society is very important and any citizen should know that building the state does not depend only on the state and the citizens should take the responsibility on them even in terms of their profession.” “The state must create an environment.” Leaders of civil society “To tell you the truth, I cannot see a leader yet. If a leader is born from society, such society has a perspective of development. Masses of people need leaders who will pave the way and they follow. I don’t see such people yet. I hope they will appear.“ Evaluation of Project Implemented by G-PAC in Georgia Focus Group Research Report Draft: 1.0 Prepared by ACT Research for IBTCI September-October 2012 Tbilisi, Georgia Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 2 / 36 Table of Contents 1. Introduction _________________________________________________________________ 3 2. Survey Methodology __________________________________________________________ 4 2.1 Research Goals and Objectives _______________________________________________ 4 2.2 Survey Techniques and Instruments ___________________________________________ 5 2.2.1 Limitation and obstacles of the focus group process ___________________________________________ 5 2.2.2 Notes on the process ____________________________________________________________________ 6 3. Report №1 –MPA Students _____________________________________________________ 8 3.1 Motives for Applying to MPA Program ________________________________________ 8 3.2 Evaluation of PMA Program _________________________________________________ 9 3.2.1 Evaluation of the learning process _________________________________________________________ 9 3.2.2 Evaluation of G-PAC-funded internship program ___________________________________________ 12 3.2.3 Evaluation of the Perceived Segment of MPA Program _______________________________________ 14 3.3 Identifying Perceived Outcomes and Effectiveness of MPA program ________________ 14 3.4 Identifying Recommendations for the MPA program ____________________________ 15 4. Summary ___________________________________________________________________ 18 5. Report №2 - Mentors _________________________________________________________ 20 5.1 Evaluation of Work and Achievements of Civil Society Organizations ______________ 20 5.1.1 The process of appointing mentors ________________________________________________________ 20 5.1.2 Evaluation of changes in the civil society __________________________________________________ 21 5.1.3 Evaluation of the achievements in the civil society _________________________________________ 22 5.1.4 G-PAC’s role in the successful implementation of the mentor’s activities _________________________ 23 5.1.5 The role of Georgian government in the successful implementation of the mentoring effort _________ 24 5.2 Evaluation of Georgian Civil Society __________________________________________ 26 5.3 Identifying Recommendations for the Development of Georgian Civil Society ________ 27 Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 3 / 36 6. Summary ___________________________________________________________________ 29 7. Annex 1 - MPA Students ______________________________________________________ 30 8. Annex 2 - Mentors ___________________________________________________________ 34 1. Introduction The present document is a focus group research report on the evaluation of projects implemented by G-PAC in Georgia. The report is prepared by ACT Research for IBTCI. The precondition for research was the IBTCI decision, on the one hand, to determine the effectiveness of G-PAC involvement as well as its role in partnership and projects in order to enable their partners to improve their performance and relations with G-PAC. This improvement is intended to help them better evaluate their work as well as to give them directions and stimuli in the future. On the other hand, the research aims to evaluate the importance, effectiveness and sustainability of their support since 2010. Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 4 / 36 2. Survey Methodology 2.1 Research Goals and Objectives The main research goal is to evaluate and monitor the projects implemented by G-PAC in the region. The following were identified as qualitative research objectives: (1) Ilia State University students: ¾ Identify the motives for applying to Ilia State University on MPA program. ¾ Identify attitudes towards the MPA program. ¾ Evaluate teaching process: identify its advantages and disadvantages. ¾ Identify the perceived effectiveness of the MPA program. (2) G-PAC grantee organization representatives/mentors: ¾ Evaluate activities of civil society organizations and identify their outcomes. ¾ Identify G-PAC role versus civil society organizations. ¾ Evaluate the Georgian civil society and identify their preferences. Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 5 / 36 2.2 Survey Techniques and Instruments Focus group (FG), a qualitative research method, was used in the research. Two focus group discussions with (1) Ilia State University’s MPA students and (2) civil society mentors took place. The final research instruments in the English language were provided by the client. The instruments were later translated into Georgian by ACT Research and were used for focus group discussions. The client also provided the interviewee database, based on which the FG participants were recruited. The focus group discussions lasted for 1.00-1.30 hour. Focus group discussion took place in Tbilisi from September 29 till 30, 2012 at the ACT Research head office. The table below provides the distribution of FG participants based on gender and location. *Refer to annex for detailed group composition. Program activity (e.g. Training topic/Grant topic/etc) No. of participants/b eneficiaries Gender Location Female Male MPA Students 6 2 4 Tbilisi Grantees / Mentors 4 2 2 Tbilisi 2.2.1 Limitation and obstacles of the focus group process The focus groups coincided with a stressed political climate in Georgia (pre￾election period). Therefore the recruitment process was rather problematic. To be more precise, multiple efforts to recruit the required number of students failed. The existing situation in the country/busy schedule was Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 6 / 36 named as an excuse. It has to be mentioned that because of the tight deadlines, it was impossible to move the FG dates. The limited number of FG participants had an impact on the process of FG discussion. This fact hindered the generation of ideas among participants to a certain extent. The recruitment process in Zugdidiwas related to a problem. Interviewees were declining the offer to participate in FGs. The FGs were planned to take place in Zugdidi. Some of the participants were to be transported from Svaneti. Due to the fact that only three organizations were G-PAC grantees in the Samegrelo-ZemoSvaneti region, the total likely number of participants was six. From the existing potential participants only one agreed to participate in the focus group discussion. The busy schedule during the pre-election period was named as an excuse for declining the offer to participate. By agreement with the client the Zugdidi focus group discussion was replaced by that of Batumi, which was scheduled after the Election Day. 2.2.2 Notes on the process The recruitment of focus group participants started on September 25. The participants were selected from the database provided by the client. Initially, three focus groups were to be held – (1) in Tbilisi with Ilia State University students, (2) in Tbilisi with mentors from the civil society organizations and (3) in Zugdidi and Svaneti with civil society representatives. The latter was replaced by Batumi and was postponed until the end of elections. Focus group with students in Tbilisi For the organization of focus groups in Tbilisi the client put us in touch with Ilia State University MPA program coordinator who ensured the participation of students in the FG discussion. Due to the fact that the semester was not yet started, reaching students was rather problematic. MPA program coordinator managed to bring six students to the Ilia State University, however to ensure the neutral environment for students, they Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 7 / 36 were transported by ACT Research to its head office. In total, six students took part in the focus groups discussion. Focus group with mentors in Tbilisi For the organization of focus group discussion in Tbilisi the client provided us with the list of 13 potential participants. Irrespective of multiple efforts, none of the students was talked into participating in the focus group discussion. Afterwards, the client put us in contact with the G-PAC representative, who ensured bringing participants in the G-PAC office. Since the neutral environment is key to holding adequate focus group discussion, ACT Research ensured the transportation of FG participants to the ACT head office. In total, four students were participating in the FG discussion. Prior to the start of the FG discussion the participants noted that they were in a hurry and that the FG discussion should have finished in an hour. Therefore the discussion was hastened that had negative influence on the diversity of opinions. Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 8 / 36 3. Report №1 –MPA Students 3.1 Motives for Applying to MPA Program The FG participants, who are currently enrolled in the MPA program, have gained their undergraduate degrees in: law, business, economics. The research has revealed that prior to applying to the MPA program interviewees were employed. Moreover, their majority had an experience of working in the public sector. The research identified the following motives for applying to the MPA program: ¾ Diversified curriculum–The fact that the MPA program was incorporating various fields (economy, law, management, education administration) was a key motivator for respondents to apply for the MPA program as the diversity of classes allowed them to get multi-profile education. Furthermore, the program was tailored to the students with various educational background and experience. ¾ Unique character of the program –According to the interviewees, the MPA program is different from the ones offered by existing universities in Georgia. ¾ Innovation–The research has found that innovation is associated with a high standard. Prior to applying the students had a feeling that the newly created program was offering them a high quality education (for example modern literature, qualified professors). ¾ USAID-funded project–The research has found out that the interviewees had information that the program was financed by USAID. This, on the one hand, was associated with an American quality and, on the other hand, was perceived as different from the state university programs and more like the one at private universities. „I studied economics and business at Javakhishvili University. I wanted to apply for something different. At the beginning I wanted to apply for the education program as I was working in education. Then I found out that Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 9 / 36 Ilia University had a new program, which seemed interesting, as it was more comprehensive. I also learned that this program included some classes on educational administration. Thus, I decided to choose a broader field.” [Woman, MPA student] „I’m a lawyer and I wanted to apply to a different program rather than law. Javakhishvili University had public administration program and I was going to apply but then I found out that this program was a similar one. Since it was a pilot program I decided to apply.” [Woman, MPA student] „I studied business administration and my decision to apply to the MPA program was conditioned by the fact that the business administration program was oriented more towards the operation of the company, accounting, finances etc., while this very program was more about economic policy. That is why I took this decision.” [Man, MPA student] 3.2 Evaluation of PMA Program The present section provides interviewee attitudes and evaluations about the learning process in their graduate program. It has to be mentioned that focus group participants are sophomores of the MPA program who have completed the first semester of their graduate program. Therefore, their evaluations are based on the experience gained in one semester. 3.2.1 Evaluation of the learning process The learning process in the graduate program is positively evaluated by interviewees and is characterized as interesting, comprehensive, cognitive, fun as well as intensive. The research identified the following advantages of the MPA program: ¾ Highly qualified professors – The research has revealed an overwhelmingly positive attitude towards professors. They claimed MPA professors to be real professionals who, besides being good at transferring knowledge, are very outgoing. Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 10 / 36 ¾ Professors’ Teaching Methods – The research has discovered that at the beginning, students found study materials difficult. This was mainly caused by heavy curriculum, a large amount of study materials and students busy schedule. On top of that, students had undergraduate degrees in other fields that made the process of absorbing study materials from different fields rather challenging. A method developed by professors themselves, which is about practicing individual approach towards students, has played a significant role in solving this issue as well as in successfully passing exams. A small number of students in classes have also helped achieve it. ¾ Team Work – The focus group research has revealed that because of class requirement, students have to do team work quite often. This makes the learning process exciting while at the same time enables the students with different educational backgrounds share knowledge and experience that eventually has a positive impact on the overall outcome. ¾ Modern Literature – Students claim that modern and well-selected literature makes the learning process more rewarding and interesting. It has to be noted that students have full access to the mentioned literature in the library as well as electronically. The research has discovered that study materials are in English about which students were informed initially (prior to applying to the mentioned program). Thus, this circumstance does not create a language barrier for the majority of students. ¾ Proper balance between theoretical and practical aspects– The interviewees claim that there is a proper balance between theoretical and practical classes. Some classes in the curriculum are theoretical, while others practical. Overall, these classes balance each other. ¾ Good management – The research has revealed a particularly positive attitude towards MPA coordinator, who is always oriented towards the creation of a more comfortable environment for students that includes a well-planned program, fully informed students on the curriculum as well as on various activities. Such an Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 11 / 36 attitude from the part of the management significantly increases the students’ motivation. ¾ Visiting professors from U.S.– The research has found that foreign visiting professors in the graduate program are very well perceived by students. They interviewees said that American professors familiarize them well with their public administration system and give them real time examples. ¾ Teaching in English – Considering the fact that the demand for proficiency in English language is high on the labor market, the ability to get high education in English language is perceived as a big priority of the program among interviewees. ¾ Monitoring research – The research has revealed that quality evaluation research is carried out rather often during the studying process where students are given the opportunity to express their opinion and provide recommendations. According to interviewees, their opinions expressed in the research are fully taken into consideration. „For me everything started really well. I just had five classes that were really hard. I had a lot of study materials that eventually popped up prior to exams. Because of this we had experienced problems, firstly with our own selves.” [Woman, MPA student] „Overall, I’m very satisfied, as they were not typical classes like the ones each of us had had at the universities. To start with, it was different because each professor did not start the class with a new material and didn’t present himself/herself by highlighting achievements in his/her own field...The stress was made on our personality, our experience, and our knowledge in the field. We also had team works. Teams were normally composed of students with different backgrounds and different past who could generate interesting ideas. We did projects, which were extremely interactive, and we were enjoying what we were doing. Students were very motivated, which is very important for me.” [Woman, MPA student] Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 12 / 36 „I would positively evaluate the quality of this program and would have given it nine out of 10. This is probably because of professors as I had been studying statistics for 4 years, however, what I have learned here in a semester I hadn’t learned in four years. This was probably caused by having few students in the class that allowed the professor to answer each question.” [Woman, MPA student] „We had our own auditorium, a newly renovated one with new desks and chairs.“ [Woman, MPA student] „I think it is well-organized. We have information about everything, even about the missed days. I think we didn’t have a problem even with books. Definitely, Achiko (program coordinator) deserves all the credits for that.” [Woman, MPA student] „I remember exactly, but we had an opportunity to give interim evaluation about the learning process generally, including professors, classes and everything two or three times. After each evaluation we could feel the change. If we couldn’t say openly our opinion or remarks because of the awkwardness, we had an opportunity to express them in the special evaluation form. Consequently, everything was taken into consideration.” [Man, MPA student] 3.2.2 Evaluation of G-PAC-funded internship program Only two focus group interviewees had had internship, thus the evaluation of the internship program is based only on the feedback of two interviewees. The research has identified that students received an offer from the MPA program management to do internship on G-PAC funded innovations research. Eventually, six students from the class got internship. According to the interviewees the internship was in summer during evening hours; thus students could combine internship and work. Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 13 / 36 The internship program is evaluated by the interviewees as interesting, cognitive and fun. The internship program, according to the interviewees, helped them put the acquired knowledge at the university into practice that has subsequently contributed to the improvement of their skills. Students perceive that the internship program will enrich their CVs. The focus group discussion has revealed that G-PAC had also funded an internship program for two students in Canada. This program was in big demand among students. According to the interviewees those with more experience were given priority in the selection process for this very internship program. „We had a G-PAC funded internship program on innovations research at the Ministry of Justice. The university was giving us credit and it was paid. Most of the students were busy in summer. Several students were selected based on interview.” [Woman, MPA student] „Right, in this respect it was very informative and gave us a lot as we had to come up with the research idea“ [Man, MPA student] „It was extremely useful, as it was about the analysis of everything we had learned and I had an opportunity to put everything into practice.” [Woman, MPA student] „It was rather useful as we learned about the public sector. We knew how the public sector worked but we learned much more during our when direct contact. I think it was quite useful.” [Woman, MPA student] „There was a different internship program prior to it, which I think is also funded by G-PAC. It’s a Ukrainian project and sends Georgians and Ukrainians to Canada.” [Woman, MPA student] Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 14 / 36 3.2.3 Evaluation of the Perceived Segment of MPA Program The MPA students are described as goal-oriented, with diversified education and interests, active, creative and risk taking. The research showed that after getting undergraduate degree MPA students could get professional experience and only afterwards apply for the graduate program. Therefore, one can say that they have a certain idea about what they want to do in the future and are confident in their goals. “Those who are active, interested and have their opinions about issues study here. One has to have diversified skills as you have to deal with everything and you have to have your own opinions on things.” [Woman, MPA student] “We are more determined and know what we want.“ [Woman, MPA student] 3.3 Identifying Perceived Outcomes and Effectiveness of MPA program The research has revealed that at the moment MPA students have already obtained important and required skills that they are already using/will use in the public sector job. ¾ Determination – It has to be noted that the MPA program has enabled the students to see their future goals, what they want to do in the future and in which direction. ¾ Independent work –students who are already employed in the public sector have become more self-confident and independent at their work and try to share their knowledge with others. ¾ Development oriented – The knowledge obtained at the MPA program coupled with work experience, have enabled the students to see the areas that need improvement in the public sector. According to the research participants, after graduating from the MPA Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 15 / 36 program, they will become much more competitive, that will help them in carrier advancement and professional development. „Speaking as a student working at a public sector, the knowledge I receive at the moment is extremely important for the realization of my goal that I want to achieve in the future, say becoming more active in politics. Everything that I study has a great impact and I hope that I will get more and become more professional in this respect in the future in order to work efficiently.” [Woman, MPA student] „I also think that it is good, as the more knowledge you have the freer you become and when they tell you at the meeting to do something you will be well positioned to say that this is not right and I know what is the right way to do and I will do it the way I think is right. It gives you confidence as you know more and you have a say.” [Woman, MPA student] “I think it will help me realize my future plans. I have an insight into what is going on in the public sector; well having an insight might not be the right word, but I’m getting there step by step, I’m learning everything. I want to relate my future to the public sector and I want to be knowledgeable in this area.” [Woman, MPA student] 3.4 Identifying Recommendations for the MPA program The research revealed that MPA program is well positioned among the existing MPA programs in the Georgian universities. However, the interviewees suggested the following recommendations for its further improvement. ¾ American professors – As it has been mentioned above, students are enjoying the seminars of the American professors as they enable them to familiarize themselves with public sector models in the U.S. However, they expressed concern with regard to the fact that these seminars are not core classes. To ensure that seminars by foreign Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 16 / 36 professors are more productive, interviewees suggested having the visiting professor lecture the core class and grade students. ¾ Internship in the public sector – The interviewees consider that it would have been significant if all students could have internship in the public sector, especially those who were devoid of this opportunity; I.e. in case of limited number of internships make sure the students are allocated. Also, those students who are employed in the public sector expressed an idea about giving them an opportunity to hire classmates (those who don’t have the experience of working in the public sector) as interns in their organizations. ¾ Create alumni network – The research revealed some kind of attachment between MPA program and students due to perceived positive environment in the program. The interviewees wish that this attachment could never stop after graduating. Thus they suggest creating an alumni network that would enable them to help each other in the professional development as well as to consult future MPA students and share their experience as mentors. At the focus group discussion it has been mentioned that the program management is planning to create such a network. „I would have advised this program to make American’s lectures more like classes. There should be an American professor; it was really a great experience and such a practice should continue. Georgian professors are also good but this program is based on the foreign example and they have a rather big experience in public administration than do Georgian professors. Every Georgian professor we have had was a professional but an American professor was much more interesting for us. In each semester there should be at least one American professor.” [Woman, MPA student] „I wish that the end of classes was not the end of everything. I know they are working on it and plans are in place to create an alumni network for reciprocal assistance. This is very important as we can provide recommendations and assistance to students who are applying; we can serve as mentors and share with them our Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 17 / 36 knowledge and experience. [Woman, MPA student] „It would have been good that whether it is memorandum or internship, those students who can’t afford, or at that moment are not employed in the public sector, to be given an opportunity to get more practice.” [Man, MPA student] „An idea just struck me that if for instance there are 20 students in class and there is some kind of program for 5 or 6 students, there should be an additional program for others to ensure that they also get experience and that everyone benefits.” [Woman, MPA student] Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 18 / 36 4. Summary Based on the focus group findings, students with various educational backgrounds study on the MPA program. The diversity of program curriculum/multi profiling and distinctiveness from other programs are named as main motivators for applying to this program. The fact that the program is funded by USAID is perceived as the main motivator as it is associated with high standards. MPA program itself is described as interesting and comprehensive. Highly qualified professors, well tailored teaching methods, renewed studying materials and good management are considered to be its main advantages. This, on the one hand, impacts the students’ motivation and, on the other hand, effects the quality of education. The G-PAC funded internship is positively evaluated by students. They consider such internship crucial for their professional development. The research has revealed that the internship had allowed students to put their theoretical knowledge into practice. The research has also revealed that motivated, target oriented students study on the MPA program. This is because students were given an opportunity to get professional experience, think about their future plans and goals after getting undergraduate degree and only afterwards make a choice among various graduate programs. The focus group discussion also discovered that after attending a semester students became more active at work and independent in their work. They maintained that the MPA program would help them in the professional advancement and development. The research has identifies students’ wishes for the further refinement of the program. It is their wish that the U.S. professors’ seminars were more like classes that would have made it more productive. The research also revealed that students wish for equal involvement in various activities and in the internship program in order to give equal opportunity to everyone, to obtain experience of working in the public sector. Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 19 / 36 Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 20 / 36 5. Report №2 - Mentors 5.1 Evaluation of Work and Achievements of Civil Society Organizations 5.1.1 The process of appointing mentors According to the FG participants, they were mentoring organizations in various regions of Georgia. They said G-PAC reviewed the database and selected mentors based on it. The research has showed that with the help of G-PAC-organized trainings, an expert evaluation was made that included the evaluation of the mentors’ experience in various fields. Mentors were assigned to organizations based on their needs. According to the FG participants, the assistance was needed not only with those issues identified in the project but with others as well. Therefore, they had to increase the scope of their responsibilities and devote more time to the mentoring effort. “Mentors are deployed based on fields, thus based on the mentor’s experience.” [Woman, mentor] “In my case, I had a lot of responsibilities, because of my experience. Starting from organizational development, PR and finished with networking. There were a lot of components.” [Woman, mentor] “It was the same in my case because it was about the evaluation of the budgetary expanses in Gardabani district and it was not only about research methods and a report on quality improvement. We were also assisting them in the conduct of an advocacy campaign, writing press releases, proofreading tests and in networking with the local government. We were providing assistance on every level.” [Woman, mentor] “Our assistance did not only cover one concrete thing. Obviously Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 21 / 36 we care about things and thus we were trying hard to provide assistance in that process.” [Man, mentor] 5.1.2 Evaluation of changes in the civil society FG participants consider that the Georgian civil society is weak, which is caused by the lack of strong civil society organizations in the country. The research has discovered that mentoring effort has contributed to the development of the civil society in the regions. ¾ Improvement of life of impoverished people - Due to mentors’ hard work those villages that have never had running water were supplied with it with the help of the government. ¾ Empowering civil society organizations for the development of civil society – Organizations that were mentored by the FG participants have realized it well that people in the regions are full citizens and thus have all rights to demand things from the government. ¾ Establishing peaceful relations with the local self-governance – civil society organizations in the regions have realized the importance of having peaceful relations with the local self-government. They have learned how to establish such a relation with them and, subsequently, use it for the good of themselves/ the society. ¾ How to write a good report – According to the FG participants, the organizations they had been mentoring have learned how to write a project and draft reports in order to ensure that their work is rewarded. The research has revealed that the relationship with mentors and the organizations in the regions did not finish with the end of mentorship. On the contrary, their relations continued in other projects as well with partnership and mutual support. “There was a provision in the contract that obliged us to devote 20% of our work to mentoring. However, there were a lot of cases when we received calls every day and every minute… Those people and that organization later became our partners. We were working Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 22 / 36 jointly on the US Embassy project. Then we employed them and have established such a good relation that now when we need something in the Gardabani district, those people and the organization solve them for us. So, it is not only about us teaching them how to do things but about mutual assistance.” [Woman, mentor] “I would also say that it was mutually beneficial in terms of learning things. When we were working on the IDP project I met interesting people who are playing an important role in this area.” [Man, mentor] 5.1.3 Evaluation of the achievements in the civil society G-PAC’s mentorship initiative was regarded as important by the research participants. They consider that for the development of civil society in the country it is important to continue the mentorship initiative, to stimulate regional organizations and implement civil and democratic values in the society. The FG discussion has discovered that motivation of the organization representatives helped mentors achieve results and increase their level of motivation. The FG participants claimed that at the beginning of their mentoring effort, the organizations had a feeling that mentors were invited in order to control them. Therefore, they were trying to be cautious in their relations with them, which was hindering the work to a certain extent. However, according to the interviewees, the organizations understood fast the idea and the importance of mentorship that led to the establishment of a cooperative relationship. “I’m not the supervisor of that organization. I don’t pay them money and they don’t pay them to me either. This means that we are cooperating and I’m motivated. Motivating factors for me were the actual results.” [Woman, mentor] Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 23 / 36 “The motivation of these people was key. I would like to underline the Adjara project. That organization was very motivated. When you see that they are motivated you can talk to them on the phone at 12 a.m., exchange emails. When I was seeing that they had such a drive, it was filling me with enthusiasm.” [Man, mentor] “It motivates me when I see actual results... We shouldn’t be talking about it In this century, but the fact that these people now have water and can live with a better life in winter, motivates me.” [Woman, mentor] “We met several times during this period and shared our impressions, challenges with one another. I remember that once when we met they thought I was going to do the check-up. They knew a mentor was assigned but they didn’t fully understand what was it about. That is why they were cautious at the beginning and were trying to portray everything in a positive manner. But soon they guessed that it was not a check-up and developed trust vey fast. The existing cooperation is a result of it.” [Man, mentor] 5.1.4 G-PAC’s role in the successful implementation of the mentor’s activities According to the interviewees, the mentorship program proved to be very successful. Thus, it can be said that G-PAC played an important role in the development of civil society in Georgia. According to the interviewees, G￾PAC was always trying to facilitate the effective work of mentors. ¾ Mentor meetings – According to the research participants, in order to promote the exchange of information among mentors G-PAC organizes coordination meetings among mentors. The information gained at the meetings was helpful as it helped in doing work in a confident and right manner. When necessary, changes were made to the projects during these meetings. It has to be noted that interviewees wished such meetings were held more often. Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 24 / 36 ¾ Putting mentors in contact with organizations – According to the research participants, G-PAC was matching right the mentor and the organization. This had a positive impact on the overall outcome. ¾ Technical support – According to the research participants, there were cases when organizations didn’t have the required technical resources. In this respect G-PAC was always ready to provide assistance to organizations and the mentors to insure that their work was not obstructed. “G-PAC had multiple roles. One of the most important one was a meeting with mentors. These were like coordination meetings where we were sharing experience to one another. I think that such meetings should become more frequent as it was very interesting for me to learn what was going on in other projects. When you work you think that things are different elsewhere, deadlines might not be met…” [Man, mentor] “G-PAC’s is doing its job well. It matches well the mentor and the organization… they were making all efforts to contribute to the process.” [Woman, mentor] “An organization might not have resources. G-PAC was always ready to provide technical assistance and technical expertise.” [Man, mentor] “It was a very good initiative. It was unique.” [Woman, mentor] 5.1.5 The role of Georgian government in the successful implementation of the mentoring effort The research has identified various opinions about the role of the government in the mentoring effort. Part of FG participants’ considers that the government is playing a small role in their work. According to interviewees, they are trying to establish friendly relations with local self- Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 25 / 36 government bodies in order to simplify relations with them. However, some participants evaluated the role of the government in their activities as important. According to them the projects developed through mentoring were acclaimed and supported by the Adjara government. It has to be mentioned that without their support the project would not have been implemented. „As regards the role of the government, it was crucial for both projects. It proved to be more important with regard to the Adjara project, as some decisions can be made only by the government.” [Man, mentor] Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 26 / 36 5.2 Evaluation of Georgian Civil Society According to the FC participants, civil society means vibrant citizens who unite around a specific goal to make a positive change. The interviewees referred to the Georgian civil society as weak and loose. The “Tbilisi’s Hamkari” was named as an example of the Georgian civil society. This organization was strengthened recently by means of the assistance from donor organizations, which to a certain extent carries the risk of transforming into a typical NGO in the future. Based on the focus group discussion it can be maintained that the country’s economic stability, government support and free media are key to the development of the Georgian civil society. According to the interviewees it is important to create small organizations/unions who will be well aware of the community issues. G￾PAC is viewed as an important supporter for the implementation of similar projects. The research participants consider that the civil society has somewhat changed for the better in the past 10 years and donor organizations should be credited for that. However, opinions were voiced that donor organizations fund projects based on their own interest and pay little attention to the community issues. Issues like gender rights, ecology etc. are considered to be interesting for donors. The FG participants are talking about non-politicized student rallies that took place after the release of prison videos. According to the interviewees, such activeness from the part of the students can be considered as a step forward towards the development of the Georgian civil society. “The Georgian civil society has changed and whether we like it or nor the donor organizations deserve credits for it. There is a lot of criticism and it is relevant to a certain extent as donors are financing the issues that are interesting for them.” [Woman, mentor] Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 27 / 36 “To my mind one of the shortcomings of the international assistance was that the NGO sector was created provisionally. Now the situation is like „in opposition from the government...“ I also consider it as an achievement that students can take to streets and stand aside from the political processes.” [Woman, mentor] 5.3 Identifying Recommendations for the Development of Georgian Civil Society The research has discovered that in order for the government to support the development of the civil society, it is important to have a decentralized local government, as it will help create a vibrant society both in Tbilisi and in the regions. Based on the FG discussion, it can be maintained that civil society, as a subject is not taught right at Georgian schools, as it is not tailored to the Georgian mentality and values. They recommend carrying out a proper education reform that would ensure that civil society subject is taught right at schools. This will help develop citizenship mentality in the early age. According to research participants, in order to ensure the increased role of donors in the development of the Georgian civil society and achieve a better outcome, it is important for the donors to become more flexible, to study the actual problem on site and start financing projects based on such analysis. On his note, the interviewees have recalled and positively evaluated the following G-PAC activities: ¾ G-PAC funded project – The FG participants have recalled the G￾PAC-funded project where mechanisms and rather than concrete objectives were identified. They considered that in such situation the project implementing organization was given more freedom and flexibility in its activities. ¾ G-PAC research – The FG participants are positively evaluating the research conducted by G-PAC about the interests of donors, NGOs and the public. The research has identified that NGOs and people have different interests. According to the interviewees, G-PAC Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 28 / 36 used this research to identify priorities for awarding grants and called on the organizations to act in the interest of people. “I think that the local government decentralization is key to the formation of the civil society since some activities should start in small villages and its communities should start to believe in something, they should believe that they can change something, they can get involved in something, express an opinion on something and be an active citizen.” [Man, mentor] “G-PAC did a public opinion survey about what interested donors and NGOs and what interested people. Based on the research findings they started awarding grants and identifying priorities. What interests donor and NGO does not interest people and thus let’s do things that bother people. People were interested in education, unemployment, employment, healthcare and issues like this.” [Woman, mentor] “It is very important when the donor is flexible, i.e. when it doesn’t set its agenda but rather studies it and carries out relevant activities.” [Man, mentor] Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 29 / 36 6. Summary Based on the FG discussion it can be maintained that there is a lack of strong civil society organizations in Georgia that hinders the development of the Georgian civil society. According to the FG participants, the G-PAC’s mentorship effort is one of the steps forward towards the development of the Georgian civil society. According to the interviewees, their mentorship effort helped strengthen and activate regional civil society organizations to an extent to enable them identify issues in the communities independently, work on them, establish friendly relations with the local authorities and use them for the benefit of the society. The FG participants evaluate the Georgian society as weak and loose and contribute it to the politicized environment in the country. They maintain that the Georgian society has changed for the better with the help of grants awarded by the donor organizations. The focus group discussion could identify the recommendations with regard to the development of the Georgian civil society. The major development with this regard would be the decentralization of the country’s local governments. As regards donor organizations, the interviewees advise them to be more flexible and focused on the real community issues. Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 30 / 36 7. Annex 1 - MPA Students Notes of the Focus group (MPA Students) Motivations for Applying to MPA Program “I studied economics and when I saw the offered classes I liked them. I got interested in the program, as it was similar to my undergraduate classes. It was interesting for me and decided to apply. The fact that the program was cooperating with the Columbia University and was funded by USAID was interesting for me, as I considered it to be more different from Georgian universities and my expectations really proved right.” [Woman, MPA student] “I graduated from Javakhishvili University where I studied social work and my desire was pursue carrier in social policy. Sadly, no university was offering graduate program in this field. Then I discovered this program at Ilia University, which was offering classes that were interesting for me as well as those that I had already studied. When I familiarized myself with the program I got interested. It was also offering more funding, was not old and obsolete. All these factors contributed to my decision to apply.” [Man, MPA student] “I studied law; then I worked in auditing. Since my work had little to do with lawyer’s profession I was thinking about becoming an economist, but it was kind of late. Then I discovered this program. When I looked through the classes it was a mixture of law and economics and since I had relations with the tax system I thought public policy would be useful in terms of the analysis of the economic policy. This is how I decided to apply Ilia State University.” [Woman, MPA student] Evaluation of the MPA Program “The only thing is that there are economist and lawyers in class and economic subjects are somewhat difficult for lawyers; understanding these graphs... It’s hard to absorb everything in class and the volume of information is huge.” [Woman, MPA student] “I would say the same. If we see things from another angle, this program would not have been so busy and these 5 classes would have been absolutely adequate for others, but due to the fact that each of us is Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 31 / 36 employed, a little time is left for studying. The good thing was that professors were explaining everything in plain language. This was not about laziness; it was more about lack of time.” [Man, MPA student] “The approach and attitude in general is very important in the process. I did my undergrad in a different university and if I compare them, the difference is huge. The professors were focused on the issue and were trying hard to take into consideration our backgrounds, our capacity. They were making sure our question didn’t remain unanswered and the provided information was in plain language.” [Man, MPA student] “At the beginning of the year we knew what classes we would be having. We also knew the books; and when we were having the introductory meeting we were fully informed not only about the classes and programs in the first semester but also about all classes to be covered in two years.” [Woman, MPA student] “Achiko (program coordinator) is something. He is always calling and sending emails to inform us about the activities and ensure our participation. They are caring. When you see how much they care and how important your participation is, you feel the responsibility. I felt how important each of us was for that program.” [Woman, MPA student] “We had a policy class; we also had practical assignments, situation analysis. But more practice was required in the economics class to my mind. But we were coming to terms with the fact that some students in the class knew economics well.” [Woman, MPA student] “The professor was yet underscoring the fact that some knew it well and that it was less boring for us. What took us al lot of time to understand was extremely easy for others. Thus, there was no balance; however, when we saw the grades we guessed that it was well balanced. [Woman, MPA student] “American professors were visiting sometimes but they were lecturing only once or twice a week. Those were just seminars. If those seminars were intensive one-week long classes and was followed by an exam, it might have been more productive.” [Man, MPA student] Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 32 / 36 Identifying Perceived Outcomes and Effectiveness of the MPA Program “Firstly, what we have gained is the skill of communicating with people – how should a public administrator communicate with people. I’m working in the public sector and am always trying to have unpretentious with every taxpayer. This is very important. Also, how to plan your work, your future, how to move foreword… You start to think about changes, what can be better for people or how to do so that people benefit from my work.” [Woman, MPA student] “I notice the result even in terms of terminology. When you have a concrete assignment you use your knowledge for instance in statistics, etc. Thus you are doing something that your coworkers can’t do.” [Woman, MPA student] “There are organizations and offices who are doing the same all the time, are not doing anything new and are satisfied with it. However, for me as a student, it is much easier to create something new, to use it for something new, to be more independent in the decision-making process, and not to be dependent on the supervisor when plan something. I have more opportunity to use knowledge in practice.” [Man, MPA student] Identifying Recommendations about MPA Program “Research that does not require much time would be interesting. Something like going home after work, writing something and then sending it to someone.” [Woman, MPA student] “Everyone should be involved, not only top five or top ten. He/she should feel that he/she is important. We are not that many to make it impossible.” [Woman, MPA student] “Those not working in the public sector should learn what is going on in the public sector.” [Man, MPA student] “An idea just struck me. Those who work in the public sector should invite those, who don’t work, to our work places to show what out job is like. This must be well organized and should not be done at my own initiative.” Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 33 / 36 [Woman, MPA student] “I had an intern who I was familiarizing himself/herself with social work at my work. I was working at the probation department and he/she was getting credit for internship.” [Man, MPA student] Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 34 / 36 8. Annex 2 - Mentors Table 1. List of people invited and participating in the survey (Mentors) Mentor’s name Location Participated in Survey (1.yes/2.non) Beneficiary Organization/s Beneficiary Organizations’ Location David Jijelava Tbilisi 1 Institute of Nationalism and Conflict Studies / Tbilisi Step Forward Batumi Salome Tsereteli￾Stephens Tbilisi 2 Svaneti Youth Center named after Guram Tikanadze Mestia, village Latali Nino Gujaraidze Tbilisi 2 Association of ‘Young Chiaturelians’ Chiatura Union of Scientists of Imereti Region ‘Speqtri’ Kutaisi Giorgi Meskhidze Tbilisi 1 Association Zekari Ambrolauri Nino Evgenidze Tbilisi 1 New Initiative in Kvemo Kartli Gardabani Young Teacher's and Psychologist's Association Tbilisi Tamar Makharadze Tbilisi 2 Club of Disabled Individuals of Gori Gori Center for Integration of Disabled People of Kakheti Region Telavi Tamar Koberidze Tbilisi 1 Association of Civil Society of Georgia Tetritskaro Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 35 / 36 Dimitri Gugushvili Tbilisi 2 Civic Activities Center Gurjaani Grigol Gvilava Tbilisi 2 Center for Public Monitoring and Research Kharagauli Human Rights and Social Justice Research Center Poti Tamar Tandashvili Tbilisi 2 Psychometric and Consulting Association Tbilisi Ketevan Gujaraidze Tbilisi 2 Stepantsminda Stepantsminda (Kazbegi)    Evaluation of Project Implemented by G-PAC in Georgia Report on Qualitative Research / September-October, 2012 36 / 36 Notes of the Focus group (Mentors) Evaluation of Work and Achievements of the Civil Society Organizations ”G-PAC has a database. It collected the information about those experts who were ready to mentor an organization. Then they were matching mentors to organizations based on their expertise. If there is a consent from both sides the cooperation starts.” [Woman, mentor] “Generally, I think that the civil society in Georgia is weak because of the lack of qualified civil society organizations. And those who are qualified should work on the knowledge transmission... The mentorship was our deed of gift in order to help make the civil society in the regions more vibrant and give them the feeling that they are citizens. Overall, with our help they achieved a big success. Havingrunning water for 15 minutes in the 21st century and now having itfor 2 hours a day can be considered as a success... We helped make their lives easier even for a bit.” [Woman, mentor] ”I agree with everything but would also like to add that for me personally the most import was not related related with the improvement of the living condition but rather with a process of teaching them how to establish contacts, how to search information, how to establish work relationship versus hostile relationship with the local government in order to achieve a result; and it doesn’t matter what the leadership has been doing. When someone is refusing to meet you ... and the fact that I was teaching them what to do in order to get positive answer on the request for a meeting is very important for them as tomorrow they will be able to do it independently.” [Woman, mentor] “They are taught based on their needs.” [Woman, mentor]