MID-TERM PERFORMANCE EVALUATION OF THE GOOD GOVERNANCE IN GEORGIA (G3) PROGRAM [June 2014] This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Mendez, England & Associates and NORC. MID-TERM PERFORMANCE EVALUATION OF USAID/GEORGIA GOOD GOVERNANCE IN GEORGIA (G3) PROJECT Final Report June 2014 Prepared under RFTOP No. SOL-114-I-13-00001 Submitted to: USAID/Caucasus Submitted by: Dr. Olga Nazario (Team Leader) Dr. Peter Epstein (Local Government Expert) Ms. Mawadda Damon (Evaluation Expert) Ms. Nina Khatiskatsi (Georgian Local Government Expert) Mr. Mikheil Kukava (Georgian Governance Expert) Contractor: Mendez England & Associates 4300 Montgomery Avenue, Suite 103 Bethesda, MD 20814 Tel: 301- 652 -4334 www.mendezengland.com Cover Photo: Courtesy of MSI DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS ACRONYMS .................................................................................................................................... EXECUTIVE SUMMARY. .............................................................................................................. I Evaluation Purpose and Evaluation Questions ................................................................................................. i Project Background ................................................................................................................................................. i Evaluation Methods ................................................................................................................................................. i Evaluation Limitations ............................................................................................................................................ ii Findings ..................................................................................................................................................................... iii Recommendations ................................................................................................................................................ vii Interventions/Approaches Recommended for Future Programming ..................................................... ixi 1.0 EVALUATION PURPOSE AND EVALUATIOON QUESTIONS .................................. 1 1.1 Evaluation Purpose .................................................................................................................................... 1 1.2 Evaluation Questions ................................................................................................................................ 1 2.0 PROJECT BACKGROUND ................................................................................................. 2 3.0 EVALUATION METHODS AND LIMITATIONS ............................................................ 4 3.1 Evaluation Methodology .......................................................................................................................... 4 3.1.1 Qualitative Data Collection ..................................................................................................... 4 3.1.2 Quantitative Data Collection .................................................................................................. 5 3.2 Evaluation Limitations ............................................................................................................................... 6 4.0 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS .......................................... 7 4.1 Question 1: Impact on Government Transparency and Accountability ....................................... 7 4.1.1 Findings ......................................................................................................................................... 7 4.1.2 Conclusions .............................................................................................................................. 16 4.1.3 Recommendations .................................................................................................................. 17 4.2 Question 2: Effectiveness of Government Mechanisms for Public Engagement .................... 18 4.2.1 Findings ...................................................................................................................................... 18 4.2.2 Conclusions .............................................................................................................................. 25 4.2.3 Recommendations .................................................................................................................. 26 4.3 Question 3: Sustainability and Impact of Capacity Development for Government Officials ................................................................................................................................................... 27 4.3.1 Findings ...................................................................................................................................... 27 4.3.2 Conclusions .............................................................................................................................. 36 4.3.3 Recommendations .................................................................................................................. 37 5.0 INTERVENTIONS/APPROACHES RECOMMENDED FOR FUTURE PROGRAMMING .............................................................................................................. 37 ANNEXES Annex I: Evaluation Statement of Work Annex 2: Evaluation Matrix Annex 3: List of Persons Interviewed Annex 4: List of Documents Reviewed Annex 5: Illustrative Questionnaires for Key Informant Interviews and Focus Group Discussions Annex 6: Mini-Survey Questionnaire Annex 7: Disclosure of Conflicts of Interest Annex 8: G3 Grantees Major Activities Annex 9: Additional Tables TABLES Table 1: Dates, Location and Focus of FGDs Table 2: Place of Employment at the Time of Training Table 3: Level of Government When Trained Table 4: Type of Training Attended Table 5: Other Capacity Building Received Table 6: Internet Usage – Capital, Urban, Rural Table 7: Increase in Competence Level Table 8: Ease of Understanding of Training Table 9.1: Effectiveness in teaching new skills and building capacity Table 9.2: Effectiveness in teaching new skills and building capacity by respondent level of government Table 10: Factors that Made Capacity Building Mechanism Effective Table 11: Factors That Made Each Type of Capacity Building Effective? Table 12: Improvements as a Result of Capacity Building Table 13: Preferred Areas of Future Capacity Building Table 14: Single Preferred Topic of Future Capacity Building Table 15: Possible Future Skill Areas for Capacity Building ACRONYMS AYEG Association of Young Economists of Georgia BDD Basic Data and Directions CAC Community/Civic Advisory Council CATI Computer Assisted Telephone Interview CCE Center for Civic Engagement CCG Chamber of Commerce Georgia CEMS Civic Electronic Monitoring System CIDA Civil Development Agency CoP Chief of Party CRA Civil Registry Agency CRRC Caucasus Research Resource Center – Georgia CSB Civil Service Bureau CSI Civil Society Institute CSO Civil Society Organization CSPA Competition and State Procurement Agency CSRDG Center for Strategic Research and Development of Georgia CTC Center for Training and Consultancy DEA Data Exchange Agency D&G Democracy and Governance EU European Union FGD Focus Group Discussion FOI Freedom of Information FOIA Freedom of Information Act G-Media Georgian Media Enhance Democracy, Informed Citizenry and Accountability GDC Georgia Dream Coalition GIPA Georgian Institute of Public Affairs GFSIS Georgian Foundation for Strategic and International Studies GIZ German Agency for International Cooperation GoARA Government of Autonomous Republic of Ajara GoG Government of Georgia GYLA Georgian Young Lawyers Association HICD Human and Institutional Capacity Development HRM Human Resources Management IDFI Institute for Development of Freedom of Information IAP Istanbul Anti-Corruption Action Plan IG Inspector General IRC Innovations and Reforms Center IRI International Republican Institute IRMS IT, Research and Metadata Solutions ISET International School of Economics at Tbilisi State University LG Local Government LSG Local Self-Government KII Key Informant Interview M&E Monitoring and Evaluation MAAP Municipal Assistance Action Plan ME&A Mendez England & Associates MoES Ministry of Education and Science MoF Ministry of Finance MoH Ministry of Health Care and Social Protection MoJ Ministry of Justice MoU Memorandum of Understanding MRDI Ministry of Regional Development and Infrastructure MSI Management Systems International MSPA Municipal Service Providers Association MTCC Municipal Training Coordination Council MTEF Medium Term Expenditure Framework NALAG National Association of Local Authorities of Georgia NAPR National Agency for Public Registry NCLA Non-Commercial Legal Entity NDI National Democratic Institute NGO Non-Governmental Organization NSO National Statistics Office OECD Organization for Economic Co-operation and Development OGP Open Government Program OGPI Open Government Partnership Initiative OSGF Open Society Georgia Foundation PAR Public Administration Reform Project PMCG Policy and Management Consulting Group PMEP Performance Monitoring and Evaluation Plan PMO Prime Minister’s Office PMP Performance Management Plan RDC Regional Development Council RS Revenue Service SAOG State Audit Office of Georgia SEDP Social and Economic Development Plan SGC Self-Governing City SIDA Swedish International Development Agency SIP Service Improvement Plan SOW Statement of Work STTA Short Term Technical Assistance SUP Strategic Urban Plan TA Technical Assistance TI Transparency International UNDP United Nations Development Programme UNM United National Movement USAID United States Agency for International Development EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS This is a report on the Mid-Term Performance Evaluation of the Good Governance in Georgia (G3) project funded by the United States Agency for International Development (USAID) Mission in Georgia. The project is being implemented by Management Systems International (MSI) between February 2011 and March 2014. The evaluation of the G3 project was conducted in December 2013 by a team assembled by Mendez England & Associates (ME&A) and NORC. The team consisted of four specialists with extensive experience in governance and local government (LG), and one evaluation expert. The overall purpose of the evaluation was to provide the USAID Office of Democracy and Governance (D&G) and the wider USAID Mission with information that will assist them in making decisions about on-going and related future programming. The D&G Office is particularly interested in knowing which interventions and/or approaches have proven to be feasible and sustainable in the changing political context, and may provide a sound foundation for future programming. The evaluation focused on the first two components of G3: Component 1, which relates to improved public administration at the federal level; and Component 2, which supports a more responsive, professional, and engaged local governance. Per USAID’s request, Component 3 was not evaluated. An integral part of the evaluation mission was to answer a set of three pre-determined questions, which are detailed in the body of this report. The questions concern: 1. Impact of G3 on government transparency and accountability 2. Effectiveness of G3 on government mechanisms for public engagement 3. Sustainability and impact of G3’s capacity development for government officials The implementation of G3 was affected by the political dynamics taking place in Georgia during the life of the project. The findings, conclusions and recommendations in this report take into consideration the impact of the country’s political changes on the project. PROJECT BACKGROUND Georgia has experienced considerable political changes during the past 20 years. The country did not start a real transition to democracy until after the 2004 Rose Revolution, which put an end to Eduard Shevardnadze’s rule of nearly 10 years. The 2004 elections brought Mikheil Saakashvili and his United National Movement (UNM) to power, initiating a period of drastic economic and political reforms. The new government made significant efforts to create a market economy and to improve tax administration, crack down on petty corruption, and improve revenues. Missing from the reforms, however, were other aspects of governance, including more transparency and accountability, as well as inclusion of citizens in the decision-making process. Thus, an overarching objective of G3, which started in February 2011, was to strengthen citizens’ ability to combat corruption and put into place stronger mechanisms and systems that prevent opportunities for corruption from developing. In October 2012, a year and a half after the start of G3, the Georgia Dream Coalition (GDC) led by Bidzinia Ivanishvili, brought together a fragmented opposition and won Parliamentary elections. The party promised more transparency, openness, and greater citizens’ participation, but its victory also created political waves throughout Georgia, primarily affecting local governance. In November 2013, the new GDC leadership came into office. As a result, power balance shifted from the President’s Office to the Office of the Prime Minister (PMO). It is under this constantly changing political environment that G3 has operated and implemented its activities. EVALUATION METHODS The methodology for this evaluation was carefully designed to respond to the evaluation questions posed in the Statement of Work (see Annex 1) and to determine whether G3 has made progress towards its initial objectives. The Evaluation Team collected quantitative and qualitative data from a broad range of stakeholders and beneficiaries to ensure objectivity as well as accuracy and i viii completeness of the findings, conclusions and recommendations. Data was collected by a number of methods and sources, including:  Critical desktop review of materials related to G3, such as project quarterly and monthly reports, annual work plans, performance management plans (PMPs), project design documents, and communication among partners. The Evaluation Team also reviewed a wide range of project-related documents, such as Georgia’s Open Government Program (OPG) and Action Plan, news articles for background information, and more.  In-depth interviews with USAID and G3 staff.  Over 100 semi-structured, face-to-face, or phone interviews with partners, project beneficiaries and stakeholders, and key informants from the government, organizations that provided consulting and training services to G3 either as subcontractors or grantees, international donors, and more.  Field visits to 4 of G3’s 10 partner communities (Rustavi and Telavi in the East; and Batumi and Kutaisi in the West), and interviews in each location with high elected officials, professional staff, civil society organizations (CSOs), media grantees, G3 regional coordinators (past and present), and Center for Civic Engagement (CCE) staff.  Six focus group discussions (FGDs) with CSOs, local journalists, students, and civil servants in different locations - 3 in Tbilisi, 1in Batumi, and 2 in Kutaisi.  One mini-survey of national- and local-level government employees who had participated in G3’s capacity development activities with a randomly selected sample of 152 participants. The Evaluation Team retained the services of IT, Research & Metadata Solutions (IRMS), a Georgian firm, to assist in implementing the mini-survey and FGDs. EVALUATION LIMITATIONS The Evaluation Team encountered some limitations inherent to the design of this evaluation and during its fieldwork in Georgia. The most relevant limitations included: Lack of baseline data. There was little or no baseline survey from the start of the program that could be used to measure progress. Halo bias. The extent to which respondents were prepared to reveal their true opinions may have varied for some questions that called upon them to assess the performance of their colleagues or people on whom they depend upon for provision of services. Limited time allotted for the evaluation. The evaluation period included several holidays in the US as well as in Georgia. Preparation time was tight and the Evaluation Team received considerable amount of documents while in route to Georgia. There were time constrains in evaluating such a large and complex project with diverse components, numerous activities, and several counterparts. Not all G3 activities could be studied in-depth by this evaluation or all documents read and analyzed accordingly. Although the Evaluation Team met with at least 100 individuals, not all key informants could be interviewed either because of short time in Tbilisi and/or scheduling conflicts. Low turnout at FGDs. Some of the FGDs were conducted with fewer than anticipated participants. Many who had been invited to participate, confirmed attendance but did not show up. The Evaluation Team tried to mitigate the above limitations by focusing on activities and meetings with key informants most closely linked to addressing the evaluation questions. It also tried to target a larger number of respondents through the use of focus groups and a mini-survey to reach government officials who benefitted from the G3’s capacity building interventions. In addition to using a variety of data collection methods to maximize the amount of information gathered, ME&A applied triangulation to increase validity and reliability of results. FINDINGS Question 1: Impact on Government Transparency and Accountability 1. Since 2004, the overall effectiveness of Georgia’s governance had improved dramatically but the openness, responsiveness and accountability of the governance sector remained problematic. G3 ix was designed to address good governance at the central and local levels, focusing primarily on citizens’ participation and improving civil servants professionalism. 2. Prior to the 2012 elections, G3 assisted Government of Georgia (GoG) institutions in establishing and strengthening administrative mechanisms and procedures. Among the institutions supported were the Civil Service Bureau (CSB), Civil Registry Agency (CRA), the Ministry of Justice (MoJ), the Chamber of Commerce of Georgia (CCG), and the National Agency for Public Registry (NAPR), as well as the Government of Autonomous Republic of Ajara (GoARA). Change of government brought more political will for reforms and opened the doors for G3 to provide more targeted assistance to GoG. 3. G3 provided continuous support to GoARA, assisting it to elaborate and improve financial as well as performance indicator data reporting and thus, improve transparency and accountability as well as enhance performance of public service delivery. 4. G3’s support to GoARA in implementing e-Governance is considered by government officials in Tbilisi as the model to follow. 5. In support of GoG’s Action Plan under the Open Government Program (OGP) initiative, G3 worked with GoG and CSOs to improve government processes in defining and implementing Freedom of Information Act (FOIA) standards. Key Informants (KIs) interviewed by the Evaluation Team, however, are concerned that GoG may be relying on donors to promote OGPI and may fail to assume full responsibility for carrying out the initiative. They would like for the government to take more ownership of the process in direct partnership with the CSOs. They are also concerned with the level of understanding of the scope of OGP among government officials, as well as the lack of knowledge among citizens of the OGP initiatives’ impact on them. 6. Availability and usage of internet in Georgia is relatively low, particularly outside of major urban centers. Use of e-Governance is also low. 7. G3 has played a positive role in increasing transparency of information to citizens and improving internal communication within GoG and GoARA institutions. Focus Groups’ participants explained that GoG, GoARA and local governments (LGs) do not exhibit the same level of efficiency and responsiveness when providing information to citizens. 8. At the local level because of the election process, there was no initiation of any real sustained work on planned Strategic Urban Plans (SUPs) and Service Improvement Plan (SIP) activities. Social and Economic Development Plans (SEDPs) were initiated but remained in preparatory stages during 2012 and made no contribution to the budget process that year. 9. The media grants that G3 provided resulted in an increase of the number of programs that covered municipal government issues. However, media representatives interviewed by the Evaluation Team confided that with the termination of grants, their stations will revert to their pre-grant level of LG coverage, meaning continuation of chat-shows and reporting but termination of sakrebulo meetings, which did not attract a wide audience. 10. G3’s CSO grantees contributed most to improving municipal web-based interfaces (web portals and Facebook pages) in Rustavi and Kutaisi, and to some lesser extent in Telavi and Akhaltsikhe. Several partner websites have features such as site design, functionality and content, which could be emulated elsewhere. However, although G3’s assistance has helped improve communication of information, more needs to be done. Electronic sites supported by G3 are not very sophisticated and often lack information that is useful and timely for citizens. 11. The Civil Electronic Monitoring Systems (CEMS), which allows residents to report common problems to the appropriate local authority, as well as monitor the responsiveness of those responsible for fixing the reported problems, represents perhaps the most innovative use of G3 CSO grant funding. Question 2: Effectiveness of Government Mechanisms for Public Engagement 1. A key element of G3 efforts to encourage Executive Branch institutions to engage citizens included assisting in promoting modern Public Relations (PR) and Communications mechanisms as a means to effectively communicate with stakeholders and the public. This first step helped in institutionalizing the use of interactive mechanisms, such as town hall meetings and other forms of outreach. 2. G3 worked with GoARA on institutionalizing public outreach and citizen engagement x mechanisms, such as stakeholder discussions, public meetings and discussions, and Town Hall meetings. It assisted GoARA in establishing Priorities Committee to facilitate GoARA-CSO engagement. These mechanisms have been officially adopted and approved by GoARA. In addition, citizen engagement has become important and obligatory part of Budget Preparation process at GoARA. 3. Many FGD participants perceived that public engagement is practically non-existent at the national level and that citizens may attend discussions but are not allowed participation. A number of them blamed this weak public engagement on the apathy of citizens instead of lack of initiative from the government. 4. Governance has centralized over the last decade, with LGs having increasingly limited fiscal and functional autonomy over service delivery. This has resulted in lack of citizens’ interest to participate in public engagement. 5. It remains unclear whether the Community/Civic Advisory Council (CAC) represents a viable vehicle for institutionalizing citizen engagement at the local level. With few exceptions, local leaders have not genuinely embraced the concept. The Batumi model of an umbrella CAC (grounded in territorial darbazys and with additional representation from media, businesses, and CSOs) seems to be the most promising CAC initiative on which to build. 6. The outcomes of the completed SEDP vary substantially in terms of citizen involvement. Based on the Evaluation Team’s interviews and documentation review, the municipalities that have the best participatory processes did not produce the best SEDP. SEDP participants revealed that the SEDP concept faces constraints in becoming of proven value and accepted as a part of LG operations. At present, SEDPs are not integrated with legally mandated planning and budgeting processes. As such, they are regarded by LG officials and senior management as extra work beyond their mandated responsibilities. 7. Counterparts believe the G3-funded technical support for Strategic Urban Plans (SUPs) was of high quality. In particular, the follow-on financing of the Master Planning is greatly valued, since the cost of the needed professional services is prohibitively expensive. 8. Of perhaps all major initiatives, the Service Improvement Plan (SIP) activity suffered the most from the pre- and post-election turmoil. As a result, with the exception of Rustavi, the SIP documents were prepared with little, if any, engagement of the LG with citizens. Given the difficulties encountered in engaging local counterparts effectively, USAID and G3 terminated the SIP initiative early. 9. Most LGs have yet to institutionalize public consultation in the budget process and priority setting at the most critical junctures in the annual budget calendar. Question 3: Sustainability and Impact of Capacity Development for Government Officials 1. Trainings designed based on a needs assessment of the agency and the civil servants, and coordinated with the agency have proved to be effective. 2. At the local level, G3 delivered training in too many topics, some of which were not related to G3’s assistance. 3. A majority of KIs expressed their satisfaction with trainers and the content of training. 37% of respondents attribute this to both content/relevance of the capacity building and quality of the individual that delivered the training. However, some of them, who had attended more than one training, felt that trainers differ in their capacity and knowledge of content, which affects the quality of training. They also stated that training sometimes is rushed and that attending too many courses is affecting the time that they spend in their offices. 4. Over 62% of mini-survey respondents identified coaching as the most effective mechanism in teaching new skills and building capacity. GoARA’s ability to produce adequate program budgeting as a result of on-the-job training and coaching supports the survey responses. 5. Internships such as that of a Performance Auditor at US General Accounting Office, have provided positive results in transferring know-how back to Georgia. 6. There is no established baseline to measure impact and sustainability of G3’s interventions in capacity development. 7. Trained public servants and trainers indicated that longer term trainings of pre-evaluated participants on specific subjects might bring better results than short-term trainings at a massive scale without previous needs assessment or pre-qualification of candidates. 8. Through key informant interviews (KIIs), the Evaluation Team found that several capacity xi development implementers and CSO representatives believe that by providing most training, international donors are freeing GoG from assuming more responsibility for training its own staff. These respondents argue that cost-sharing increases government’s sense of ownership. 9. According to the mini-survey, participants would like to see more training in topics such as Program Budgeting, Human Resources (HR) Management, and Public Relations. Program Budgeting, in particular, was mentioned as a top priority by over 22% of respondents and was also self-reported as less than a “normal” competence level. Data from the mini-survey points to a need for continued capacity building given self-reports of lower than normal competence on average across the training topics, and the request for additional training for almost all training topics. CONCLUSIONS Question 1: Impact on Government Transparency and Accountability 1. G3’s interventions followed on a continuum along with the GoG’s democratic transition process and in response to opportunities granted by changes in levels of political will. 2. Assistance provided to GoG prior to elections helped strengthen institutions and establish the groundwork for further support after the change of government. 3. G3’s assistance to GoARA, which included training and on-the job coaching, resulted in GoARA preparing a program budget for 2014 that surpassed the Evaluation Team’s expectations. 4. GoARA’s more effective e-Governance is the result of having a smaller government and of G3’s assistance in conducting a needs assessment that helped provided more targeted support. 5. G3’s technical assistance to GoG and dialogue facilitation with CSOs, particularly on the OGP Action Plan, has been very useful and helpful. Many government officials appreciate the support and timeliness of G3 response to their needs. CSOs believe that this assistance has helped the GoG become more transparent, yet are expressing valid concerns in terms of the impact that donor assistance could have on the GoG’s willingness to assume ownership of the process. 6. Increased use of e-Governance may depend on expansion of Internet availability throughout the country and more people becoming familiar with the system. 7. Level of understanding and commitment of public officials to provide information to citizens may explain the disparity in the availability of information among institutions and regions. 8. The delay in implementing a number of G3 initiatives at the local level, resulted from the election process and turmoil created by some of the results of national and municipal elections. 9, 10 G3’s regional media grants did result in reasonable value that USAID received for its investment in terms of the number and hours of broadcasting dedicated to municipal affairs. This increased coverage translated into some enhanced awareness and understanding of important local governance matters in the short-term. In addition, participating media personnel gained more experience with municipal issues. However, the grants resulted in substantial increase in quantity rather than quality of the coverage on municipal government affairs. Therefore, the sustainable impact and overall cost-effectiveness of this approach is questionable. 11. CEMS is a useful platform with good prospects for replication in other larger, more urbanized jurisdictions. Question 2: Effectiveness of Government Mechanisms for Public Engagement 1. Perhaps it is too early for the initiatives and mechanisms supported by G3 for public engagement at the national level to become institutionalized. Although mechanisms have been created and/or are in the process of being established, there is not enough evidence of their sustainability and there is no baseline to measure it. As in most other initiatives, much will depend on the political will of the leadership to formalize mechanisms and reach out to stakeholders. 2. The close collaboration and continuity to GoARA shows positive results and often contrasts with more modest achievements in other regions. 3. Many citizens avoid public participation and engagement, especially at the national level, because they do not see the value of it and are not aware of their rights. 4. KIs interviewed believe that without a more genuine local self-governance, no donor program will be able to promote citizen engagement at the local level. 5. CACs work best when addressing specific “bread and butter” issues close to people’s xii immediate concerns, and when LG has already established a relationship with the citizens. Batumi’s CAC is a successful outlier in respect to both the form and quality of engagement. 6. SEDP type strategic planning at the local level remains an important objective. However, in order for SEDPs to become a valuable and acceptable tool, they must define action plans with projects defined and costed out in sufficient detail to be useful for annual planning purposes. Counterparts indicated that they would have liked more technical consulting services in achieving this level of specificity. 7. G3’s support (still in progress) for SUP exercises promises to demonstrate the value of this type of participatory planning for preparing the legally mandated, Spatial and Land Use Master Plans. 8. Given the political turmoil in partner LGs, it is hard to assess the merits of the SIP concept as implemented by Municipal Service Providers Associations (MSPA). The SIP would seem to have some unrealized potential in demonstrating the inherent value of integrating performance management concepts into the delivery of core municipal services. 9. Citizen engagement in budget adoption, except in the GoARA, has sometimes focused too narrowly on end-of-the year hearings, rather than on more strategic points in the annual budget calendar. Some precedents for officials from G3 LGs getting out to neighborhoods and villages early in the budget cycle can be confirmed (most notably in Batumi and Rustavi), but one cannot yet describe such consultations as an institutionalized mechanism. They do, however, provide positive models for other LGs to emulate. In GoARA, however, citizen involvement during the budget preparation spans at different points in the year and is an important part of GoARA budget planning process. Question 3: Sustainability and Impact of Capacity Development for Government Officials 1. Conducting a needs assessment for agencies prior to designing and providing training is an essential first step to achieving better results for the institutions. 2. At the local level, the major curriculum based series of training workshops seemed to have been delivered but, perhaps on too many topics. Eight of the eleven workshops were unconnected to on-going G3’s technical assistance. 3. Although most participants praised the trainers and the content of training, some felt that there were considerable disparities in the training capacity and knowledge of different trainers. In addition, some interviewers felt that they were not allowed enough time to prepare, and that staff is often required to spend too much time out of their office. 4. Based on KIIs, most effective training and sustainable capacity development efforts have resulted from a combination of training and coaching, or on-the-job assistance to apply the training, as seen on the program budget results in GoARA. 5. The State Audit Office (SAO) is another example of sustained results due to the long term four-month training of a key staff member. 6. Lack of baseline data makes it difficult to measure the impact and sustainability of the project’s capacity development interventions. 7. Pre-evaluation of selected participants was considered very important and a step that might produce better and more sustainable results than short-term training of civil servants that are not screened for their qualifications. 8. Among CSO representatives, there is a concern that international assistance may be allowing the GoG to assume less responsibilities in capacity building than it has the capacity to assume. Many respondents feel that cost-sharing increases government’s sense of ownership. 9. While there are no clear self-reported priority areas for future capacity building, the mini-survey suggests possible topics that should be further explored must include Program Budgeting, HR Management, and Public Relations, with Program Budgeting being a top priority. There is also a need for continued capacity building for almost all other topics. 10. Looking only at the mini-survey results, at the GoG level, coaching and facilitation were very highly rated as effective capacity building mechanisms, whereas among the GoARA coaching was more favored than facilitation. RECOMMENDATIONS Question 1: Impact on Government Transparency and Accountability 1. Take advantage of the political will and momentum to deepen reforms in areas where G3 xiii has proven successful. 2. Consider providing structural anticorruption support by assisting GoG implement the recommendations of the IPA Monitoring Report, as well as those recommendations made by TI- Georgia to the GoG. 1 3. Replicate, to the extent possible, methodology used for capacity building in GoARA. 4, 5 Continue building consensus between the government and the governed at every level. 6,7 Continue to encourage GoG and LG to improve dissemination of information and increase transparency, which is still limited. 8. Increase people’s awareness about the usefulness and benefits of e-Governance. E-Governance should concentrate on areas where service delivery may be improved, and ensure that it is user- friendly and effective. Attention should be given to co-sponsoring e-Governance assistance with appropriate GoG and national NGO partners, and providing annual or biannual awards for the best national, regional, and local e-Governance initiatives in different categories – e.g. service delivery to citizens, business-friendly governance, and internal government operations. Such an award could include periodic documentation and assessment of best practice models. 9,10 In future USAID programming, meet with CSOs to hear their views on the OGP and role of international donors. Consider offering CSOs a broader and more expanded role. Include academics and private sector as part of the CSO network for the OGP. 11. Continue to work with GoG and LGs in order to increase their awareness about the importance of timely and transparent information. In respect to LGs, as being considered by TI Georgia, encourage GoG regulation that would specify a core set of proactive disclosure requirements for LGs, including a consolidated list of documents whose publication is already mandated by law. 12. Work with partner institutions in providing electronic information that is more users friendly and timelier to citizens’ needs. 13. Future projects should establish a baseline to measure results. 14. Lessons learned from the G3 media grants, together with approaches tested through the IREX media programs, suggest that there are other viable options for including media coverage of LG affairs in future USAID local governance programming that might promise a more cost￾effective impact than just providing grant funding to regional TV outlets directly. According to IREX, these include hiring experienced journalists who can spend time working directly with the journalists; and not granting full ownership of the project to local journalists/station because they need good external consultants, among others. 15. Encourage more experimentation with interactive website features, such as CEMS and use of social media. The annual e-Governance award might offer an attractive vehicle for advancing this end. Question 2: Effectiveness of Government Mechanisms for Public Engagement 1. Establish baselines that will facilitate the measurement of success of public engagement mechanisms. 2. Encourage GoG to systematically provide information to citizens and encourage them to fully participate in public engagement. Increase citizens’ education in civic engagement and rights. 3. Promote further reform of intergovernmental relations and civil service to truly energize citizen engagement at the local level. 4. Monitor and assess the Batumi CAC model to examine its replication in jurisdictions of varying sizes and degrees of urbanization. 5. Provide technical assistance to counterparts in achieving a higher level of specificity when drafting SEDPs. 6. Look to amending the governing laws on spatial planning to make a SUP type process an integral, front-end part of the legally mandated master planning process. Policy reform should also 1 TI-Georgia recommendations include: Addressing corruption risks arising from the movement of government officials between public and private sectors and the abuse of public office for the benefit of specific private companies; ensuring the transparent management of Georgia’s defense budget; and improving the administrative capabilities and oversight capacity of the Parliament, especially over the law enforcement and investigative agencies. See: Georgia in TI’s 2013 Corruption Perception Index Civil Georgia (Tbilisi, 3 December 2013) http://www.civil.ge/eng/article.php?id=26755. xiv consider moving oversight of local spatial plans from the Ministry of Economic Development and Science (MoES) to MRDI, which is more fully coordinated with central government financial support for local infrastructure investment. 7. In any future USAID local governance planning, consideration should be given to revisiting the most promising aspects of the SIP concept. That said, less emphasis should be placed on producing stand-alone technical documents, and more on support for modified SIPs that engage citizens in service improvements identified as most immediate, and utilize citizen participant observation in monitoring results. 8. In future programming, look to institutionalize public engagement during the critical March to August span of the budget calendar, and allocating regional development funds and village improvement funds calendar. Question 3: Sustainability and Impact of Capacity Development for Government Officials 1. Course offering should be better coordinated with the needs of the agencies and their staff. 2. Provide more targeted training, learning from the best practices experienced by GoARA and other national institutions or agencies. 3. Standardize quality of trainings and trainers, and facilitate training materials in advance. 4. Allow for more time and better planning when preparing for capacity development of civil servants to ensure employees have adequate time to prepare. To minimize employees’ time outside of the office and encourage good governance practices, consider developing required on￾line courses on basic topics for all civil service employees, such as conflict of interest, transparency, management of government property, discrimination, sexual harassment, and others, to be conducted periodically. 5. Consider focusing more on longer-term training combined with on-the-job training rather than short-term, large scale training. 6. Any future project should establish baselines that will assist in measuring success of the sustainability of capacity development interventions. 7. Selection of participants for certain training should be made based on their qualifications and their needs. 8. Encourage the GoG to share costs for training of its employees, as cost-sharing is known to increase government’s sense of ownership. 9. Focus and embed training in ongoing technical assistance (TA) in applying concepts learned since many civil servants pointed to the need for continued capacity building. Provide more training in the areas identified as more important and problematic, including Program Budgeting, HR Management, and Public Relations. 11. Consider providing coaching and on-the-job training for better and more sustainable training results. INTERVENTIONS/APPROACHES RECOMMENDED FOR FUTURE PROGRAMMING • Numerous G3 interventions and approaches have proven successful and should be continued in future programming. However, G3 initiatives have succeeded whenever political will to carry out reforms has been present meaning that without political will, a program like G3 should not be funded. • In the future, USAID should attempt to strike a balance between hands-off and hands-on approaches. There have been cases in which failure to follow up and monitor an agency’s implementation of training/coaching has led to the activity not achieving expected results. This is the case, for example, of the MoJ’s Performance Appraisal for which G3 did not do a follow-on when its implementation was placed on hold. • While G3 has served as “mediator” in bringing GoG and CSOs closer, it should consider coaching and assisting the GoG in engaging CSOs directly as essential collaborators and also critical watchdogs. KIs suggested that G3 and future programming could help GoG understand that it should work directly with think tanks and CSOs. These organizations support the government, through international donors, in capacity development, research and other activities. They would like to provide services directly to the government. • Use of local talent as USAID consultants is a welcomed approach. However, KIIs as well as the mini-survey that the Evaluation Team conducted with representatives of CSOs and civil servants, xv revealed that the combination of local and international experts is considered the best option because the international experts provide experience and fresh ideas while the local experts help contextualize them. • The contributions G3 has made in building knowledge and providing international experts who conduct in-depth studies on key areas, such as the drafting of the civil service reform concept, are well-appreciated by GoG as well as by local implementers and CSOs representatives. GoG would like that type of assistance to continue. In fact, the PMO would like to be able to use a similar methodology to that used for the civil service reform concept paper to move toward a full public administrative reform with USAID’s support. • Look at forthcoming local election campaigns as an opportunity to increase awareness of local governance issues and inject into campaign dialogue the discussion of needed intergovernmental relations and LG reforms. • Promote the replication of local CEMS, and possibly support LGs in performance-based management practices. Implementation plans should include a more organized approach to training and mentoring LG system administrators and coordinators from the outset, including incorporating new features of available open-source software packages. • In future local governance programming, more focus should be paid to developing and utilizing performance measures in key targeted sectors of LG operation. Explore opportunities to do so in cooperation with the ongoing MoF/GiZ initiative to support fuller implementation of the now mandated program budgeting at the local level. Consider possibilities for incorporating joint LG/CAC monitoring of CEMS databases in any such performance measurement system. • If supporting the further development of civil society and media watchdog capabilities, do so through a separate CSO-focused contracting or grant vehicle. Focus on developing more sophisticated capabilities to monitor the core accountability functions with respect to budget execution, asset management (land sales/leases), and procurement. • Explore opportunities for USAID to provide planned certification programs for financial managers, auditors, procurement officers, and municipal lawyers. • At the local level, conduct cluster-based training that brings professional peers together to prepare action plans and reconvene them periodically to foster peer mentoring and best practice sharing among LGs and the development of professional expert networks, and possibly new associations (e.g. for HR officers, for procurement officers). • In the near term, especially if G3 is extended into the summer of 2014, USAID should look for ways to involve the media and CSOs in injecting a more sophisticated dialogue on LG policy issues into the local electoral campaign. It may still be timely to cooperate with IREX’s planned Regional Election Initiative in 2014, particularly by helping provide content on front burner LG issues into electoral coverage, priming journalists to probe candidates’ positions on topics such as merit-based hiring practices at the local level.  In any major new local governance project, rather than providing grants directly to TV stations themselves, consideration might be given to building the capacity of broadcasting personnel and print journalists in covering municipal matters. This could include relatively-low cost concepts such as: - Media Breakfasts: invite media reps to brunches with panelists on LG topics of current interests, and take advantage of visiting short-term international experts to share their experience. Providing capacity building in this type of more informal/social setting has proven more effective in engaging practicing journalists who are reluctant to take the time off to attend formal trainings. - Specialized training on LG topics with promising angles as the subject for investigative journalism. Starting in March 2014, IREX is planning to host a 9-day school of journalism. For this training or others, USAID might support the development of a module on one or more topics where corruption, waste, and conflicts of interest are most rife at the local level. Land relations, including the misuse of eminent domain powers and non-competitive land sales, might prove fruitful topics. • Procure high quality programming content to share with broadcast media throughout the country. Through direct grants, USAID should contract with a TV production or documentary film company to develop a series of programs on priority topics for LG reform. The programs xvi could then be provided free of charge to any Georgian TV station interested in showing it. This approach could deliver higher quality content to a larger number of viewers than funding multiple regional media outlets directly. In a well-designed, 3-5 year project, this approach can prove effective during the initiative’s concluding two years as a vehicle to document and promote best practices and lessons learned. • If and when USAID again supports broadcasters directly, the Evaluation Team recommends that eligible grantees be defined broadly enough to permit a competitive application process, in the interest of eliciting more creative proposals and allowing prospective media grantees to be screened based on their management qualifications. As part of any such grant program, USAID might also look for practical ease to document viewer responses to the supported programming. This documentation could then be used by the media grantee to help market public programming to prospective advertisers after the grant support has wound down. • Surveys by the Caucasus Research Resource Center – Georgia (CRRC) and others indicate that over 90% of the Georgian adult population relies primarily on TV for information on public affairs. Thus, any donor program looking to inform the public at large needs to include a TV component. Relying solely on print, internet, and radio is not a viable option. Social media should also be a viable means. 1 1.0 EVALUATION PURPOSE AND EVALUATION QUESTIONS 1.1 EVALUATION PURPOSE This is a report on the Mid-Term Performance Evaluation of the Good Governance in Georgia (G3) project funded by the United States Agency for International Development (USAID) Mission in Georgia. The project is being implemented by Management Systems International (MSI) between February 2011 and March 2014. The evaluation was conducted in December 2013, and covered the period from G3’s start in February 2011 through December 2013 – practically the project’s full extent. Although G3 has three components, the evaluation focused on only two of them: Component 1, which relates to improved public administration at the federal level; and Component 2, which supports a more responsive, professional, and engaged local governance. Component 3, which deals with citizens’ participation, was not evaluated per USAID’s request. According to the Statement of Work (SOW) (see Annex 1), the overall purpose of the evaluation was to comply with the task order’s monitoring and evaluation (M&E) plan requirement for a mid￾term evaluation, as well as to inform on-going and related future programming for the USAID Office of Democracy and Governance (D&G) and the wider USAID Mission. The D&G Office is particularly interested in knowing the interventions and/or approaches that: 1) have proven to be feasible/sustainable in the changing political context; and 2) provide a sound foundation for future programming. USAID/Georgia, particularly its D&G Office, is the intended audience of this evaluation although results will also be shared with MSI and G3 staff. The evaluation was conducted by a team of five experts: Dr. Olga Nazario (Team Leader and Governance Expert), Dr. Peter Epstein (Local Government Expert), Ms. Mawadda Damon (Evaluation Expert), Ms. Nina Khatiskatsi (Georgian Local Governance Expert), and Mr. Mikheil Kukava (Georgian Transparency and Accountability Expert). Ms. Aytan Gahramanova, USAID/Baku Economic Development Officer, joined the Evaluation Team for the second and third weeks of fieldwork as part of an experience exchange among USAID Caucasus Missions’ staff. 1.2 EVALUATION QUESTIONS The Evaluation Team was tasked with answering a number of specific questions, outlined below. Taking into consideration the political context in which G3 operated, the team emphasized its research on these questions and sub-questions; however, it also analyzed the project’s overall progress towards goals and objectives, as well as the overall effectiveness of its implementation. The Evaluation Matrix (see Annex 2) describes the methods and sources used to research each question. The questions included: Question 1: Impact of Government Transparency and Accountability • What results did G3 achieve both before and after the 2012 change of government? • How have G3 efforts in e-governance advanced transparency and effectiveness in the agencies targeted for assistance? • How have G3’s efforts in advancing freedom of information (FOI) compliance and facilitation of Open Government Program (OGP) Action Plan implementation process contributed to transparency, openness or responsiveness of the targeted Government of Georgia (GoG) institutions? • On the local level, what contributions have G3 grants to regional media outlets and local Non￾Government Organizations (NGOs) made to citizen awareness and engagement related to transparency and accountability of local government (LG)? Question 2: Effectiveness of Government Mechanisms for Public Engagement • To what degree has the G3 approach been successful in building and sustaining effective mechanisms for public engagement at the national [Government of Autonomous Republic of Ajara (GoARA); State Competition and Procurement Agency (CSPA); Ministry of Justice (MoJ)] and local levels? 2 • Which aspects worked, which did not and why? • On the municipal level, have partner LGs institutionalized mechanisms of public engagement (public hearings on budgets and other major functions, public calendars, bulletin boards, press releases, display boards or other forms of public hearings)? Question 3: Sustainability and Impact of Capacity Development for Government Officials • Which G3 training and capacity building efforts within national and local level government institutions have been most effective? • To what extent have targeted institutions sustained results of G3 interventions? • When they did, what were the key factors that contributed to this? • Which training and capacity building approaches would be most useful to adopt for future efforts to sustainably improve the capacity and skills of government officials? 2.1 PROJECT BACKGROUND Georgia has experienced considerable political changes during the past 20 years. The country did not start a real transition to democracy until after the 2004 Rose Revolution, which put an end to Eduard Shevardnadze’s nearly 10 years rule. The 2004 elections brought Mikheil Saakashvili and his United National Movement (UNM) to power, initiating a period of drastic economic and political reforms. Since then, Georgia has made significant strides in carrying out anticorruption reforms during the transition to democracy. The Saakashvili government succeeded in reducing administrative petit corruption, including the full revamping of the traffic police, creating e-Procurement and improving the tax administration. In 2010, the GoG committed itself to adopting the Second Round Monitoring Report of the Istanbul Anti-Corruption Action Plan (IAP). 2 In 2011, Georgia joined the Open Government Partnership Initiative (OGPI), agreeing to a transparent and collaborative process in public service and other grand challenges. Thus, the GoG was on a reform path when G3 started in February 2011. The reforms and anticorruption commitments, however, did not include other aspects of governance. There was little opportunity for citizens’ participation or consultation, and the civil services remained patronage-based. There was not much transparency and accountability. In October 2011, just a few months after the implementation of G3 began, Bidzinia Ivanishvili created the Georgia Dream Coalition (GDC), which brought together a fragmented opposition. In October 2012, the GDC - which promised more transparency, openness, and greater citizens’ participation - won a majority of seats in Parliament. At the same time, the emerging new party created political waves throughout Georgia, affecting local leaders and governance. In November 2013, the new GDC leadership came into office. As a result, power balance shifted from the President’s Office to the Office of the Prime Minister (PMO). G3 has operated under this changing political environment, seeking to maximize opportunities for collaboration with the GoG. G3 is a $16 million complex project designed to serve as a continuation of the USAID/Public Administration Reform (PAR) project. G3 is designed to address the various challenges facing governance in Georgia at the local and national levels. The project seeks to strengthen citizens’ abilities to combat corruption and to put into place stronger mechanisms and systems that prevent opportunities for corruption to develop. G3 has three components: • Component 1: Improve Governance and Transparency at the Federal Level 2 Georgia endorsed the IAP in 2003. It is the main sub-regional initiative in the framework of the Organization for Economic Co￾Operation and Development (OECD) Anti-Corruption Network for Eastern Europe and Central Asia. IAP’s implementation includes systematic and regular peer review of legal and institutional framework for fighting corruption in the covered countries. See, OECD, Anti-Corruption Network for Eastern Europe and Central Asia, Istanbul Anti-Corruption Action Plan, Third Round of Monitoring, Georgia Monitoring Report. 3 • Component 2: Promote More Responsive, Professional, and Engaged Local Governments • Component 3: Create, Manage, and Maintain Centers for Civil Engagement throughout Georgia This evaluation will cover only the first two components. Component 1, Improve Governance and Transparency at the Federal Level, was designed to promote improved public administration at the federal level, and advance public sector reform in target institutions by using human and institutional capacity development approaches. Under this component, G3 has supported training of public officials and civil service reform as political will has allowed. This training was aimed at: 1) improving the capacity of government institutions to plan, implement and evaluate strategies and action plans; 2) developing skills and qualifications of civil servants; 3) enhancing communication between the central government and different stakeholders; and 4) improving citizen￾related data management systems. Within Component 1, G3 worked with several ministries and national government agencies, including: CSPA; GoARA; MoJ; Civil Registry Agency (CRA); Civil Service Bureau (CSB); Data Exchange Agency (DEA); the Ministry of Education and Science (MoES); the Ministry of Regional Development and Infrastructure (MRDI); State Audit Office of Georgia (SAOG); and more recently, with the PMO and the Ministry of Finance (MoF). Component 2, Promote More Responsive, Professional, and Engaged Local Governments, was designed to assist in increasing the responsiveness, professionalism, and engagement of local governments (LGs). Under this component, G3 has been working in 10 partner communities: all 4 cities outside Tbilisi with self-governing status (Batumi, Kutaisi, Poti, and Rustavi) and 6 municipalities (Akhaltsikhe, Gori, Marneuli, Ozurgeti, Tevali and Zugdidi). Until recently, G3 maintained 3 field offices (Batumi, Kutaisi and Telavi), each with a Regional Coordinator charged with overseeing the delivery of program support. The change in government following the October 2012 national election and the emerging political coalition created political waves throughout the country, affecting G3’s capacity both at the national and local level. The impact of the change of government at the national level (Component 1) was gradual and focused primarily in the increased role of the PMO in guiding the reforms. G3 became more engaged with the PMO in strengthening the new functions acquired under the new Constitution. Given its interest in initiating civil service reform, the PMO became a focal point for the G3 project in its work on the development of a concept for civil service and facilitation of the process to this reform. The change in government had a significant impact on Component 2, however. Elections disrupted local councils and resulted in the loss of continuity in G3 partners. Since the elections, all 10 of G3’s partner LGs have experienced at least one change in council chair and/or chief executive (mayor/gamgebeli) (see Annex 9). Kutaisi, being the extreme case, had one change in Sakrebulo chair and 6 changes of mayor. Typically, these leadership changes have also been accompanied by the replacement/shuffling of senior local staff. Given these circumstances, and the prospect of further sweeping changes in local leadership and staff following local elections in the spring of 2014, USAID and G3 curtailed most project activities over the summer of 2013, including closing all but one of the project’s 5 regional offices and terminating the staff’s regional coordinators (the Kutaisi office and coordinator being the lone exceptions). In effect, the major Component 2 activities anticipated in G3’s SOW were delivered in a concerted manner within only a 16-month period, from April 2012 through July 2013, and with much disruption and lack of continuity within this brief period. 4 3.0 EVALUATION METHODS AND LIMITATIONS The methodology for this evaluation was carefully designed to respond to the three main questions posed in the SOW (see Annex 1) and to determine whether G3 has accomplished its initial objectives. 3.1 EVALUATION METHODOLOGY The Evaluation Team collected quantitative and qualitative data from a broad range of stakeholders and beneficiaries to ensure independence of the evaluation process, as well as accuracy and completeness of the subsequent conclusions and recommendations. 3.1.1 Qualitative Data Collection The qualitative data collection began with a critical desk-top review of materials related to G3, including project quarterly and monthly reports, annual work plans, performance management plans (PMPs), project design documents, and communication among partners. The Evaluation Team also reviewed a wide range of project-related documentations, such as Georgia’s Open Government Program (OPG) and Action Plan, news articles for background information, and more. A full list of materials reviewed is found in Annex 4. The Evaluation Team conducted in-depth interviews with USAID and G3 staff, and key informant interviews with partners, beneficiaries, and stakeholders such as the State Chancellor; CSPA: MoJ; CSB; MoF; MRDI; representatives from United Nations Development Programme (UNDP) and German Society for International Cooperation (GIZ); GoARA and municipal officials; as well as with representatives from the organizations that provided consulting and training services to G3 either as subcontractors or grantees [Art Studio, Center for Training and Consultancy (CTC), Georgian Institute of Public Affairs (GIPA), Institute for Development of Freedom of Information (IDFI), International School of Economics at Tbilisi State University (ISET), Municipal Service Providers Association (MSPA), National Association of Local Authorities of Georgia (NALAG), and Policy and Management Consulting Group (PMC Research Group), among others]. In total, almost 100 individuals were interviewed between December 1 and December 20, 2013. In addition to interviews in Tbilisi, the Evaluation Team traveled to 4 of G3’s 10 partner communities (Rustavi and Telavi in the East; and Batumi and Kutaisi in the West) and, in each location, met with high elected officials, professional staff, civil society organizations (CSOs), media grantees, G3 regional coordinators (past and present), and Center for Civic Engagement (CCE) staff. Further, the Team conducted phone interviews with the LG staff who participated in public relations website development trainings. For a full list of key informants, see Annex 3. The Evaluation Team retained the services of IT, Research & Metadata Solutions (IRMS), a Georgian firm, to assist in implementing the mini-survey and focus groups discussions (FGDs). To expand upon the information gained through individual interviews, especially from the point of view of beneficiaries, IRMS conducted 6 FGDs in different locations – 3 in Tbilisi, 1in Batumi, and 2 in Kutaisi. Discussions focused on public engagement, transparency, and accountability at the national and local levels. Participants included CSOs, local journalists and students. Table 1: Dates, Location and Focus of FGDs Date Location Focus 12-18-2013 Tbilisi National level 12-19-2013 Batumi GoARA level 12-20-2013 Kutaisi Local Government, Public Engagement 12-20-2013 Kutaisi Local Government, Transparency and Accountability 12-21-2013 Tbilisi Local Government, Public Engagement 12-21-2013 Tbilisi Local Government, Transparency and Accountability Participants provided verbal consent at the start of each FGD. Discussions were then recorded, 5 transcribed, translated, and coded into NVIVO qualitative data analysis software. Questions asked during FGDs are provided in Annex 5. 3.1.2 Quantitative Data Collection IRMS, on behalf of the Evaluation Team, conducted one mini-survey of national- and local-level government employees who had participated in G3’s capacity development activities. G3 trained more than 2,000 civil servants since January 2013. Out of the 2,000 trainees, 152 participants - who were employed or had been employed by GoG, GoARA or LGs - were randomly selected to participate in the mini-survey. In total, 82 responses were collected from the above sample, representing a 54% response rate. All respondents had at least a Bachelor or Master’s degree and 2% had a Ph.D. Respondents were 55% female and 45% male. Among respondents, 33% worked or had worked at the national government, 19.5% at GoARA, and 47% at LGs (see Table 2). Table 2: Place of Employment at the Time of Training Level of Government When Trained Number Percent National Government 27 32.9 Government of Ajara 16 19.5 Local Government 39 47.6 Total 82 100 At the time of the survey, only 4 of the total number of respondents (5%) had left the government, and the remaining 78 had changed their positions. The survey focused on those trained in 2013 since it was believed that recall error would be too large for those trained in earlier years. Therefore, it is important to point out that this statistic is not representative of attrition of all those trained by the project. Respondents had spent an average of 5 years working in the government; on average, those in GoARA had worked in the government slightly longer than those in the central and LG (see Table 3). Table 3: Level of Government When Trained Level of Government When Trained Mean Median Minimum Maximum N Central Government 5.1 4.0 .5 15 27 Government of Ajara 6.6 7.0 1 17 16 Local Government 5.1 3.0 1 20 39 Total 5.4 4 1 20 81 The 82 respondents had collectively attended 143 trainings in a wide variety of areas. Women had attended more trainings than men, 63% vs. 37%. Strategic planning and governance were the most frequently attended trainings (see Table 4). Table 4: Type of Training Attended Skill Area No. Women No. Men Total No. Percent Percent of Cases 1. Public Relations 9 0 9 6.3 11.0 2. E-Communication Training 4 2 6 4.2 7.3 3. Governance 13 6 19 13.3 23.2 4. HR Management 4 9 13 9.1 15.9 5. Legal Framework of Georgia 1 0 1 0.7 1.2 6. Policy Development 3 1 4 2.8 4.9 7. Presentation and Writing Skills 5 2 7 4.9 8.5 8. Process of Policy Development 0 0 0 0 0 6 Skill Area No. Women No. Men Total No. Percent Percent of Cases 9. Program Budgeting 10 4 14 9.8 17.1 10. Project Management 3 4 7 4.9 8.5 11. Public Policy 6 4 10 7.0 12.2 12. State Procurement 3 6 9 6.3 11.0 13. Strategic Planning 11 10 21 14.7 25.6 14. Design and Management of Municipal Web Portals 4 0 4 2.8 4.9 15. Investor Relations 2 2 4 2.8 4.9 16. Managing IT Assessments, Strategies and Capacity Development Plans 1 0 1 0.7 1.2 17. Project Design, Management and Evaluation 0 2 2 1.4 2.4 18. Skills facilitation 0 0 0 0 0 19. Legal Drafting and Juridical Documentation 1 0 1 0.7 1.2 20. Generating Recommendations for Citizen Participation in Local Self-Government Workshops 1 1 2 1.4 2.4 21. Civil Service Reform Workshops 1 0 1 0.7 1.2 22. Strategic Urban Planning Workshops 4 0 4 2.8 4.9 23. Action Plan Drafting Workshops 2 0 2 1.4 2.4 24. Establishing a PR Workshops 2 0 2 1.4 2.4 25. Open Government Partnership Action Plan Workshops 0 0 0 0 0 Total 90 53 143 100 Over half of the GoARA and GoG respondents received training as well as coaching or facilitation as part of G3’s capacity building activities. Almost 54% of respondents who received coaching and 59% of those who participated in some type of facilitation were central government employees. The remaining 46% who received coaching and 41% that participated in facilitation worked at GoARA. Coaching and facilitation seem to not have been offered to officials at the local government level. Table 5: Other Capacity Building Received Other Capacity Building Received No. Women No. Men Total No. % of GoARA and GoG Respondents Coaching 17 9 26 60.5 Facilitation 13 9 22 51.2 Coaching was defined as frequent contact with an expert who supports the thinking or learning process for an individual or group. Facilitation was defined as discussion within a group, team, or department facilitated by an expert. 3.2 EVALUATION LIMITATIONS The Evaluation Team encountered a few limitations during its fieldwork in Georgia. Some of the most relevant limitations are listed below: Lack of Baseline Data. There was no baseline conducted at the start of the project and, therefore, no data existed against which progress could be measured. Halo Bias. The extent to which respondents were prepared to reveal their true opinions may have varied for some questions that called upon them to assess the performance of their colleagues or people on whom they depend upon for provision of services. Limited time allotted for the evaluation. The evaluation period included several holidays in the US as well as in Georgia. Preparation time was tight and the Evaluation Team received a considerable amount of documents while in route to Georgia. There were time constraints in evaluating such a large and complex project with diverse components, numerous activities, and several counterparts. Not all 7 G3 activities could be studied in-depth by this evaluation or all documents read and analyzed accordingly. Although the Evaluation Team met with at least 100 individuals, not all key informants could be interviewed either because of short time in Tbilisi and/or scheduling conflicts. Low turnout at Focus Groups Discussions (FGD). Most FGDs were conducted with fewer participants than anticipated as many that had been invited to participate confirmed attendance but did not show up. To mitigate the above limitations, ME&A used a variety of data collection methods to maximize the amount of information gathered, and applied triangulation to increase validity and reliability of results. 4.0 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS 4.1 QUESTION 1: IMPACT ON GOVERNMENT TRANSPARENCY AND ACCOUNTABILITY 4.1.1 Findings 4.1.1.1 Results Achieved Before and After 2012 Change in Government GoG has made significant strides in carrying out anti-corruption reforms since 2010, after adopting the Second Round Monitoring Report of the Istanbul Anti-Corruption Action Plan (IAP). The most salient initiatives of the Saakashvili administration included revamping the traffic police; reforming the licensing system by creating “one-stop-shop” and “silence-is-consent” principles for issuing licenses; creating a new tax code that streamlined tax and customs procedures; and introducing e￾Governance initiatives, particularly a fully electronic procurement system for both national and LGs. According to Transparency International (TI) Georgia, the GoG’s e-Procurement portal serves as a best practices case worldwide. When G3 started in February 2011, many of these anti-corruption initiatives were already underway. In fact, in 2011, Georgia joined the OGPI, committing itself to transparent and collaborative processes in public service, among other grand challenges. Yet, while overall effectiveness of governance had improved, the GoG placed little emphasis on institutionalizing public consultations, ensuring democratic and transparent decision-making processes, and on the accountability of governmental agencies. G3 was designed to promote improved public administration at the central and regional level and to support local governance decentralization. Work with the Central Government: G3 sought to improve public administration at the National Government and advance public sector reform in target institutions by using human and institutional capacity development approaches. Prior to the 2012 change of government, GoG showed little interest in providing venues for greater transparency or citizens’ participation, despite its reform agenda. Thus, G3’s main activities prior to the 2012 elections focused on building the human resources management capacity in several ministries and agencies, including MoJ, the Chamber of Commerce of Georgia (CCG), CSB, the National Agency for Public Registry ( NAPR), CRA, as well as GoARA. G3 facilitated revisions and updates to the strategic plans of MoJ, CRA, and CCG, as well as to GoARA ministries. G3 also assisted them in implementing internal tools for their M&E plans, as well as provided training and assistance in the development of procedure manuals that would serve to institutionalize them. Within the program context, G3 provided GoG with technical assistance to map Tbilisi within six months, matching approximately one million Georgians to their addresses. Key informant interviews (KIIs) indicated that this brought unexpected results as citizens became more aware of the need to participate in activities affecting them. G3 initiatives to improve transparency and accountability were designed to focus primarily on 8 citizens’ participation and improvement of civil servants professionalism and not on the areas of operation and oversight that are more central to transparency and accountability. Nor did they look at the relation between GoG oversight agencies [e.g. CSPA, the State Auditors’ Office (SAO), the Parliamentary Committee on Finance and Budget] and the G3 partner LGs. Thus, G3’s activities prior to the 2012 elections succeeded not necessarily in promoting transparency and accountability mechanisms but in establishing and strengthening administrative mechanisms and procedures in several GoG institutions. This included: providing strategic planning assistance to SAO; support to CSPA to enhance operational procedures related to its Dispute Resolution Board; media trainings on the SAO’s new mandate; workshop for SAO with GoG agencies; Trainer of Trainers (ToT) program for SAO on strengthening auditing capacity; technical assistance to DEA on data protection and security in compliance with ISO standards, 2012; workshops and trainings for GoG, media, students and CSOs on FOI; and development of FOI manual for GoG FOI officers. The MoJ was the main G3 counterpart prior to the 2012 elections. G3 focused on helping the Ministry meet its commitments in the OGP Action Plan (which MoJ was in charge of coordinating since 2011) by bringing officials from several ministries and CSOs to the table to discuss the first Action Plan. In support of the GOG’s Action Plan under the OGP initiative, G3 provided technical assistance and international experts, and prepared studies to improve government processes for the implementation of Freedom of Information Act (FOIA) standards. Workshops were offered to government officials and to civil society representatives. Participants interviewed by the Evaluation Team stated that these studies and workshops were very useful and helpful. G3 worked with the GoG and CSOs in defining a list of information that the GoG should proactively disclose as part of improving FOI, and trained more than 50 senior and middle managers from partner agencies to build their knowledge and capacity to respond to requests for information, in preparation for opportunities for increased transparency. Once Parliament adopted amendments to the General Administrative Code concerning proactive disclosure of information that introduced new terms, obligations and allocated resources for the purpose of widely and routinely publishing public-sector information, G3 shared public relations and communication strategies with all government agencies that had been developed for the MoJ, along with public relations manuals that had been developed working with the GoARA. G3 also helped introduce a performance appraisal system at the MoJ in 2011. However, the Ministry placed it on hold shortly thereafter but might have plans to restitute it in the near future. With the GoG showing greater political will for openness and citizen engagement after the October 2012 elections, G3 began to capitalize and expand on previous initiatives. Those interviewed by the Evaluation Team indicated that G3 rapidly promoted a CSO network on FOIA implementation, which brought together key organizations that had already been involved in pushing for greater FOI. G3 also moved forward and accelerated the OGP dialogue between the government and CSOs. It played a critical role in drafting the July 2013 decree on implementation of the OGP Action Plan. Following the elections of 2012, G3 became PMO’s key partner as it assisted this institution in designing more policy level initiatives. G3 has assisted the PMO in strengthening its coordination and public communications capacity. G3 has also supported the PMO in initiating civil service reform. By December 2013, G3 had provided assistance in drafting a comprehensive civil service reform concept paper and facilitated the processes related to this reform. This concept paper could serve as the basis for a fundamental reform of GoG’s civil service, as suggested by the PMO. Work with GoARA. G3 achieved consistent results in GoARA before and after the 2012 elections. Building on and strengthening activities started earlier under PAR, G3 worked in all four of GoARA ministries and practically all government agencies. G3 has conducted a comprehensive set of capacity building activities in GoARA, which entailed working on strategic planning, institutionalization of M&E tools, and design of a comprehensive e-communication system. Most trainings are standardized for all GoARA agencies and are followed by on-the-job coaching. One of the key results of this type of follow-up to training has been the capacity developed by GoARA to successfully prepare program budgets, as explained in other sections of the report. The changes of government in 2012, both in Georgia and in Ajara, brought into office a leadership with 9 greater political will to engage citizens. Since then, G3 continued providing GoARA capacity development in a number of key areas, including: M&E; development of key performance indicators; program budgeting; public communication; and much more. With the assistance of local and international consultants, G3 produced four policy papers on which new activities would be conducted. These papers included: Program Budget Methodology; Public Relations and Communications Manuals; e-Communication Systems and Manuals; and Human Resource Management Manuals. All policy documents and manuals developed through G3’s assistance at GoARA have been officially approved (through decrees) and have become obligatory for all GoARA institutions. According to G3’s coordinator in the region, strategic planning training provided by G3 h e l p e d change GoARA’s senior management views of and approaches to governing. This has contributed to more commitment on the part of senior leadership to implement reforms. GoARA’s small government allows for closer relations with heads of ministries. It also allocates more resources to transparency improvement, since most of its budget comes from tax revenues and not the central government. Work with Local Government (LG). Although G3 commenced in February 2011, letter agreements with the 10 partner LGs were not signed until November of that year through January 2012. Thus, activities launched at the local level prior to the October 2012 elections had limited time to show outcomes. A $350,000 grant to the National Association of Local Authorities of Georgia (NALAG) was executed early in 2012 and, pursuant to the grant’s SOW, work on Civic Advisory Councils (CACs) formation and Social and Economic Development Plans (SEDPs) commenced. According to KIIs and G3 reports, by fall 2012, CACs had been established in 8 partner LGs but were lagging in Zugdidi and Telavi.3 However, an internal assessment of CACs prepared under the aegis of the new CoP in September-October 2012, recounted struggles in launching CACs on a proper footing.4 G3 awarded several grants to local CSOs and regional media between April and July 2012, enabling them to initiate targeted activities prior to the October 2012 elections. G3’s 7th Quaterly Report and interviews conducted in four municipalities revealed that the grants helped some CACs to improve their functions. LG web portals in Rustavi and Kutaisi were upgraded with CSO grantee assistance, and media grantees began broadcasting promised programming on municipal issues. However, some activities were delayed due to the political situation and the election process. For example, there was no initiation of any real sustained work on planned Strategic Urban Plans (SUPs) and Service Improvement Plan (SIP) activities. As per SEDPs, although initiated, they remained in preparatory stages during 2012 and made no contribution to the budget process that year. Since the 2012 elections, G3 has successfully concluded the major activities it had contemplated at the local level in Work Plan Year 3. Moreover, through the above activities, G3 has produced a number of manuals, training guides, and other technical materials that could be of interest and benefit to other LGs and parties with an interest in strengthening LG reform. 4.1.1.2 Progress in Transparency and Effectiveness Related to e-Governance Assistance Georgia’s most significant success in e-Governance preceded G3. This is the e-Procurement system that publishes every procurement transaction and allows for questioning of government actions. No other system in Georgia compares to this level of e-Governance. In GoARA, G3 conducted a needs assessment, designed and supported the establishment of an interagency e-Communication system, provided the equipment, and trained staff on the new system. 3 G3 Sixth Quarterly Reports states: “During the previous Quarter, CACs were established in eight partner cities and municipalities. In the municipalities of Zugdidi and Telavi, however, the process of CAC formation was slow, owing to resistance from members of the municipal Councils, who feared that the new body would be immediately politicized as a result of the competitive electoral context in the run-up to the October parliamentary elections. During the present Quarter NALAG with the support of G3 management was able to reach agreements with the respective municipal Councils, both of which approved resolutions establishing CACs.” 4 Civic Advisory Councils—Challenges and Impediments, discussed at Oct. 5, 2012, meeting with G3 Regional Coordinators 10 The e-Governance system links GoARA’s four line ministries and all sub-agencies. According to GoARA’s Head of Media and Public Relations Department, internal communications have improved notably within government institutions. The Ministry of Health Care and Social Protection (MoH) and IT officials indicated that the new system is already simplifying and expediting processes by transmitting information electronically between agencies. Previously, this process was conducted manually, suffered from inefficiencies, and was vulnerable to negligence and delays. Nonetheless, the GoARA e-Communication system was not developed to provide e-services. Government officials in Tbilisi look at G3’s support to GoARA in implementing e-Governance as the model to follow. Most GoG ministries and entities in Tbilisi have multiple IT platforms that are often not compatible. For them, GoARA provides better internal communication and greater transparency. Urgency for e-Governance has increased as greater FOI has placed demands on GoG institutions to publish and provide information. The Evaluation Team did not find clear evidence that G3’s assistance to some institutions regarding e- Governance has resulted in progress in transparency and effectiveness. While there might be more information available on some ministries or agencies, more research would be required to analyze the level of progress. FGD participants, in particular, commented that web portals exist but are not user-friendly and often lack up-to-date information, especially in the regions. Typical comments include: “Kutaisi self-governance has its own web-page but the page contains very poor information.” (Local NGO, Baghdati) “Yes, there is such information (about budget, tenders, procurement) but they delay updating it” (Women’s Club CSO and journalist, Tbilisi) However, in some LGs, participants at FGDs expressed satisfaction with the new level of access to information: “Last year, I was called to discuss Marneuli budget. I wanted to see a review of the budget on the website but there was nothing there. I protested and after 2 months the local municipality created a site which is operating perfectly now. You can see any information you need there.” (Local NGO from Marneuli) “Municipality’s web site works very well in Gori. There is a lot of information.” (Charity Humanitarian Center Abkhazia, formerly at Evaluation Association). Still, not all information can go online. Interviewees in GoARA and other places told the Evaluation Team that creating high quality websites for sharing information is challenging since many documents are only hard copies5. More importantly, the Evaluation Team also found that, in general, citizens do not appear very attracted to e-Governance. The service is too new and not widely available; therefore, it may take some time for people to become familiar with seeking information electronically. 4.1.1.3 Transparency, Openness and Responsiveness as Result of FOI, OGP Action Initiatives In 2013, following changes in the government, G3 increased attention to the FOI and OGP initiatives. Working closely with both the GoG officials and CSOs, G3 hosted roundtables and conferences, commissioned papers, and provided international expertise on both initiatives. In addition, G3’s CoP rapidly reached out to the PMO and assisted in promoting decrees and monitoring mechanisms. G3’s CoP has already developed close and constructive relationships with the new leadership at PMO. The Evaluation Team found that government officials who deal with FOI and OGP, as well as CSO representatives who participated in roundtables and on-going dialogues, appreciate G3’s and its 5 GoARA’s citizen portal is currently being developed in full coordination with the GoARA senior management. Accordingly, the GoARA’s citizen’s portal will have functionality and capacity to publish all documents required as mandated by the recent decree on proactive disclosure of public information. 11 management’s support and assistance, and are particularly thankful for prompt responses to their needs. They also made reference to G3’s discreet style in facilitating the dialogue. One CSO representative who participated in a roundtable was not even aware that G3 was the facilitator. CSO network members told the Evaluation Team that G3 has helped in making OGP meetings on the Action Plan more effective. They claimed a noticeable contrast from processes prior to the elections, when meetings were poorly coordinated and “too informal.” By the same token, these meetings are helping define a real OGP Action Plan for 2014-2015, in contrast to the previous Action Plan, which lacked fresh ideas and civil society input. In recent meetings, particularly those held in late 2013, different agencies presented their priorities and explained progress achieved so far. The GoG’s decision to issue a decree on proactive disclosure of public information can be seen as a positive result of the OGP Action Plan dialogue facilitated by G3. While civil society participants view G3’s role in facilitating communication between GoG and CSOs as positive, they also expressed concern that GoG may be relying on G3 to promote OGPI and failing to assume full responsibility for carrying out the OGPI. Some expressed concern that in the absence of G3, the initiative may not be sustainable. Further, some CSOs believe that not all government ministries/agencies have fully grasped that OGP requires a-whole-of-government approach to governing and thus, level of commitment among different entities varies. CSOs would also like to see their role in the OGP and other initiatives expanded. They often feel that they have little room for input and would like to see future USAID programming be more assertive in making the OGP agenda broader and more inclusive. Academics and the private sector, for example, are not part of the CSO network for the OGP. Regarding the FOI, the Institute for Development of Freedom of Information (IDFI), a lead CSO in promoting FOI, explained to the Evaluation Team that its organization has systematically requested information from GoG over the past few years but has received practically no response. Under the new government, IDFI recorded a substantial improvement in the government’s response to its request for information. 6 This information is documented in the IDFI’s Access to Public Information in Georgia, Bulletin No. 5, 2012-2013, that IDFI handed to the Team. IDFI’s Director pointed to G3’s contributions in raising public awareness on the topic, coalescing CSO commitment around the topic, and providing input from international consultants on learning how FOI is handled in other parts of the world. In general, CSOs believe that G3, through its technical assistance, has helped GoG become more transparent. A participant at one of the FGDs conducted by the Evaluation Team commented: “In my opinion, transparency of information increased [since the G3 project started]” (Women’s Club CSO and journalist, Tbilisi). However, there are different degrees of efficiency and responsiveness between GoG, GoARA and LGs. Whereas in Ajara, for example, at least 2 FGD participants indicated that they never had any problems obtaining information from state institutions, others mentioned that it is often easier to do so, particularly on the budget, from the regions than from the central government and, that overall, it has been easier to get information after the October 2012 elections. FGD participants also explained that the 10-day deadline for providing information is not always met and that agencies may not have the information requested: “In many cases state agencies, which are responsible for various problems, do not have the databases, [they do not update the information about program beneficiaries], because in many cases the problem started many years ago. Now the situation is changing and needs to be re-explored. If state agencies updated information at least every year, there will not be any problem. In this case, the specific example is the problem of migrants; they use data which are 15 years old, while numbers have changed drastically.” (GoG/GoARA, Youth Scientist Union Intellect). 4.1.1.4 At the Local Level: Results of G3 Grants on Citizen Awareness and 6 See, http://www.idfi.ge/uploadedFiles/files/En-2013%20report%20opendata.pdf. The substantial increase in freedom of information was limited to response to information requested by IDFI from some ministries. It does not measure level of response to all requests. 12 Engagement Regarding Transparency and Accountability G3 grants to local CSOs sought to support citizen awareness and engagement regarding both LG “transparency and accountability” and obtain citizen input for various forms of LG planning and programming (see Annex 8, Grantee Major Activities). LG transparency and accountability are addressed here and under Question 3. Eight CSOs received grants, with the Georgian Evaluation Association carrying out activities in 3 partner LGs (Gori, Marneuli and Poti). In mid-2012, G3 awarded grants (ranging from $25,000 to $50,000 each) to 7 regional broadcast companies, with the recipients covering all G3 partner jurisdictions with the exception of Batumi, Gori, and Zugdidi. It is useful to analyze internet usage in Georgia, since important aspects of grant-supported activities required website access to advance transparency and accountability objectives. As Table 6 indicates, internet usage remains relatively limited outside Tbilisi. That said, in urban areas, internet usage (about 40% logging on at least weekly) has reached thresholds at which it has become an important and viable medium for informing and interacting with the public. However, in rural areas, a mere 4% of residents log on daily. Table 6: Internet Usage – Capital, Urban, Rural Every day At least once a Week At least once a month Less often Never DK/RA Capital 41% 10 3 7 39 1 Urban 28% 11 1 11 46 3 Rural 4% 6 3 6 65 15 Source: Caucasus Barometer 2011 (http://crrc.ge/oda/) Social media, such as Facebook and social forums, seems to be an important and effective mechanism used by civil society and NGOs to spread information about government activities. A FGD participant in Rustavi indicated that: “[Spreading information about civil servant car prices and frequency of buying new cars in Ajara] was due to the social media…employing social media, forums is very important. I’m a very active user of forums. They say that government uses them to review the thoughts and suggestions of people and then make decisions.” (Manager of CAI and Rustavi CAC representative, CIDA staff) Grants Impact on Municipal Website. CSO grantees contributed meaningfully to municipal website functionality in two LGs.7 Batumi staff did not participate in the PMCG website training and CSIC (the local CSO grantee) did not provide website development assistance.8 Partner municipal web-portals evidence some proactive disclosure of information, although limited in scope. G3 attempted to improve the sharing of information with the public primarily through CSO grants, several of which funded direct assistance to create or improve web-portals and Facebook webpages, as well as to prepare and disseminate budget calendars. The Evaluation Team reviewed all 10 G3 partner web-portals and conducted phone interviews with the lead staff that are responsible for website development and maintenance in 8 of the 10 (see Table 2 under Annex 9). The website review focused on its functionality, in general, and on the scope and quality of information proactively shared with the community through the municipal web-portal, in particular. Some observations resulting from the review include: • As a whole, the websites are not user friendly; a visitor will find them challenging to navigate to locate specific information. The Akhaltsikhe web-portal lacks a search engine altogether. Many sites contain duplicate windows with the same information, while others are empty. • Most documents are attached as Word, Power Point, or Excel files, which must be downloaded and are not searchable on the web-page. • None of the websites contain budget or event calendars. In most cases, event announcements, 7 The summer 2013 Policy and Management Consulting Group (PMCG) training included two modules related to the transparency and accountability objectives – one on website design and management, the other on public relations and communications (see Evaluation Question 3 for more details on trainee perceptions of the utility of this training). 8 CSI did link the website it created for the CAC to the council’s site. 13 if made at all, appear on the front pages as news feeds prior to meetings and not in a readily identifiable schedule. • All websites lack council minutes and other important meeting information. Three of the 10 websites post brief references devoid of information on issues discussed and decisions made. • Only 3 portals include budget execution reports of any kind. Poti’s website is the only one that had posted its independent audit report.9 • Just 3 partner LGs had posted their planned 2014 budgets at the time of the website review in the second half of December, 2013. • Four of the 10 partner LGs post information on up-coming tenders; none have information on tender awards. • Only 3 websites posted 2013 information on up-coming land auctions – though this could partly reflect the absence of any planned sale or lease activity. A few websites have information on auctions that had been conducted in prior years. • In most cases, municipalities update only the news feed on the front page describing activities and events attended by municipality officials – activities perceived as having PR value. • Only 2 websites have their adopted SEDPs posted. • Substantive information generally seems targeted more to visitors and, to some extent, investors – i.e., general descriptions of the community, economy, tourism, and entertainment attractions. Participants in FGDs confirmed the above findings. -“Kutaisi self-governance has it is own web-page, but the information there is very poor.” (Kutaisi TA, local NGO operating in Baghdati) -“Yes, there are some web-pages. But it is a fact that they do not work properly. They are just dead, compared with foreign web-sources.” (Kutaisi, ASA Association) -“We made information portals for regional self-governments. Some municipalities publish their orders and their web sites immediately. The main problem is that information is scattered. There is no database of information in government sector. Most documents are hardcopies. I think it would be better if they provide online databases and even online registry form for requesting information besides writing letters.” (GoG/GoARA, Youth Scientist Union Intellect) -“There is an issue determining the authenticity of online information. Government agencies will be able to correct data and change some figures after some time and it will appear that we would receive incorrect data… Maybe if information will be online after printing it, there should be some sign or seal that this information is authentic. That’s why we prefer hardcopies now.” (GoG/GoARA, Youth Scientist Union Intellect) Among G3’s10 partner LGs, Rustavi and Poti boast the most informative websites. Poti posts all adopted ordinances and other official documents. Rustavi’s informative website may reflect the support it received from G3 and the fact that the city adopted the previously referenced ordinance on Proactive Disclosure of Public Information. Kutaisi, Telavi and Batumi maintain relatively better websites than the rest. It should be noted that G3 contributed most directly through its CSO grants in Rustavi and Kutaisi. Media Grants. Media grants resulted in a substantial increase in quantity, rather than quality, of the coverage of municipal government affairs, with a limited long-term impact.10 Media grants succeed in intensifying the media coverage of municipal affairs. As shown in Annex 8, each grantee typically produced 3 types of programming in sizable quantities: 1) live chat and call-in shows; 2) educational programs on municipal governance issues; and 3) shorter reporting on daily municipal matters for interpolation into regular news programming. Additionally, 2 stations in Telavi and Kutaisi live streamed sakrebulo meetings on their websites. Although all supported programming was supposed to be routinely uploaded onto each station’s website, the Evaluation Team’s spot check of the Akhaltsikhe grantee found no G3-funded program on the station’s site. 9 Although not investigated by the Evaluation Team, it is understood that the statutory requirements for publishing these and other documents are generally disregarded, in part because they are lengthy documents not lending themselves to publication in the local press media. 10 Media grantees in Kutaisi, Telavi and Batumi indicated that the grants helped them just in increasing quantity of coverage and that they had most of the skills and expertise required for the task prior to the grants. 14 CSOs and journalists interviewed suggested that the production and content of the grant-supported programs were of reasonably good quality and that cooperation with local CSO grantees, with respect to suggesting topics, contributing speakers, and providing some content, was excellent. On the whole, participation of LG leaders was responsive, though the post-election political turmoil sometimes curtailed their availability and, not surprisingly, some officials were shy of exposing themselves to critical comments in a live chat-show format. The actual impact of the enhanced program coverage of municipal affairs on viewers during the grant period itself, and on the likely content of future broadcasting by grant-supported TV stations, is harder to assess. Based on the Evaluation Team’s interviews with members of the supported media stations and other informed local observers, the following observations can be made: • No hard data exists on regional TV viewership (share of regional TV markets overall and the numbers of viewers tuning into specific G3-supported programming). 11 By all accounts, regional broadcast media are watched by only a small percentage of the viewing households in their markets, although clearly some regional TV markets (e.g. Batumi and Kutaisi) are considerably larger than others. Among G3 grantees, Channel 25’s programming reportedly attracts a relatively high viewership for a regional TV station. • Absent a baseline and post-broadcast surveys of sample viewers, nothing can be said about the ways and the extent to which viewer understanding of municipal issues may have been affected by viewed broadcasts.12 • Several interviewed grantees believed that live chat shows generated the most animated viewer interest based on the volume of phone calls during broadcasts and post-show communication from viewers. No stations had monitored the hits on the programs posted on their websites, nor were they clear whether they even had the technical capability to disaggregate data of hits on particular posted content. Yet, Tanamgzavri TV station in Telavi reported to G3 an encouraging 60% increase in hits during the life of their grant compared to the prior period, but this could not be confirmed. • Importantly, interviewed media representatives all confirmed that, with two exceptions, broadcasting was not new either in form or in content but that G3 funding supported a substantial increase in the frequency of such broadcasting. The live streaming of council meetings was new in Telavi and had begun 6 years ago in Kutaisi, although it was terminated by the new LG in April 2013. Channel 25 in Batumi experimented with converting its call-in show to a live question and answer studio session, allowing its audience to pose questions to guest panelists on a face-to-face basis. • 9th Wave Broadcasting in Poti is notable for reporting the LG’s actions taken in response to problems raised during live call-in shows, including: repairing a damaged road, solving difficulties with canalization, and halting the building of a gas station after the population’s protest was reported. • Interviewed media representatives confided that, with the termination of their G3 grants, their stations would (for financial reasons) revert to their pre-grant level of LG coverage, likely meaning: some continuation of chat-shows and reporting; a reduction in educational programming (more expensive to produce and less attractive to advertisers); and termination of sakrebulo streaming. Channel 25 will revert to the standard call-in format, which requires fewer cameras and technicians to produce. • The representative from one station pointed to the increased sophistication of their personnel in municipal governance affairs as the main, truly sustainable impact of participating in the grant program. In the FGDs that the Evaluation Team conducted in different locations, there was some positive feedback about TV shows and progress in the media sector. -“I must say that it is important that the shows exist, but the most important is that they are easy to understand.…I can see that there is a progress in media sector.” (Kutaisi, ASA Association) -“[Interviewer: Are there any TV reporting about public affairs, or issues in connection with self￾government? What kind of impact has it on the involvement of citizens?] They are upgrading their knowledge and it is great!” (Kutaisi, The Mediators Association) 11 IREX staffer confirmed that such data is available in Georgia only from AGB Nielsen and only for nationwide stations. 12 The modest scale of the grant program itself would not have justified such survey work, even if it had been contemplated. 15 -“The interest about city council meetings is increasing… it is great!” (Kutaisi, ASA Association) However, at the local level, the debates or shows on the local TV stations were not viewed as very effective by three respondents because the focus was on TV stations that are not very popular; therefore, there was little interest even though the shows were of good quality. The suggestion from a CIDA staff member and Rustavi CAC representative was to focus on the national TV stations (Rustavi-2, Maestro, Imedi). -“G3 was focused on regional TVs which aren’t popular and are not viewed by a wide audience. Rustavi TV showed my interview everyday but no one has told me that they have watched it. When my interview was broadcasted at Rustavi-2, only 1 minute clip, half of the city saw it.” (Manager of CAI, Rustavi CAC representative, CIDA staff member) -“Yes, everybody watches popular TV stations. It is very important to have debates on TV. If there will be a discussion about the 2014 budget, everyone will watch it. This way, the civil society will know about the positive and negative aspects of the budget.” (Tbilisi, Women’s club CSO, journalist) -“Ozurgeti is not as active as Kutaisi. Therefore television shows everything. I cannot say who is watching. There is very low signal and also there is no interest in local people…” (Kutaisi, The Mediators Association in Gori) At the local level, the Civil Electronic Monitoring Systems (CEMS) represents perhaps the most innovative use of G3 CSO grant funding with good prospects for replication in other larger, more urbanized jurisdictions. The Civil Development Agency (CIDA), as part of its G3 grant, established CEMS in Rustavi and, under the extension of its grant through 2013, has been looking to replicate this initiative in Kutaisi, Poti and Gori. CEMS operates on a customized version of the FixMyStreet open￾source software platform introduced in the United Kingdom in 2007, and adapted since in over 15 countries internationally. The platform allows residents to report common problems such as potholes, unreliable trash collection, and defective street lighting to the appropriate local authority, as well as to monitor the responsiveness of those responsible for fixing the reported problems. Reports can include uploaded photos and their location is displayed in interactive map form. In the four LGs where CIDA brought the websites into operation, G3 reports fairly impressive utilization rates, as seen below: City Problems Reported Problems Solved Rustavi 181 50 Kutaisi 162 90 Poti 200 39 Gori 192 18 As of mid-December 2013, it was somewhat unclear how far CIDA had advanced in fully establishing CEMS in the other cities it has targeted. The link does not yet appear on the LG websites, and a few local journalists and CSO leaders with whom the team spoke by phone were unaware of the initiative.13 4.1.2 Conclusions 4.1.2.1 Results Achieved Before and After 2012 Change in Government • G3’s interventions followed on a continuum along with the GoG’s democratic transition process and in response to opportunities offered by changes in levels of political will. • G3 assistance prior to the 2012 elections helped strengthen central government institutions and to establish the groundwork for further support after the 2012 elections. • After the 2012 elections, the GOG was more receptive to initiative geared to improve transparency. In that context, G3 provided technical assistance and facilitation skills that helped to open a useful and helpful dialogue between the GoG and CSOs around the OGP Action Plan. Many government officials appreciate the support and timeliness of G3 response to their needs. 13 The Evaluation Team was informed by G3 staff that on December 16, 2013, CiDA conducted an official presentation of the CEMS activity in Tbilisi, which was attended by representatives of LSGs, CSOs and media from Rustavi, Kutaisi, Gori and Poti. The event was covered by major national TV channels. 16 • G3’s assistance to GoARA, prior and post elections, included training and on-the-job coaching and that sustained support has resulted in increased transparency and accountability. Of note is the GoARA preparing a program budget for 2014 that surpassed the Evaluation Team’s expectations. 4.1.2.2 Progress in Transparency and Effectiveness Related to e-Governance Assistance • G3’s assistance has contributed to improving internal communication within institutions and also to providing information to citizens on some GoG sites. • GoARA’s more effective e-Governance might be the result of having a smaller government and of G3’s assistance in conducting a needs assessment that helped provide more targeted support. • Increased use of e-Governance may depend on expansion of internet availability throughout the country and more people becoming familiar with the system. • Level of understanding and commitment of public officials to provide information to citizens may explain the disparity in the availability of information among institutions and regions. 4.1.2.3 Transparency, Openness and Responsiveness as Result of FOI, OGP Action Initiatives • G3’s management has facilitated communication within GoG and a dialogue between GoG and CSOs on FOI and OGP. G3’s assistance on OGP meetings on Action Plan has been very useful and helpful. Many government officials appreciate this support and are thankful for the prompt responses to their needs. • CSOs believe that this assistance has helped the GoG become more transparent, yet are concerned that donor assistance could reduce the required GoG willingness to assume ownership of the OGP process. They are also concerned with the level of understanding of the scope of OGP among government officials, as well as the lack of knowledge among citizens of the OGP initiatives’ impact on them. • CSOs input to the OGP Action Plan remains limited. • The GoG’s decision to issue a decree on proactive disclosure of public information can be seen as a positive result of the OGP Action Plan dialogue facilitated by G3. 4.1.2.4 At the Local Level: Results of G3 Grants on Citizen Awareness and Engagement Regarding Transparency and Accountability • The delay in implementing a number of G3 initiatives, particularly at the local level, resulted from the election process and turmoil created by some of the results of national and municipal elections. However, since 2013, the pace of G3 initiatives picked up, focusing extensively on OGP and FOI, as well as continuing support for GoARA and human resource development in several agencies. • G3’s CSO grantees contributed most to improving municipal web-based interfaces (web portals and Facebook pages) in Rustavi and Kutaisi, and to some lesser extent in Telavi and Akhaltsikhe. Several partner websites have features such as site design, functionality and content, which could be emulated elsewhere. However, although G3’s assistance has helped improve communication of information, more needs to be done. Electronic sites supported by G3 are not very sophisticated and often lack information that is useful and timely for citizens. • G3’s regional media grants did result in reasonable value that USAID received for its investment in terms of the number and hours of broadcasting dedicated to municipal affairs. This increased coverage translated into some enhanced awareness and understanding of important local governance matters in the short-term. In addition, participating media personnel gained more experience with municipal issues. However, the grants resulted in substantial increase in quantity rather than quality of the coverage on municipal government affairs. Therefore, the sustainable impact and overall cost-effectiveness of this approach is questionable. • Showing programs that discuss municipal issues in popular regional TV stations rather than local stations is a more effective way to reach a wider audience. • CEMS is a useful platform with good prospects for replication in other larger, more urbanized jurisdictions. • Social media seems to be a very important mechanism to spread information and raise awareness about government activities. 4.1.3 Recommendations 4.1.3.1 Results Achieved Before and After 2012 Change in Government 17 • USAID should take advantage of the political will and momentum to deepen reforms in areas where G3 has proven successful. • Consider providing structural anticorruption support by assisting GoG implement the recommendations of the IPA Monitoring Report, as well as those recommendations made by TI- Georgia to the GoG.14 • Replicate, to the extent possible, methodology used for capacity building in GoARA. • Continue building consensus between the government and the governed at every level. 4.1.3.2 Progress in Transparency and Effectiveness Related to e-Governance Assistance • Continue to encourage GoG and LG to improve dissemination of information and increase transparency. • Continue support to GoARA to institutionalize gains in e-Governance. Increase people’s awareness about the usefulness and benefits of e-Governance. E-Governance should concentrate on areas where service delivery may be improved, and ensure that it is user-friendly and effective. • Attention should be given to co-sponsoring e-Governance assistance with appropriate GoG and national NGO partners, and providing annual or biannual awards for the best national, regional, and local e-Governance initiatives in different categories – e.g. service delivery to citizens, business- friendly governance, and internal government operations. Such an award could include periodic documentation and assessment of best practice models. 4.1.3.3 Transparency, Openness and Responsiveness as Result of FOI, OGP Action Plan Initiatives • In future USAID programming, consider meeting with CSOs to hear their views on the OGP and role of international donors. Consider offering CSOs a broader and more expanded role. Include academics and private sector as part of the CSO network for the OGP. • Continue to encourage GoG and LG to improve dissemination of information and increase transparency, which is still limited. In respect to LGs, as being considered by TI Georgia, encourage GoG regulation that would specify a core set of proactive disclosure requirements for LGs, including a consolidated list of documents whose publication is already mandated by law. 4.1.3.4 At the Local Level: Results of G3 Grants on Citizen Awareness and Engagement Regarding Transparency and Accountability • During the close-out phase of G3, an effort should be made to document and share aspects of LG partners, web-portal function and content that could serve as models for other LGs. • Work with partners in providing electronic information that is more user friendly and timelier for citizens’ needs. • Future projects should establish a baseline to measure results. • Lessons learned from the G3 media grants, together with approaches tested through the IREX media programs, suggest that there are other viable options for including media coverage of LG affairs in future USAID local governance programming that might promise a more cost￾effective impact than just providing grant funding to regional TV outlets directly. According to IREX, these include hiring experienced journalists who can spend time working directly with the journalists; and not granting full ownership of the project to local journalists/station because they need good external consultants, among others. • Consider working with popular regional or national TV stations to increase the number of viewers of programs that cover municipal affairs. A real competitive process for grant funding might elicit a more interesting array of broadcast concepts. • Encourage more experimentation with interactive website features, such as CEMS and use of social media, especially in rural communities, and SMS messaging (e.g. subscription to SMS alert services of various kinds) to communicate with citizens and businesses. 14 TI-Georgia recommendations include: Addressing corruption risks arising from the movement of government officials between public and private sectors and the abuse of public office for the benefit of specific private companies; ensuring the transparent management of Georgia’s defense budget; and improving the administrative capabilities and oversight capacity of the Parliament, especially over the law enforcement and investigative agencies. See: Georgia in TI’s 2013 Corruption Perception Index Civil Georgia (Tbilisi, 3 December 2013) http://www.civil.ge/eng/article.php?id=26755. 18 4.2 QUESTION 2: EFFECTIVENESS OF GOVERNMENT MECHANISMS FOR PUBLIC ENGAGEMENT 4.2.1 Findings 4.2.1.1 Degree of Success in Building and Sustaining Effective Mechanisms for Public Engagement at the National Level One of the main objectives of G3 has been the promotion of governance and transparency at the national level through the support of government and non-government mechanisms for oversight of the Executive Branch. Mechanisms included hearings, open meetings, websites, report cards, and citizen review boards. G3 also promoted events, such as workshops, conferences, round-tables, public forums and similar activities that contributed to improving public engagement. A key element of G3 efforts to encourage Executive Branch institutions to engage citizens included assisting in promoting modern Public Relations (PR) and Communications mechanisms as a means to effectively communicate with stakeholders and the public. This first step helped in institutionalizing the use of interactive mechanisms, such as town hall meetings and other forms of outreach. This approach was practically piloted at the MoJ, and it was started by reviewing existing mechanisms at the ministry to identify ways to improve them. G3 also designed and implemented a capacity development program for MoJ PR personnel to sensitize them to the public relations from the public’s perspective, which included developing a PR/Communication Policy Framework and an action plan. The Plan sought to encourage the ministry to provide public access to planned activities; increase legal awareness of the public; facilitate public involvement in the decision-making process and improve the PR/communication processes to enhance outreach. This was the model to follow by other institutions. At the CSPA, G3 provided technical assistance for strengthening the communication process of the Dispute Resolution Board (DRB). With the support of PMC Research, G3 proposed a comprehensive procedure manual that included user registration, procurement plans and reports and rules of E-tender and simplified E-tender process. As a result of G3’s assistance, the DRB’s web-page uses social media for the purpose of disseminating information and obtaining public feedback regarding its operations. In addition, G3’s implementing partner, GEPRA, worked with several partner institutions, including PSDA, NAPR, CSB, SAO, RS, MoES and others to improve their communication and outreach capacity. The approach was multi-facetted and included individual interviews with communications staff members, study of existing PR practices in each of the entities, and a report describing the existing approaches to PR, rules and regulations for implementing PR activities and management, coordination and impact assessment. The Evaluation Team could not reach KIs from all these institutions to confirm if the initiatives are being fully implemented. G3 also assisted the GoG on meeting the OGP Action Plan commitments as it related to civic engagement. This included training more than 50 senior and middle managers of GoG partner agencies in how to disclose public information to better inform citizens and civil society. Several civil society representatives who had attended some public events felt that some of the activities do not yet represent two-way communication but rather opportunities to inform citizens and not exactly to consult with them or invite participation. G3 also launched an NGO network on FOIA implementation to actively contribute to the OGP by promoting proactive disclosure of public information. Through a series of workshops, G3 helped CSOs develop a series of recommendations on changes to the OGP Action Plan, which were delivered to the PMO. G3 assisted several ministries develop websites that would provide information to citizens. Success on these was not always even. Information on some government websites continues to be incomplete, and often out of date. In GoARA, G3 assisted the Media and Public Relations Department to develop an annual public outreach plan as well as a Public Relations/Communication Policy Document and manual. The policy and manual have been officially approved by GoARA and have become obligatory for each GoARA 19 agency. G3 also trained the staff to organize public activities. This included providing public relations specialists to train personnel on speech-writing as well as to develop the capacity to hold town hall meetings with local citizens. As a result, internal communication has improved notably, although the staff seems to yet not be fully comfortable with the full concept of public relations and continues to treat communication as a one-way street. Yet, GoARA through G3 assistance conducted three phases of citizen engagement during the 2014 budget preparation. Stakeholder discussions have been conducted by each GoARA agency in May and September to discuss draft versions of GoARA Priorities Document and 2014-2017 programs and sub-programs, respectively. Comments and recommendations from stakeholders have been considered during the preparation of the final draft budget. In addition, GoARA’s Program Budgeting Manual includes citizen engagement mechanisms that have been approved by GoARA. Furthermore, GoARA public relations and media communication department systematically oversees the implementation of PR policy and coordinates GoARA agencies’ PR activities and communication with citizens. During the previous government, G3 also assisted GoARA in establishing the Regional Development Council to oversee the implementation of the Regional Development Strategy. The Council includes more than 30 members from government, academia, business, civil society and municipal governments. Later, G3 assisted the new government to establish a Priorities Committee that consists of GoARA senior managers, local self-governing representatives, as well as CSOs and business representatives. The Priorities Committee is based on G3 recommendations and serves to discuss GoARA’s agencies priorities, programs, sub-programs, monitoring results and more. During the Evaluation Team’s discussions with focus groups, almost all respondents felt that obtaining public information is generally not an issue; however, receiving it often takes an extra effort. Some of the key challenges related to information request include: (i) it is based on the political will of the person from whom you request the data; some share information and others do not; (ii) the deadline of 10 days is sometimes not long enough to gather the information, and information is often delivered way beyond this deadline; (iii) attempts are made to not provide the full information requested; (iv) answers are sometimes inaccurate; (v) information is sometimes not released because it is deemed personal and not to be shared publically; and (vi) statistical information is the most difficult to obtain because it is often viewed as sensitive. -“Really, I never had any kind of problem. Especially after the government changed… They help us... at least they do not prevent us.” (Kutaisi, The Mediators Association in Gori) -“I also agree, there is no problem of getting information.” (GoG/GoARA, Youth Scientist Union Intellect) -“We got the information because of our personal relationships. Receiving information also depends on who is the head of the local government.” (Rustavi CAC Representative) -“There is a political will and we are getting the requested information from the Ministry of Defense; however, we can’t get it from the Ministry of Internal Affairs- because of the lack of will.” (IDFI) -“One main problem is deadlines which they cannot meet. Deadline of receiving information is ten days… This problem is everywhere in Georgia not only in Ajara.” (GoARA, Transparency International Georgia) -“It is a problem to identify the dividing line between personal and public information... Since 2011, the main problem is the personal data rule, which is not clarified, and is used to avoid sending us requested information.” (GoG/GoARA, Youth Scientist Union Intellect) -“Everything is ok, if you do not ask some kind of sensitive information [statistics]… Do you know why the Ministry of Internal Affairs isn’t sharing the crime statistics, at a detailed level? They don’t want to share, so they are not making it public; this is usually the case, when governmental bodies don’t want to share the information.” (IDFI) 20 In GoARA, DPA (a Latvian IT company), under contract with G3, is in charge of conducting a feasibility study for the Civic Engagement Portal and its implementation. According to GoARA’s Head of IT Department and the Head of IT Department of the MoH, the portal is still a work in progress. Despite initiatives in GoARA towards citizens’ engagement, focus group respondents mentioned a lack of accountability of the GoARA towards its citizens. “I think there isn’t a good connection between the public and government. The government is not public oriented… I think the public is not aware that they are part of a state and government.” (Transparency International Georgia representative based in Ajara) However, some focus group respondents blamed the lack of public engagement with citizens on the apathy of citizens instead of lack of initiative from the government. They mentioned that considerable education in civic engagement and rights is needed. -“Information, education. People need to know that government is elected by them. We need to integrate international practices…That’s why I think that civic education is very important. There are several ways to address civic issues, such as petitions, peaceful demonstrations etc. But people should know that they have the rights to do that, they should be aware of them.” (Manager of CAI, CAC Rustavi, CIDA staff member) -“Well it reminds me of the communist regime. It is difficult for the government to work properly when civil society does not exist…There is no will in Georgia. Potential exists, but there is no will. There have already been several studies about this.” (Local NGO, Marneuli) -“I think that the attendance of the public discussion is high from the non-governmental sector, but as to the civil society it is very low. Citizens don’t care and don’t want to come, listen and participate at all.” (Charity Humanitarian Centre Afkhazia, formerly at Evaluation Association) 4.2.1.2 Degree of Success in Building and Sustaining Effective Mechanisms for Public Engagement at the Local Level In speaking with CSO and media representatives, the Evaluation Team frequently encountered skepticism about the ability of donor programs (no matter how well conceived and executed) to foster meaningful and enthusiastic citizen engagement in the absence of more genuine local self￾governance. Governance has centralized over the last decade, with LGs having increasingly limited fiscal and functional autonomy over service delivery.15 Thus several key informants argued that further reform of intergovernmental relations and civil service is a precondition for fully realizing G3’s aspirations to truly energize citizen engagement at the local level. One LG focus group said there were no mechanisms in place for public engagement. Any public engagement happens informally: individuals from civil society go to LG offices and try to “catch” someone to speak with about their problem, or get a “permit” to meet someone from the government. However, it seems that LG does engage with those who create a nuisance for them (demonstrations near city council for example). -“Yes, sometimes there are some meetings/demonstrations near city council and that’s why officials are talking to them and listening to their problems.” (Kutaisi, TV/Radio company) -“They are not hiding information consciously. There just isn’t a mechanism of spreading information and they are not working on it.” (Kutaisi, Association of Young Economists) At the local level, G3 supported four major mechanisms for citizen engagement: • CACs, as already assessed to some extent under Evaluation Question 1, in respect to their contribution to transparency and accountability. • Three forms of participatory planning – SEDPs, Strategic Urban Plans (SUPs), and SIPs – which G3 only extended through the plan preparation process, not into implementation and performance 15 Most municipalities receive only 5% to 20% of their revenues from their own sources; LGs perform quite limited service functions, with pre-schools/kindergarten, local roads, markets, sports and recreation, and solid waste collection accounting for the majority share of LG expenditures; Solid waste disposal (i.e. land fill operation) was centralized at the beginning of 2012. 21 monitoring. Community/Civic Advisory Council. Early in G3’s implementation, project management decided to promote CACs as a major vehicle for implementing SOW activities under Component 2, Task 1 (Municipal Council and Executive Body Functioning) and Task 2 (Civil Society Participation in Policy Development) with respect to LG dialogue with CSOs and promoting various forms of citizen engagement and information sharing. The SOW itself makes no direct reference to CACs. G3 envisaged CACs to be public forums for improving two-way communication between a LG and its citizenry, particularly with local CSO leadership. During G3’s first year, NALAG took the lead in “breaking the ice” to promote buy-in to the CAC concept among G3’s lead LG counterparts, and to introduce model procedure rules. Towards mid-2012, local CSO grantees took over the CAC activity from NALAG to better constitute CACs and to inform and support their functioning. The G3 letter agreement with LGs required CAC formation as a condition of participation. In retrospect, it seems that, with a few exceptions, local leaders have not genuinely embraced the concept. Partner LGs were given wide discretion in composing these bodies in respect to size, membership, and functioning. • Gori’s CAC stands out for the openness of how it was formed, with the sakrebulo chair extending an open invitation for citizen representatives to volunteer as members. • In Gori and Poti local journalists asserted that only LG officials and non-commercial legal entities (NCLAs), created and funded by local self-governments (LSGs), attend CAC meetings – which are too formal and of little interest to independent CSOs and the public at large. • Kutaisi’s CAC proved the most problematic with the Chair appointing 9 CAC members, 5 of whom were city employees and the rest prominent individuals with no active interest in municipal affairs. The Association of Young Economists of Georgia (AYEG), the G3 grantee, was not included. The CAC met only about 3 times and operation lapsed. Moreover, a key informant in Kutaisi felt there were no mechanisms in place at all for public engagement. “If someone needs something from the local government they just go to the building, catch the first member of government and try to talk about the problem. Is it a mechanism? And also – citizen can go, get a permit and then meet somebody from government…as usual there are problems with permits and of course, there are problems with “catching” the officials. Members of the council have an obligation to report to the people once per quarter…but there is no mechanism specified in the law. So, officials meet relatives, or people who are a nuisance.” (Kutaisi, Association of Young Economists) • In Rustavi, all but 4 original CAC members were drawn from the mayor’s administration. However, the CAC is run by the Deputy Council Chair and has been one of the most active CACs in terms of frequency of meetings and regularity in meeting at housing complexes with interested citizens. That said, some local CSOs remain skeptical of the city’s interest in genuine two-way engagement and transparency, and believe that the CAC is used as a public relations vehicle and means to promulgate official messages. According to focus group respondents, LG representatives in the Rustavi CAC are actively involved in the CAC due to positive personal relationships. “Personal attitudes play an important part… Non formal relationships are very important in our reality. So the involvement levels from LG are pretty good.” (Manager of CAI project and CAC Rustavi representative) • In June 2012, the Telavi CAC Chair, rather than recruiting CSO leaders with an active interest in municipal affairs, chose to invite a small number of prominent citizens (head of the theatre, director of the university) to compose the CAC. He confided to the Evaluation Team that he attended an initial meeting or two and then lost interest. The CAC has been relatively inactive since February 2013. • Batumi’s CAC is an outlier in terms of its structure, ability to engage effectively with a succession of city leaders, and of the consequent relative vitality of its operation. The CAC is composed of 19 members, including: 2 selected by local CSOs, 2 journalists selected by local media representatives, 2 selected by the Chamber of Commerce, and the remainder comprised of the 22 heads of public halls (darbazys) formed in each of the city’s 13 territories. The CAC has operated effectively to promote citizen engagement in the annual budget process and as a forum for bringing citizen concerns to the city council’s attention. In this latter respect, it has succeeded in getting the city to change several policies regarding citizen and small business concerns, notably the street cleaning and waste collection fees charged by legal entities. Participatory Planning Mechanisms. Each of the three G3-supported participatory planning mechanisms (SEDPs, SUPs, SIPs) have some merit, but none have clearly demonstrated effectiveness and sustainability in the current Georgian context. An important common denominator to these initiatives is that they have some sanction in the legal framework in Georgia; however, none is mandatory. Social and Economic Development Plan. SEDPs were initiated in Georgia in 2009 with the Council of Europe, UNDP, and other donors piloting them in cities such as Rustavi, Kutaisi, Ozurgeti, and Poti. G3’s approach to the SEDP process in Georgia focused on providing technical resources that adapted EU models to the Georgian context, and in a number of LGs completing a participatory process of SWOT analysis, working groups, and public participation in the final work product, based on the model. Despite G3’s and other donors’ efforts, the Evaluation Team’s interviews with LG officials and other SEDP participants revealed that the SEDP concept faces constraints in becoming of proven value and acceptance as a part of LG operations. Most importantly, at present, SEDPs are not integrated with legally mandated planning and budgeting processes. As such, LG officials and senior management, conscious of their limited human resource responsibilities, tend to regard these strategic planning exercises as extra work beyond their mandated responsibilities. A number of them also expressed skepticism about the value-added of the effort required for meaningful citizen engagement and the potential for eliciting strategic input and overall community-wide priorities from citizens. NALAG, pursuant to its grant from G3, took the lead in developing SEDPs in all G3 partner municipalities, starting in the spring of 2013. NALAG updated and refined the SEDP Guidebook it had prepared with the Council of Europe, and provided two trainings - one in participatory planning and one in SWOT analysis - to about 300 local counterparts. By the second quarter of 2012, all partner LGs had completed and adopted SEDPs for the 2014-2017 timeframe. In Batumi, the City’s cooperation with NALAG broke down and the envisaged enhancement of the draft plan, both in terms of technical content and citizen input, never moved forward. Based on discussions with both City staff and other local parties, this resulted from Batumi’s disappointment with the technical expertise it received in upgrading their pre-existing document (particularly, in better defining and costing priority projects). Overall, NALAG’s budget provided for TA in the participatory process, not the provision of specialized expertise in project finance and statistical analysis. The outcomes of the completed SEDP activities vary substantially in terms of the quality of the technical planning documents themselves, and in the character of the citizen involvement. Based on the Evaluation Team’s interviews and document review, Batumi and Kutaisi produced the best documents from a technical perspective, e.g. grounding of the strategy in data collection and analysis, and the costing out of priorities. By contrast, the Gori, Marneuli, and Telavi documents seem incomplete: all are missing basic tables and charts, and Telavi has not included a budget annex at all. From the perspective of several neutral parties interviewed, Gori and Poti may have had the best participatory processes. In virtually all the partner LGs, G3 facilitators supported basic SWOT analysis exercises with citizens and the business community participating and vetting the final draft document in a well-advertised public forum before adoption. In the best cases, sector working groups combined public participants with LG technical staff, in fleshing out priorities and project possibilities. From its interviews, the Evaluation Team found no convincing evidence that SEDPs were used during 2013 as an important resource in preparing 2014 budgets and little likelihood that LGs have the motivation to maintain and update them. One reason for this is that by law, all LGs, as part of the annual process, must produce Basic Data and Directions (BDD) document (Medium Term Expenditure Framework), which is quite voluminous. The main difference between SEDP and BDD is that SEDP has a general description of cities’ features: brief historical overview, socio-economic profile, and SWOT analysis. Moreover, as pointed out by Kutaisi and Batumi counterparts, the SEDPs do not define action plans with projects defined and costed out in sufficient detail to be useful for annual planning purposes. 23 These counterparts would have liked more technical consulting services in achieving this level of specificity. Although, upon examination, a number of SEDPs are generally consistent with annual budgets, it is more in terms of the broad expenditure classifications and most likely reliant on 2012 BDDs to prepare SEDP budget annexes. Some officials did cite the value of SEDPs in speaking with donors and the government about possibilities for project funding - particularly in being able to demonstrate that a given project has been identified as a priority through a participatory, strategic planning process. One reason Kutaisi cut short the public vetting of SEDP priorities was to adopt the plan in time to meet a March MRDI deadline in connection with allocating 2013 Regional Development Fund monies. Strategic Urban Plans (SUPs). SUPs are conceived as a way to ground the legally mandated Master Plans (spatial plans and land use/zoning type ordinances) in a more strategic framework developed by participatory means. SUPs were completed in three self-governing cities (Poti, Rustavi, and Batumi) and initiated but halted in Kutaisi due to the political turmoil there. This has been followed by larger subcontracts to prepare complete Master Plans in Poti and Rustavi. Batumi’s SUP was completed too late (it is yet to be adopted officially) to receive this follow-on assistance. (Overall, support to Batumi was suspended in late 2012, in part due to poor cooperation on this activity; support was only resumed with the arrival of new city leadership in the first quarter of 2013). The quality of civic engagement seems reasonably high; for example, in July alone CSI helped conduct 4 workshops, bringing together representatives from all 13 districts; Rustavi reported over 500 attendees at the final presentation of the plans. Counterparts believe the G3-funded technical support was of high quality. In particular, the follow-on financing of the Master Planning is greatly valued, since the cost of the needed professional services is prohibitively expensive. As a result, virtually no LGs in Georgia have current master plans, even though they are the prerequisite for issuing construction permits. Service Improvement Plans (SIPs). In March of 2012, G3 awarded a contract to the Georgian Municipal Service Providers Association (MSPA) to prepare SIPs on a participatory basis, initially in the 5 eastern Georgia municipalities, followed by plans for those in the west. About a decade ago, USAID supported a very similar initiative as part of its Georgia LGRI project. SIP type plans were prepared in 5 LGs for street cleaning, water supply, juvenile crime, and solid waste collection. Under this earlier model, there was emphasis on zeroing in on 1or 2 desired improvements, and defining related performance measures with citizens. Additionally, the model involved citizens, especially youth, in monitoring the results before surveying citizens to confirm whether an increased satisfaction with the service was achieved. Of perhaps all major initiatives, the SIP activity suffered the most from the pre- and post-election turmoil. MSPA, in its August 2012 Narrative Report, notes that “in June, the Akhaktsikhe, Telavi, and Marneuli municipal leaders suggested that the MSPA consultants develop the analysis without the involvement of the working group.” In Gori, despite interest on the part of the sakrebulo, the municipal executive office also declined to participate. The end result was that, with the exception of Rustavi, the SIP documents were prepared with little, if any, engagement of the LG with citizens. Given the difficulties encountered in engaging local counterparts effectively, USAID and G3 terminated the SIP initiative early, in August, without proceeding with planned replication in the 5 western, G3 partner LGs. Almost everyone interviewed rated MSPA highly in respect to the quality of the technical expertise and materials provided. Two other envisaged citizen engagement initiatives did not come to fruition. • As reflected in letter agreements, MSPA was to provide partner LGs with chancellery software for chancellery purposes (managing correspondence, etc.) in connection with one-stop shop back room operation. However, sometime in 2012, GoG gave all LGs a January 2013 deadline to select and pay for the use of 1of 2 software systems already available for this purpose through servers at the MoF and MoJ. Since continued G3 support would have thus involved paying one government entity to provide services to another, this activity was dropped from the MSPA grant. • G3 hoped to demonstrate the use of citizen petitions, as authorized in the LSG Law, to initiate a grass-roots policy change. Through this means, the Center for Strategic Research and Development of Georgia (CSRDG) in Telavi looked to secure funding for a new kindergarten 24 through a petition reflecting village needs. In the end, CSRDG concluded that the petition process was not necessary, and the desired result was accomplished by other, more expeditious means. 4.2.1.3 Municipal Level Partner Local Governments Institutionalization of Mechanisms of Public Engagement Important G3 support for several public engagement mechanisms has already been assessed above - for CACs in general and municipal web-portals (Q.1), and for participatory planning (Q.2). Here the focus is public engagement in the planning process through public hearings and other means, the most important of the other ways in which G3 looked to strengthen public participation in municipal affairs. Most LGs have yet to institutionalize public consultation in the budget process and priority setting at the most critical junctures in the annual budget calendar. Here it is important to recognize that, per Article 78 of the Budget Code, all local councils must convene a “public discussion” of the draft annual budget (in effect, a public hearing), prior to voting on its adoption. Although some LGs disregard this mandate, most have at least a perfunctory hearing of some kind. Thus, it is the quality of these public discussions that needs improved engagement. Perhaps, more importantly, it is imperative to secure public input much earlier in the budget adoption cycle, with ongoing parameter iterations between the GoG and LGs over the ensuing months. Most of the G3 CSOs and media grantees managed to work with local authorities to secure better￾attended public discussions in adopting the annual budget, reportedly relatively more interactive than previously and with good media coverage. The Evaluation Team found out that no one had prepared “budgets-in-brief” for these public discussions – i.e. a newsletter type that distills the voluminous official budget down to an easily understandable page or two, with simple graphics that highlight “where the money comes from” and “where-it-goes.” Moreover, by the time of these budget hearings, the budget itself is essentially final, with little scope for meaningful revision. The City of Batumi, in concert with the CAC and CSI (the G3 CSO grantee) has followed a process that could serve as a model for other LGs in Georgia. As early as March, meetings were held with each of the 13 territorial darbazies. Subsequently, city department heads for service sectors meet a number of times with elected heads of each darbazy, and, come August, the citizen input, consolidated upward from the territorial darbazy through the CAC, is merged into the budget draft.16 CSRDG in Telavi reports that in spring 2013 they managed to organize budget discussion, including field visits in villages with council members. At present, in so far as the Evaluation Team could determine through their interviews, no G3 partner LGs have sought citizen engagement into the LGs’ access to the Regional Development Fund (about 225 million GEL in 2013) and Village Improvement Fund (about 50 million), both administered through MRDI. Funding decisions are typically made in the first quarter of the year, separately from the local budget process. In 2013, MRDI announced that, in addition to proposing projects in 4 targeted categories (roads, bridges, and shore protection), LGs could also apply for other project purposes identified in an adopted SEDP. Although hard to judge definitively, it seems unlikely that G3 contributed meaningfully to institutionalizing the other types of mechanisms cited in this research question. Several of the CSO grantees (e.g. AYEG, GEA) prepared, printed and distributed budget calendars, but the Evaluation Team is not aware of any LGs taking this over – and, post G3, these CSOs will revert to less costly postings on their websites. The newest government in Kutaisi has, in fact, cut back on the frequency of TV and radio announcements of upcoming council meetings. The new Batumi administration, with the CAC, has been convening frequent public meetings in the territories on topics such as street cleaning, but it is too soon to conclude that this spurt in public consultations will be sustained and institutionalized. Orzugeti temporarily increased meetings in villages. Telavi officials had some village meetings in 2012 through April 2013 but these were not continued thereafter. As shown in Annex 8, 2 of the 6 trainees who attended trainings on Public Relations and 16 Somewhat ironically, Kutaisi (where cooperation with the city G3 administration broke down almost altogether) from 2009 to 2011 maintained a quite similar consultative process under the banner “Budget With People”. 25 Communications reported some improvement in press releases and systematic posting of information on webpages, bulletin boards, etc. This was more in terms of increased personal conscientiousness than institutionalizing a new or improved mechanism. 4.2.2 Conclusions 4.2.2.1 Degree of Success in Building and Sustaining Effective Mechanisms for Public Engagement at the National Level (GoARA, CSPA, MOJ) • Perhaps it is too early for the initiatives and mechanisms supported by G3 for public engagement at the national level, to become institutionalized. Although mechanisms have been created and/or are in the process of being established, there is not enough evidence of their sustainability and there is no baseline to measure it. As in most other initiatives, much will depend on the political will of the leadership to formalize mechanisms and reach out to stakeholders. • Many citizens avoid public participation and engagement, especially at the national level, because they do not see the value of it and are not aware of their rights. 4.2.2.2 Degree of Success in Building and Sustaining Effective Mechanisms for Public Engagement at the Local Level • KIs interviewed believe that without a more genuine local self-governance, no donor program will be able to promote citizen engagement at the local level. • It remains unclear whether the CAC represents a viable vehicle for institutionalizing citizen engagement at the local level. With few exceptions, local leaders have not genuinely embraced the concept. The Batumi model of an umbrella CAC (grounded in territorial darbazys and with additional representation from media, businesses, and CSOs) looks to be the most promising CAC initiative on which to build. According to FGDs, CACs with a majority of representatives from LG do not function well. -“Speaking about Poti and Kutaisi CACs, all members are from LGs and main issue is that they don’t function at all. Gori is in a better situation, but in this case all members were from NGOs…this was our recommendation for the governor, to get more NGO involvement…There will be elections, we shall see the changes…” (Manager of CAI project and CAC Rustavi representative) • SEDP type strategic planning at the local level remains an important objective. However, in order for SEDPs to become a valuable and acceptable tool, they must define action plans with projects defined and costed out in sufficient detail to be useful for annual planning purposes. Counterparts indicated that they would have liked more technical consulting services in achieving this level of specificity. • G3’s support (still in progress) for SUP exercises promises to demonstrate the value of this type of participatory planning for preparing the legally mandated, Spatial and Land Use Master Plans • Given the political turmoil in partner LGs, it is hard to assess the merits of the SIP concept as implemented by MSPA. The SIP would seem to have some unrealized potential in demonstrating the inherent value of integrating performance management concepts into the delivery of core municipal services. • Unexploited opportunities exist for soliciting citizen input to the annual application process for LSGs accessing the special funds available from MRDI. Doing so might elicit more animated engagement than is achievable through comprehensive planning exercises such as SEDPs and SUPs, since the prospect for the funding is real and decision-making can focus on specific, very tangible project possibilities of concrete benefit to specific villages, neighborhoods, or small businesses. 4.2.2.3 Municipal Level Partner Local Governments Institutionalization of Mechanisms of Public Engagement • Citizen participation in budget adoption in most LGs has sometimes focused too narrowly on end￾of-the year hearings, rather than on more strategic points in the annual budget calendar. Some precedents for officials from G3 LGs getting out to neighborhoods and villages early in the budget cycle can be confirmed (most notably in Batumi and Rustavi), but one cannot yet describe such consultations as an institutionalized mechanism. They do, however, provide positive models for other LGs to emulate. • The City of Batumi has followed a process that could serve as a model for other LGs in Georgia by holding meetings with each of the 13 territorial darbazies as early as spring 2013. 26 4.2.3 Recommendations 4.2.3.1 Degree of Success in Building and Sustaining Effective Mechanisms for Public Engagement at the National Level (GoARA, CSPA, MOJ) • Establish baselines that will assist in measuring success of public engagement mechanisms. • Encourage GoG to systematically provide information to citizens and encourage them to fully participate in public engagement. Increase citizens’ education in civic engagement and rights. 4.2.3.2 Degree of Success in Building and Sustaining Effective Mechanisms for Public Engagement at the Local Level • Promote further reform of intergovernmental relations and civil service to truly energize citizen engagement at the local level. • Monitor and assess the Batumi model to examine its replication in jurisdictions of varying sizes and degrees of urbanization. • Consolidate the technical resources developed under different donor auspices and determine on which precedent best encapsulate best practices that other LGs could emulate. Integrate SEDP type strategic input into the mandated Budget Code’s BDD Medium Term Expenditure Framework requirement. Provide technical assistance to counterparts in achieving a higher level of specificity when drafting SEDPs. Make the current SEDP more of a formal precondition in allocating Regional Development Fund monies and other targeted central government transfers. • Look to amending the governing laws on spatial planning to make a SUP type process an integral, front-end part of the legally mandated master planning process. Policy reform should also consider moving oversight of local spatial plans from the MoE to MRDI, which is more fully coordinated with central government financial support for local infrastructure investment. • In any future USAID local governance planning, consideration should be given to revisiting the most promising aspects of the SIP concept. That said, less emphasis should be placed on producing stand- alone technical documents, and more on support for modified SIPs that engage citizens in service improvements identified as most immediate, and utilize citizen participant observation in monitoring results. 4.2.3.3 Municipal Level Partner Local Governments Institutionalization of Mechanisms of Public Engagement • Look to institutionalize public engagement during the critical March to August span of the budget calendar and allocating regional development funds and village improvement funds calendar. • Introduce user friendly, budget-in-briefs as a standard practice in sharing proposed, adopted, and executed budget data with the public over the course of the year. 4.3 QUESTION 3: SUSTAINABILITY AND IMPACT OF CAPACITY DEVELOPMENT FOR GOVERNMENT OFFICIALS 4.3.1 Findings 4.3.1.1 Most Effective Training and Capacity Development Efforts Within National and Local Levels Capacity development of civil servants became one of G3’s top priorities in the latter stages of the project. As such, within three years, the project provided capacity development to approximately 6,000 civil servants at GoARA and at the national and LG levels, in the form of short￾and longer-term training, coaching (on-the-job training), and facilitation. Most training courses included: Human Resource Management; Leadership Skills; Public Relations; Basics of Project Management; Ethics and Integrity in Civil Service; and Leadership and Innovations. Some public officials were also trained on policy paper drafting skills and several other subjects. Almost half of the civil servants received their training in 2013. Between July and December 2013, G3 trained almost 3,000 mid- and higher-level civil servants drawn from all GoG ministries in the following areas: • Governance • FOI and the OGP Action Plan • Ethics in Civil Service 27 • HR Management • Project Management • Monitoring and Evaluation • Strategic Planning • Legal Framework • Policy Making • Citizen Participation in Governance • Presentation and Writing Skills The 2013 training was coordinated by the Georgian Institute of Public Affairs (GIPA), the lead in a consortium that included the Center for Training and Consultancy (CTC), the Policy and Management Consulting Group (PMCG), and Robakidze University. According to GIPA’s Director, G3’s large-scale capacity development for civil servants in 2013 was coordinated with the PMO. It was defined as a pilot study designed to help decision makers determine the most effective approach to increase civil servants’ capacity development. Most civil servants were able to register for any course they wished to take. Others were sent by their offices to take specific courses from a schedule developed by the consortium. Some ministries, such as the Ministries of Internal Affairs, Justice, and Agriculture enrolled numerous participants. Others, such as the Ministries of Foreign Affairs, Defense, and Finance embraced the training opportunities more slowly. Feedback from consortium members and participants about the training varied. For example, several consortium members said the training was too intense and carried out too quickly; because of summer vacations, many of the capacity development programs did not start until September 2013 and continued for 3 consecutive months. Training participants, however, generally expressed positive feedback except in some cases where employees felt their supervisors mandated that they take the courses. Trainers, on the other hand, felt that bringing together employees from different ministries to attend the training helped develop communication and networks. In interviews conducted by the Evaluation Team with the Civil Service Bureau (CSB) - the bureau that coordinated G3 training of civil servants prior to the PMO - and trainers involved in the pilot project, the 2013 large-scale training coordinated in collaboration with the PMO created difficulties for the CSB because they no longer had a say in the selection process of employees to be trained. The CSB claimed that the PMO had bypassed them, while training of civil servants was main part of CSB’s responsibilities. Other KIIs argued that the training should have been better planned in terms of length of training and participants’ selection. However, trainers involved in conducting the training stated that while training 3,000 civil servants represents less than 5% of Georgia’s total civil servant population, it allowed at least 2-3 individuals from each institution to receive a well-structured training. Instructors suggested that some topics, particularly ethics and other similar courses, should be obligatory and taken regularly by civil servants in all government departments. The Evaluation Team conducted in-depth interviews with individual trainers and those from training institutions. All pointed out that conducting a needs assessment prior to designing and administering capacity development initiatives is very effective. They contended that this allows participants to take courses that best meet their needs and the needs of the institutions where they work. At the local level, the main G3 training curriculum, developed and delivered by PMCG, consisted of 11 topics selected based on G3’s survey of their partner LGs’ training priorities. These were: • Strategic Planning • Procurement • Project Design/Management/Evaluation • Asset Management • Media Relations • Project Auditing • Legal Drafting 28 • Municipal Web Portals • Managing IT Assessments • Investor Relations • Human Resources For only 3 of these 11 topics did G3 provide any related mentoring, facilitation, consulting or grant support, or other technical assistance that would have supported application of concepts learned in the PMCG workshops, as was done in the GoARA. Interviewees who had taken the courses found them interesting and informative but often not related to their work. They could not recall how they had used the training in their work. In most cases, participants interviewed indicated that the courses were too short to be able to impart comprehensive knowledge. To find out more about the results of trainings, the Evaluation Team also administered a mini￾survey with a randomly selected sample of 152 civil servants who had participated in at least one of G3’s trainings in 2013, at the local, national or GoARA level. Of this sample, the team received responses from 82 participants who collectively had attended 143 trainings on a wide variety of topics (see Table 4: Types of Training Attended, in Section 3.1.2). Training courses focusing on strategic planning and governance were the most frequently attended. In addition to short-term courses, more than half of the GoG and GoARA respondents also received coaching or facilitation skills training, as shown in Table 5: Other Capacity Building Received, also in section 3.1.2. Table 7, below, aggregates the results from the mini-survey as it relates to a self-reported increase in competency levels as a result of the G3 training. When asked, “How would you rate your increase in competence in specific skill areas as a result of the G3 program capacity building activities?” almost 84% of civil servants reported a large or fundamental increase in competence. Table 7: Increase in Competence Level Increase in Competence Level No. Percent Negligible Increase 5 3.5% Modest Increase 18 12.6% Large Increase 87 60.8% Fundamental Increase 33 23.1% Total 143 100.0% When asked to rate the ease of understanding of the training on a scale from 1-5, where 1 is easy to understand and 5 is difficult to understand, 43% of respondents rated it as quite easy to understand, 47% rated it right in the middle, and 11% rated it as difficult. Table 8: Ease of Understanding of Training Quality of training N Percent 1 (easy to understand) 30 21.0% 2 31 21.7% 3 67 46.9% 4 14 9.8% 5 (difficult to understand) 1 0.7% Total 143 100.0% Table 9.1 aggregates results from the mini-survey as it relates to effectiveness by type of capacity development provided by G3 (training, coaching though on-the-job training, and facilitation). All three types received a “good” or “excellent” rating from at least 83% of respondents. Of the three types, coaching through on-the-job training received the highest rating, with 88% of respondents rating it as an excellent or good mechanism for building capacity. In- 29 depth interviews conducted by the Evaluation Team confirmed this finding. This was particularly true in GoARA where coaching, combined with training in Strategic Planning and Program Budgeting, led to GoARA’s preparation of their 2014 Program Budget, which set a precedent among government budgeting b y including all the major features of a program budget. Table 9.1: Effectiveness in teaching new skills and building capacity Excellent (%) Good (%) Fair (%) Poor (%) N Training 39.0 43.9 13.4 3.7 82 Coaching 61.5 26.9 7.7 3.8 26 Facilitation 45.5 40.9 9.1 4.5 22 If we disaggregate this data by the level of government of the respondents, we can see in Table 9.2 below that while the majority of central government respondents gave a rating of excellent to effectiveness of coaching and facilitation (there was a split between good and excellent for training), the majority of GoARA respondents gave coaching a rating of excellent and facilitation only a rating of good (there was similarly a split between good and excellent). Local government respondents only received training and so did not rate the coaching and facilitation mechanisms. Table 9.2: Effectiveness in teaching new skills and building capacity by respondent level of government Type of Capacity Building Government Level GoG (%) Local (%) GoARA (%) Total (%) Training Excellent 44.4 33.3 43.8 39.0 Good 40.7 46.2 43.8 43.9 Fair 7.4 17.9 12.5 13.4 Poor 7.4 2.6 .0 3.7 Total 100.0 100.0 100.0 100.0 Coaching Excellent 64.3 .0 58.3 61.5 Good 21.4 .0 33.3 26.9 Fair 7.1 .0 8.3 7.7 Poor 7.1 .0 .0 3.8 Total 100.0 .0 100.0 100.0 Facilitation Excellent 61.5 .0 22.2 45.5 Good 23.1 .0 66.7 40.9 Fair 7.7 .0 11.1 9.1 Poor 7.7 .0 .0 4.5 Total 100.0 .0 100.0 100.0 Table 10, below, aggregates results from the mini-survey as it relates to factors respondents feel made capacity building mechanism more effective for them. Of those who gave a rating of “excellent” or “good” for at least one capacity building mechanism (68 individuals or 83% of respondents), 37% thought that the mechanism was effective because of content/relevance of the program and quality of individuals that delivered the training, while 34% thought that it was more the quality of the individual that contributed to the high rating than the content or relevance of the capacity building (12%). Table 10: Factors that Made Capacity Building Mechanism Effective What Made the Capacity Building Mechanism Effective? No. Percent of cases Both content/relevance of the capacity building and quality of individual that delivered capacity building 25 36.8 Only quality of individual that delivered the capacity building 23 33.8 30 What Made the Capacity Building Mechanism Effective? No. Percent of cases Only content/relevance of capacity building 12 17.6 Other 8 11.8 Total 68 100 Table 11 aggregates results from the mini-survey as it relates to factors contributing to the effectiveness of different types of capacity development mechanisms. A disaggregation of responses from those who rated the capacity building’s effectiveness as “good” or “excellent”, shows little or no difference among the types; 57% of respondents who rated the mechanism as good or excellent attributed it to the quality of the capacity building provider; while 43% of respondents attributed it to the content/relevance of the topic (see Table 11). Table 11: Factors That Made Each Type of Capacity Building Effective?* Quality of Capacity Building Provider Content / Relevance of Topic N Percent of respondents N Percent of Respondents Training 48 56.5 37 43.5 Coaching 19 57.6 14 42.4 Facilitation 15 57.7 11 42.3 * Question was asked only to respondents who thought the capacity building was good or excellent. While results from the mini-survey provided positive feedback on the capacity building interventions, the Evaluation Team’s in-depth interviews with key informants revealed that there are also problems that need to be addressed. For example, some participants felt there were considerable disparities in the training capacity and knowledge of different trainers. Others argued that some of the trainings were hastily assembled and that civil servants did not have enough time to prepare. Still others felt that rather than allocating the same number of days for each training, some courses should have been scheduled for more or less time, depending on the topic. These interviews also revealed that some directors and managers feel the course offerings are not coordinated with the needs of the agency and/or their staff and that the staff is often required to spend too much time out of their offices. Table 12, below, provides aggregated results from the mini-survey as it relates to the improvements that respondents see within their division as a result of what they learned in the trainings provided by G3. While more than 90% of respondents felt the courses they attended improved the knowledge within their division, less than half of the respondents saw improvement in the operation of their institutions in all other categories. The Evaluation Team believes this result is attributed to the recency of participants receiving training (2013) and the few persons within each institution being trained. Table 12: Improvements as a Result of Capacity Building Improvement in: No. Percent Transparency 20 24.4% Efficiency 29 35.4% Quality of service 34 41.5% Timeframe of the service 24 29.3% Increased engagement with civil society 28 34.1% Better communication 37 45.1% Better planning 36 43.9% 31 Improvement in: No. Percent Better management 35 42.7% Improved knowledge 74 90.2% No improvement in any of the categories 4 4.9% 4.3.1.2 Extent to Which Targeted Institutions Sustained Results of G3 Interventions and Key Contributing Factors For several years, the MoJ and GoARA have been recipients of USAID’s technical assistance in institutional strategic planning, first through PAR and then through G3. Through these projects, both institutions updated their strategic plans, performance metrics, program budgets and communications plans. Yet, the results have not been the same in each case. The Evaluation Team did not explore in-depth the reasons for these differences, but information they gathered through KIIs suggest the scale of training, the size of government agency, and political will of the agency’s leadership are contributing factors influencing the outcome. In the absence of baseline data, the Evaluation Team has little to measure sustained results of the G3’s capacity development interventions. However, positive examples such as GoARA’s ability to produce adequate program budgeting, described above, demonstrates the effectiveness of consistent, long-term capacity development using a variety of mechanisms. GoARA’s officials were trained in strategic planning and program budgeting, and were assisted with on-the-job training through coaching from national and international experts. The long-term investment has proven successful in achieving a 2014 program budget that far exceeds what the MoF and other central government ministries have accomplished. In contrast to GoARA, despite similar G3 interventions, the MoJ failed to reflect the results of the strategic planning in its 2013 and 2014 program budgets, which was included in G3’s Year2 work plan. Working with Evaluation and Management Consulting Ltd. (EMC), G3 assisted the MoJ in carrying out a participatory review and revision of its vision, mission, corporate values and goals. With EMC’s facilitation, MoJ completed the design of the 2012-2014 Strategic Plan, which incorporates performance outcomes and time-bound action plans that link the strategies to the MoJ’s day-to-day activities. EMC also collaborated with the MoJ to design a performance monitoring plan (PMP) to monitor and evaluate implementation of the Strategic Plan. It was expected that with these tools the MoJ would have been able to monitor and report on its own progress towards agreed goals, and take remedial actions to improve performance. Despite efforts to develop the appropriate tools, the 2014 MoJ program budgets show no significant improvements. The 2012-2014 MoJ Strategic Plan incorporates performance outcomes and time-bound action plans that link the strategies of the MoJ to day-to-day activities but they are not reflected in the 2014 program budget. The Evaluation Team could not explore the numerous capacity development activities G3 carried for SAO. Yet, it learned of G3’s successful capacity building initiative in support of Mr. Giorgi Kapanadze, a Performance Audit Department auditor at the State Audit Office (SAO) (former CCG). Mr. Kapanadze participated in a four-month training at the U.S. General Accounting Office in 2012. According to G3’s report, after Mr. Kapanadze’s return, the SAO carried out one of its first performance audits of the state of the public internal financial control in Georgia, which exceeded the expectations of experts, including those that monitor budgets at TI. During the interview with the Evaluation Team, Mr. Kapanadze stated that the intensive four-month training helped him form his vision as a performance auditor, which he shares with his colleagues both in Tbilisi and Kutaisi. 4.3.1.3 Training and Capacity Building Approaches Most Useful to Adopt for Future Efforts to Sustainably Improve Capacity and Skills of Government Officials During in-depth interviews, the Evaluation Team sought input from respondents about their opinion on how G3’s capacity building efforts could be improved. Many respondents interviewed suggested that longer term training of pre-evaluated participants on specific subjects might bring better results than the large-scale short-term training, especially training that was provided absent of a needs assessment or pre-qualification of candidates. Georgian Foundation for Strategic and International 32 Studies (GFSIS), a local organization who has provided capacity building for the Ministry of Defense (MoD) for several years with good results, argues that pre-qualification of students, a practice they regularly incorporate in their training, allows for designing courses that best fit the interests, capabilities, and needs of participants and the agencies where they work. GFSIS applied this same methodology in a longer-term (3 month) G3-funded training of 22 civil servants selected from a pool of 160 civil service applicants, and found it successful, even though the training was shortened from GFSIS’ standard 6-month training. The Evaluation Team learned from many of those who participated in KIIs, including individuals from the State Chancellor, that conducting a needs assessment for agencies prior to designing and providing training is an essential first step to achieving better results for the institutions. An example demonstrating this practice includes several G3 trainings offered to officers and managers at Georgia’s Revenue Service Prevention Department. G3 carried out a needs assessment and capacity evaluation for tax officers; designed the training modules for tax officers; developed Trainer of Trainers sessions, which included training, coaching, mentoring and facilitation approaches for skill and knowledge development of tax officers; and delivered the courses. G3 also provided the HRD of the Revenue Service with HRD procedures focusing on development needs assessments, including planning and evaluation, tools and formats. Through KIIs, the Evaluation Team also found that several capacity development implementers and CSO representatives believe that by providing most training, international donors are freeing GoG from assuming more responsibility for training its own staff. These respondents argue that cost-sharing increases the government’s sense of ownership. While many government ministries may not currently have the funds to invest in professional development of their staff, some ministries such as the MoF and the MoD, have done so successfully. Table 13, next page, aggregates results from the mini-survey as it relates to currently employed civil servants’ priority areas of capacity building for the next five years. As the table indicates, respondents’ interest spanned across a large number of priority topics, with the majority most interested in Public Relations (30%), HR Management (26%), Program Budgeting (19%), and Strategic Planning (19%), among others. Table 13: Preferred Areas of Future Capacity Building Capacity Building Topic N Percent of respondent Public Relations 23 29.5% HR Management 20 25.6% Program Budgeting 15 19.2% Strategic Planning 15 19.2% Governance 14 17.9% Investor Relations 12 15.4% Legal Framework of Georgia 9 11.5% State Procurement 8 10.3% Project Management 7 9.0% Public Policy 7 9.0% Presentation and Writing Skills 5 6.4% Project Design, Management and Evaluation 5 6.4% E-Communication Training 4 5.1% Design and Management of Municipal Web Portals 4 5.1% 33 Strategic Urban Planning Workshops 4 5.1% Managing IT Assessments, Strategies and Capacity Development Plans 3 3.8% Skills facilitation 3 3.8% Generating Recommendations for Citizen Participation in Local Self￾Government Workshops 3 3.8% Action Plan Drafting Workshops 3 3.8% Establishing a PR Workshops 3 3.8% Policy Development 2 2.6% Process of Policy Development 2 2.6% Audit 2 2.6% Economic Development 2 2.6% International Practices 2 2.6% Legal Drafting and Juridical Documentation 1 1.3% Civil Service Reform Workshops 1 1.3% Ecology, Environmental Protection 1 1.3% Agriculture Projects 1 1.3% Foreign Languages 1 1.3% Energy Sources 1 1.3% International Security 1 1.3% When respondents of the mini-survey were asked to select only one area of capacity building, the results were similarly spread out across a large number of topics (see Table 14, below). The top three capacity building preferences are the same as above, but in a slightly different order: Program Budgeting (22%), HR Management (13%), and Public Relations (9%). Strategic Planning is less of a priority. Table 14: Single Preferred Topic of Future Capacity Building Capacity Building Topic N %t of respondents Program Budgeting 17 21.8% HR Management 10 12.8% Public Relations 7 9.0% Governance 6 7.7% Project Management 5 6.4% Strategic Planning 4 5.1% Investors Relations 4 5.1% E-Communication Training 3 3.8% State Procurement 3 3.8% Strategic Urban Planning Workshops 3 3.8% Legal Framework of Georgia 2 2.6% Presentation and Writing Skills 2 2.6% 34 Capacity Building Topic N %t of respondents Design and Management of Municipal Web Portals 2 2.6% Audit 2 2.6% Public Policy 1 1.3% Managing IT Assessments, Strategies and Capacity Development Plans 1 1.3% Project Design, Management and Evaluation 1 1.3% Ecology, Environmental Protection 1 1.3% Economic development 1 1.3% Agriculture projects 1 1.3% Foreign languages 1 1.3% Marketing/Tourism 1 1.3% Total 78 100.0% When respondents of the mini-survey were asked to self-report their current competency as of December 2013, the average respondent’s current competence rating was less than a “normal” level and, in aggregate across the training topics, in 92.3% of the cases, additional training on the topic was requested. The skill areas with less than a “normal” level on average are highlighted in Table 15, below. Table 15: Possible Future Skill Areas for Capacity Building Skill area Competence Rating (1 -4) Average rating 1=no competence 2=little 3=normal 4=expert Would like additional training % trained that said yes N (total number of resp. trained) % of total number of resp. Women Men All 1. Public Relations 2.89 N/A 2.89 100 9 11.0 2. E-Communication Training 2.50 3.50 2.83 83.3 6 7.3 3. Governance 2.46 2.50 2.47 100 19 23.2 4. HR Management 3.00 3.33 3.23 92.3 13 15.9 5. Legal Framework of Georgia 3.00 N/A 3.00 100 1 1.2 6. Policy Development 2.33 3.00 2.50 75.0 4 4.9 7. Presentation and Writing Skills 2.80 2.50 2.71 85.7 7 8.5 8. Process of Policy Development N/A N/A N/A N/A 0 0 9. Program Budgeting 2.60 2.75 2.64 100 14 17.1 10. Project Management 2.67 2.75 2.71 85.7 7 8.5 11. Public Policy 2.83 3.00 2.90 100 10 12.2 12. State Procurement 2.67 3.00 2.89 100 9 11.0 13. Strategic Planning 3.09 3.00 3.05 95.2 21 25.6 35 Skill area Competence Rating (1 -4) Average rating 1=no competence 2=little 3=normal 4=expert Would like additional training % trained that said yes N (total number of resp. trained) % of total number of resp. Women Men All 14. Design and Management of Municipal Web Portals 2.50 N/A 2.50 100 4 4.9 15. Investor Relations 3.00 2.50 2.75 100 4 4.9 16. Managing IT Assessments, Strategies and Capacity Development Plans 2.00 N/A 2.00 0 1 1.2 17. Project Design, Management and Evaluation N/A 3.00 3.00 100 2 2.4 18. Skills facilitation N/A N/A N/A N/A 0 0 19. Legal Drafting and Juridical Documentation 3.00 N/A 3.00 0 1 1.2 20. Generating Recommendations for Citizen Participation in Local Self-Government Workshops 2.00 2.00 2.00 50.0 2 2.4 21. Civil Service Reform Workshops 3.00 N/A 3.00 100 1 1.2 22. Strategic Urban Planning Workshops 3.00 N/A 3.00 75.0 4 4.9 23. Action Plan Drafting Workshops 3.00 N/A 3.00 50.0 2 2.4 24. Establishing a PR Workshops 3.00 N/A 3.00 100 2 2.4 25. Open Government Partnership Action Plan Workshops N/A N/A N/A N/A 0 0 4.3.2 Conclusions 4.3.2.1 Most Effective Training and Capacity Development Efforts Within the National and Local Levels • Conducting a needs assessment for agencies prior to designing and providing training is an essential first step to achieving better results for the institutions. • At the local level, the major curriculum based series of training workshops seemed to have been delivered but, perhaps on too many topics. Eight of the eleven workshops were unconnected to on- going G3 technical assistance. • Although most participants praised the trainers and the content of training, some felt that there were considerable disparities in the training capacity and knowledge of different trainers. In addition, some interviewers felt that they were not allowed enough time to prepare, and that staff is often required to spend too much time out of their office. • Based on KIIs, most effective training and sustainable capacity development efforts have 36 resulted from a combination of training and coaching, or on-the-job assistance to apply the training, as seen on the program budget results in GoARA. 4.3.2.2 Extent to Which Targeted Institutions Sustained Results of G3 Interventions and Key Contributing Factors • Several factors contribute to institutions sustaining results of G3 interventions. The prime factor seems to be the presence of political will of the leadership. Without a commitment to reforms, interventions are rarely successful in implementation. The size of the institution, or the government as in the case of GoARA, also plays a key role. These two factors, among others, may explain why MoJ and GoARA, which have received similar technical assistance over the years, have had different outcomes in sustaining results. GoARA’s experience also seems to indicate that long-term training and on-the-job support brought best results to G3 interventions. • Lack of baseline data makes it difficult to measure the impact and sustainability of the project’s capacity development interventions. • SAO is another example of sustained results due to the long term four-month training of a key staff member. 4.3.2.3 Training and Capacity Building Approaches Most Useful to Adopt for Future Efforts to Sustainably Improve Capacity and Skills of Government Officials • Pre-evaluation of selected participants was considered very important and a step that might produce better and more sustainable results than short-term training of civil servants that are not screened for their qualifications. • Among CSO representatives, there is a concern that international assistance may be allowing the GoG to assume less responsibilities in capacity building than it has the capacity to assume. Many respondents feel that cost-sharing increases government’s sense of ownership. • While there are no clear self-reported priority areas for future capacity building, the mini￾survey suggests that possible topics that should be further explored must include Program Budgeting, HR Management, and Public Relations, with Program Budgeting being a top priority. There is also a need for continued capacity building for almost all other topics. • Looking only at the mini-survey results, at the GoG level, coaching and facilitation were very highly rated as effective capacity building mechanisms whereas among the GoARA coaching was more favored than facilitation. 4.3.3 Recommendations The Evaluation Team provides the following recommendations for USAID and the GoG to consider in response to the three sub-questions under Question 3: Sustainability and Impact of Capacity Development for Government Officials. Most of these recommendations are supported by information discussed in-depth during the Evaluation Team’s interviews with key informants. They look at future USAID programming and reflect recommendations made by both government officials and civil society representatives. • Course offering should be better coordinated with the needs of the agencies and their staff. • Provide more targeted training, learning from the best practices experienced by GoARA and other national institutions or agencies. • Standardize quality of trainings and trainers, and facilitate training materials in advance. • Allow for more time and better planning when preparing for capacity development of civil servants to ensure employees have adequate time to prepare. To minimize employees’ time outside of the office and encourage good governance practices, consider developing required on￾line courses on basic topics for all civil service employees such as conflict of interest, transparency, management of government property, discrimination, sexual harassment, and others, to be conducted periodically. • Consider focusing more on longer-term training combined with on-the-job training rather than short-term, large scale training. • Any future project should establish baselines that will assist in measuring success of the sustainability of capacity development interventions. • Selection of participants for certain training should be made based on their qualifications and their needs. • Encourage the GoG to share costs for training of its employees as cost-sharing is known to 37 increase government’s sense of ownership. • Focus and embed training in ongoing TA in applying concepts learned since many civil servants pointed to the need for continued capacity building. Provide more training in the areas identified as more important and problematic, including Program Budgeting, HR Management, and Public Relations. • Consider providing more coaching and on-the-job training for better and more sustainable training results. 5.1 INTERVENTIONS/APPROACHES RECOMMENDED FOR FUTURE PROGRAMMING • Numerous G3 interventions and approaches have proven successful and should be continued in future programming. It should be noted that G3 initiatives have succeeded only whenever political will to carry out reforms has been present. Without political will, G3’s best interventions and approaches have been held back by the institution’s leadership. • In the future, USAID should attempt to strike a balance between hands-off and hands-on approaches. There have been cases in which failure to follow up and monitor an agency’s implementation of training/coaching has led to the activity not achieving expected results. This is the case, for example, of the MoJ’s Performance Appraisal for which G3 did not do a follow-on when its implementation was placed on hold. • While G3 has served as “mediator” in bringing GoG and CSOs closer, it should consider coaching and assisting the GoG in engaging CSOs directly as essential collaborators and also critical watchdogs. KIs suggested that G3 and future programming could help GoG understand that it should work directly with think tanks and CSOs. These organizations support the government, through international donors, in capacity development, research and other activities. They would like to provide services directly to the government. • Use of local talent as USAID consultants is a welcomed approach. However, KIIs as well as the mini-survey that the Evaluation Team conducted with representatives of CSOs and civil servants, revealed that the combination of local and international experts is considered the best option because the international experts provide experience and fresh ideas while the local experts help contextualize them. • The contributions G3 has made in building knowledge and providing international experts who conduct in-depth studies on key areas, such as the drafting of the civil service reform concept, are well-appreciated by GoG as well as by local implementers and CSOs representatives. GoG would like that type of assistance to continue. In fact, the PMO would like to be able to use a similar methodology to that used for the civil service reform concept paper to move toward a full public administrative reform with USAID’s support. • Look at forthcoming local election campaigns as an opportunity to increase awareness of local governance issues and inject into campaign dialogue the discussion of needed intergovernmental relations and LG reforms. • Promote the replication of local CEMS, and possibly support LGs in performance-based management practices. Implementation plans should include a more organized approach to training and mentoring LG system administrators and coordinators from the outset, including incorporating new features of available open-source software packages. • In future local governance programming, more focus should be paid t o d e v e l o p i n g a n d utilizing performance measures in key targeted sectors of LG operation. Explore opportunities to do so in cooperation with the ongoing MoF/GiZ initiative to support fuller implementation of the now mandated program budgeting at the local level. Consider possibilities for incorporating joint LG/CAC monitoring of CEMS databases in any such performance measurement system. • If supporting the further development of civil society and media watchdog capabilities, do so 38 through a separate CSO-focused contracting or grant vehicle. Focus on developing more sophisticated capabilities to monitor the core accountability functions with respect to budget execution, asset management (land sales/leases), and procurement. • Explore opportunities for USAID to provide planned certification programs for financial managers, auditors, procurement officers, and municipal lawyers. • At the local level, conduct cluster-based training that brings professional peers together to prepare action plans and reconvenes them periodically to foster peer mentoring and best practice sharing among LGs and the development of professional expert networks, and possibly new associations (e.g. for HR officers, for procurement officers). • In the near term, especially if G3 is extended into the summer of 2014, USAID should look for ways to involve the media and CSOs in injecting a more sophisticated dialogue on LG policy issues into the local electoral campaign. It may still be timely to cooperate with IREX’s planned Regional Election Initiative in 2014, particularly by helping provide content on front burner LG issues into electoral coverage, priming journalists to probe candidates’ positions on topics such as merit-based hiring practices at the local level. • In any major new local governance project, rather than providing grants directly to TV stations themselves, consideration might be given to building the capacity of broadcasting personnel and print journalists in covering municipal matters. This could include relatively-low cost concepts such as: - Media Breakfasts: invite media reps to brunches with panelists on LG topics of current interests, and take advantage of visiting short-term international experts to share their experience. Providing capacity building in this type of more informal/social setting has proven more effective in engaging practicing journalists who are reluctant to take the time off to attend formal trainings. - Specialized training on LG topics with promising angles as the subject for investigative journalism. Starting in March 2014, IREX is planning to host a 9-day school of journalism. For this training or others, USAID might support the development of a module on one or more topics where corruption, waste, and conflicts of interest are most rife at the local level. Land relations, including the misuse of eminent domain powers and non-competitive land sales, might prove fruitful topics. • Procure high quality programming content to share with broadcast media throughout the country. Through direct grants, USAID should contract with a TV production or documentary film company to develop a series of programs on priority topics for LG reform. The programs could then be provided free of charge to any Georgian TV station interested in showing it. This approach could deliver higher quality content to a larger number of viewers than funding multiple regional media outlets directly. In a well-designed, 3-5 year project, this approach can prove effective during the initiative’s concluding two years as a vehicle to document and promote best practices and lessons learned. • If and when USAID again supports broadcasters directly, the Evaluation Team recommends that eligible grantees be defined broadly enough to permit a competitive application process, in the interest of eliciting more creative proposals and allowing prospective media grantees to be screened based on their management qualifications. As part of any such grant program, USAID might also look for practical ease to document viewer responses to the supported programming. This documentation could then be used by the media grantee to help market public programming to prospective advertisers after the grant support has wound down. Further, the type of monitoring and results reporting initiated by the Poti and Marneuli media grantees should be encouraged. • Surveys by the Caucasus Research Resource Center – Georgia (CRRC) and others indicate that over 90% of the Georgian adult population relies primarily on TV for information on public affairs. Thus, any donor program looking to inform the public at large needs to include a TV component. Relying solely on print, internet, and radio is not a viable option. Social media should also be a viable means. 39 ANNEXES 40 ANNEX I: EVALUATION STATEMENT OF WORK 41 SECTION C- DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK STATEMENT OF WORK MID-TERM PERFORMANCE EVALUATION OF THE GOOD GOVERNANCE IN GEORGIA (G3) PROGRAM Summary Name of the Project (to be evaluated): Good Governance in Georgia (G3) Project Number: AID-114-TO-11-00001 Project Dates: February 3, 2011 to March 13, 2014 Project Funding: Ceiling: $16,193,008 Implementing organization/s: Management Systems International (MSI) COR/ACOR: Lina Panteleeva, Khatuna Khvichia The Good Governance in Georgia (G3) program addresses the various challenges facing governance at the local and national levels and offers public space for citizens of Georgia to engage with each other, their elected officials, the media, civil society and members of political parties. The G3 program aims to improve transparency at all levels of government, thereby reducing opportunities for corruption. It strengthens citizens’ abilities to combat corruption and puts into place stronger mechanisms and systems that prevent opportunities for corruption to develop. This evaluation will include specific aspects of two components of G3: component one which relates to improved public administration at the federal level; and component 2 regarding more responsive, professional and engaged local governance. The evaluation will not cover the third component of G3, which provides public gathering spaces dedicated to civic discourse and offers the impetus for organizations to utilize that space. Summary of Specific Technical Requirements The contractor shall ensure that the evaluation team will provide the following deliverables within the terms defined by the contract: 1. Draft evaluation design 2. In-brief with USAID management to present the detailed evaluation design and work plan 3. Conduct evaluation of G3 project in accordance with the USAID-approved evaluation design 4. Provide draft and final evaluation reports to USAID in accordance with the reporting guidelines 5. Out-brief with USAID management to present initial findings, conclusions and recommendations I. Background When the G3 activity was designed in 2009, the overall effectiveness of Georgia's governance had improved dramatically in many ways since the Rose Revolution. At the same time, the openness, responsiveness and accountability of the governance sector remained problematic. While the state’s efficacy - its capacity to achieve its goals - had increased substantially, its connection with society had been subordinated to the imperative of carrying out a broad array of reforms quickly. Government had successfully tackled petty corruption in state agencies, improved service provision significantly, and enlarged usage of e-governance systems across state agencies, but had put little emphasis on developing a human resources system for state employees or increasing proficiency of public servants. Further, little 42 emphasis was placed on institutionalizing public consultation, ensuring democratic and transparent decision-making processes and the accountability of governmental agencies. Addressing good governance on the central and local levels, and providing a safe place for citizen interaction around current political and social topics, G3 has supported and encouraged the institutionalization of transparency, accountability and responsiveness to the public in all levels of government structures. Under the first component, G3 promotes improved public administration at the federal level, advancing public sector reform in target institutions using human and institutional capacity development approaches. G3 has supported training of public officials and civil service reform as political will has allowed. Under component one, G3 has worked with several ministries and agencies of the national government, including: the Civil Service Bureau (CSB), Ministry of Regional Development and Infrastructure (MRDI), Ministry of Finance (MOF), Ministry of Justice (MoJ), State Competition and Procurement Agency (CSPA), State Audit Office of Georgia (SAOG), Data Exchange Agency (DEA), Civil Registry Agency (CRA), Ministry of Education and Science (MOES), and the Prime Minister's Office (PMO), as well as the Government of the Autonomous Republic of Ajara (GoARA). G3's relationship with the PMO is new developing since the 2012 parliamentary elections. Due to the fact that the PMO will have new functions after the new Constitution enters into force in late 20 13the G3 project has engaged with the PMO in strengthening its capacities necessary for fulfillment of those functions. In particular, G3 will be assisting PMO in strengthening its coordination and public communication capacities. The PMO has become a key partner in coordination of training of up to 3,000 public servants during the last year of G3 project implementation. Given the interest of the GOG in initiating civil service reform, the PMO has become a focal point for the G3 project in its work on the development of the concept of civil service and facilitation of the processes related to this reform. The Ministry of Finance is also a new partner since late 2012. At the request of the Minister, USAID and G3 have begun to support the introduction of a results-based management system at the MOF. Local governance decentralization reform is among the key priority issues on the new government's agenda, and G3 has supported this process through its work with the MRDI. Among other things, G3 has supported functioning of the Secretariat for the Advisory Council of Local Governance Decentralization jointly established by MRDI and the Parliamentary Committee on Local Government issues. G3 has been providing technical and administrative support to the work of the Advisory Council, as well as supporting public discussions on key issues of the reforms. Under the second component, G3 aims to increase the responsiveness, professionalism, and engagement of local governments. G3 advances the development of professional cadres in the local governing sector, increasing their capacity for informed decision making and improving target municipalities' ability to ret1ect the interests of their communities while approaching the central government for funding support. G3 's second component works with ten partner municipalities, chosen on the basis of demographic, political, social and geographic factors: Akhaltsikhe, Batumi, Gori, Kutaisi, Marneuli, Ozurgeti, Poti, Rustavi, Tevali and Zugdidi. 43 The change of government following the October 2012 Parliamentary elections also brought many changes at the municipal level across Georgia. In the months immediately following the election, the cont1ictive political context in most of G3's partner local governments interfered with many of G3's planned activities. Conflicts arose within the elected councils, the result of the realignment of council political factions. UNM members left to establish new factions, whether as Georgian Dream factions or as putative "independent" factions. There were also conflicts within the UNM itself, as national and regional party leaders jockeyed for position in the new political context. Following the elections, in all but one of G3's partner local governments-Rustavi is the exception-there was a change in the mayor/gamgabeli and/or the council chair. The immediate effects of the change have included disruption of council decision-making, rotation of senior managers, loss of continuity in relationships between partner local governments and G3 program staff, and weakened interest in G3 interventions. While G3 took measures to rebuild relationships, meeting with newly appointed mayors/gamgabelis and deputies to present program objectives and activities, the process of political transition at the local level meant a loss of several months in the implementation of most G3 activities under component 2. In some of the municipalities G3 had to stop its interventions partially or completely. For instance, G3 suspended all its activities with the Batumi Municipality. G3 reduced its assistance in urban plan development from five to two municipalities. G3 made the decision not to continue supporting Service Improvement Plan (SIP) development in 5 municipalities in Western Georgia after analyzing results of this exercise in 5 municipalities in Eastern Georgia and given lack of interest of the remaining municipalities. Major results of the activity to date:  During program years I and 2, an important activity under component I was to provide the GOG with technical assistance to link persons with their residential address within the city of Tbilisi. The major outputs of G3's addressing work were 365,481 mapped buildings in Tbilisi utilizing GIS and incorporated into National Agency of Public Registry's mapping systems along with 985,443 physical persons' records.  Under component I, G3 carried out activities to strengthen the capacity of its partner agencies in strategic planning and institutional development, providing them with technical assistance that built upon their specific needs (strategic planning, monitoring and evaluation, public communication, human resources management, etc.). During Year I, G3 worked with the MOJ, CRA, the SAOG and GoARA in this area. The ongoing assistance in Year 2 was based on demand and expanded the number and type of governance work areas as well as institutions supported by G3. In addition, G3 focused on key cross￾government work areas that lend to program delivery for the whole of government, e.g. human resources management, FOIA, and other areas.  G3 has conducted a comprehensive set of capacity building activities at GoARA, which entailed working on strategic planning, institutionalization of monitoring and evaluation tools, and design of a comprehensive e-communication system.  Under component I, G3 has provided support to the GOG in the implementation of FOIA standards, in part to support the GOG's Action Plan under the Open Government Partnership (OGP) initiative. G3 facilitated a series of workshops and roundtables for not only high level government officials, but also for civil society. The work with government officials led to a transfer of knowledge and skills on consistent and workable FOI compliance practices across government agencies to disseminate public information. G3's work with CSOs led to enhanced mechanisms for accountability of government and its institutions to deliver information in a timely manner. 44  Under component 2, G3 initiated capacity development activities in 13 areas of local government management and governance at I 0 partner municipalities. However, at the request of the GoG and as approved by USAID, from time to time G3 capacity development interventions encompassed almost all local governments in the country. For instance, G3, in cooperation with the MoF, provided training in program budgeting for relevant staff of all 65 municipalities.  G3 has supported urban strategic planning activities for partner municipalities. Preparatory work started with five municipalities, but ended up with only two, Rustavi and Poti. G3 also did preparatory work in helping municipalities with their socio-economic plan development processes; all I 0 partner municipalities started the process, but only nine completed it, due to the suspension of activities with the city of Batumi. G3 helped five partner municipalities develop service improvement plans.  G3 focused on establishing Civic Advisory Councils (CACs) made up of LG representatives and CSOs to improve civic engagement in local decision-making. G3 supported establishment and operation of 10 CACs at partner municipalities through grants to CSOs.  G3 provided grants to local TV outlets to improve the frequency and quality of TV coverage of municipal affairs and raise public awareness on LG issues in seven partner municipalities. II. Purpose of the Evaluation and Its Intended Use An evaluation of the G3 program is needed at this time because the task order itself calls for a mid￾term evaluation of the program in its monitoring and evaluation plan. Further, the Office of Democracy and Governance and the wider USAID Mission want performance information from the G3 evaluation to inform ongoing and related future programming. In particular, the Office of Democracy and Governance is interested to know: 1) which interventions/approaches have proven to be feasible/sustainable in the changing political context; and 2) which interventions/approaches provide a sound foundation for future programming (continue, expand, modify, etc.). The information is needed in CY 2013 to ensure utilization in the design of related future programs. The political dynamics surrounding the 2012 change of government have had a significant impact not just on the governance institutions in Georgia, but also on the ability of the G3 project to have an impact through working with those institutions. The evaluation team will need to formulate their findings in relation to those changes. Given that the answers to all evaluation questions are potentially affected by those changes, it will be important for the answers to indicate what was accomplished prior to and after the changes, including any important shift in dynamics. III. Evaluation Questions and Methodology The evaluation design will be submitted by the contractor in response to the RFTOP and reviewed by USAID. The finalized evaluation design must be submitted to the TOCOR three workdays prior to the team’s arrival in-country. The evaluation design must outline in detail what methods the contractor will use to get answers for each evaluation question. The evaluation design must include a detailed evaluation matrix (including the key questions, methods and data sources used to address each question and the data analysis plan for each question), draft questionnaires and other data collection instruments or their main features, known limitations to the evaluation design, a work plan, and a dissemination plan. This information together with the Mission’s comments will be discussed in detail during the in￾brief meeting with USAID. The work plan must include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation team. 45 In order to evaluate the questions below related to G3 performance, the evaluation team must review pertinent background documents relating to G3 and the broader governance context in Georgia on the federal and local levels, as well as conduct interviews in Washington, DC and field work in Georgia for three weeks. Among other methodologies, the team must collect information from primary sources such as Georgian counterparts, the beneficiaries of the assistance; CSOs involved with G3; and other donors and implementing partners. For the purposes of answering the evaluation questions below, "institutionalized" means not only establishment of some mechanism, but regular utilization of the mechanism by the targeted institution without project support, or the institution's expression of a strong commitment to do so. The questions the evaluation is expected to specifically look at are as follows: 1. Impact on Government Transparency and Accountability. G3 was designed to increase transparency and accountability at all levels of government. What results did G3 achieve both before and after the 2012 change of government? How have G3 efforts in e-governance advanced transparency and effectiveness in the agencies targeted for assistance? How have G3’s efforts in advancing freedom of information compliance and facilitation of OGP Action Plan implementation process contributed to transparency, openness or responsiveness of the targeted GOG institutions? On the local level, what contributions have G3 grants to regional media outlets and local NGOs made to citizen awareness and engagement related to transparency and accountability of local government? Which of the selected interventions/approaches are recommended for future programming? 2. Effectiveness of Government Mechanisms for Public Engagement. G3 has worked to build public engagement mechanisms at the national and local levels through direct work with the government institutions and through/with CSOs. To what degree has the G3 approach been successful in building and sustaining effective mechanisms for public engagement at the national (GoARA, CSPA, MOJ) and local levels? Which aspects worked, which did not and why? On the municipal level, have partner local governments institutionalized mechanisms of public engagement (public hearings on budgets and other major functions, public calendars, bulletin boards, press releases, display boards or other for a for public hearings)? 3. Sustainability and Impact of Capacity Development for Government Officials. G3 provided a variety of capacity development activities to government officials on public policy development and drafting, monitoring and evaluation, strategy development, improving inter￾and intra-agency communication, human resources management, and program budgeting at the national and local levels. Which G3 training and capacity building efforts within national and local level government institutions have been most effective? To what extent have targeted institutions sustained results of G3 interventions and, when they did, what were the key factors that contributed to this? Which training and capacity building approaches would be most useful to adopt for future efforts to sustainably improve the capacity and skills of government officials? The evaluation matrix below is illustrative and USAID expects that the contractor will suggest the best methods that would generate most reliable and evidence-based answers to the key evaluation questions. 46 Research Question Target Interlocutors Data Source Methodology 1. Impact on Government Transparency and Accountability. G3 was designed to increase transparency and accountability at all levels of government. What results did G3 achieve both before and after the 2012 change of government? How have G3 efforts in e-governance advanced transparency and effectiveness in the agencies targeted for assistance? How have G3’s efforts in advancing freedom of information compliance and facilitation of OGP Action Plan implementation process contributed to transparency, openness or responsiveness of the targeted GOG institutions? On the local level, what contributions have G3 grants to regional media outlets and local NGOs made to citizen awareness and engagement related to transparency and accountability of local government? Which of the selected interventions/approaches are recommended for future programming? MOJ, GoARA, civil society, citizens in target municipalities Interviews with counterparts, USAID, MSI staff, surveys of citizens, review of television schedules, review of ministries’ websites, review of public bulletin boards of ministries and local elected bodies Interviews, document review, survey. 47 2. Effectiveness of Government Mechanisms for Public Engagement. G3 has worked to build public engagement mechanisms at the national and local levels through direct work with the government institutions and through/with CSOs. To what degree has the G3 approach been successful in building and sustaining effective mechanisms for public engagement at the national (GoARA, CSPA, MOJ) and local levels? Which aspects worked, which did not and why? On the municipal level, have partner local governments institutionalized mechanisms of public engagement (public hearings on budgets and other major functions, public Calendars, bulletin boards, press releases, display boards or other fora for public hearings)? MOJ, CSPA, GoARA, civil society, citizens in target municipalities, members of CACs, staff of NGOs that received grants under G3 Interviews with target interlocutors, review of public bulletin boards of ministries and local elected bodies Interviews, roundtable groups document discussions or review, focus 48 3. Sustainability and Impact of Capacity Development for Government Officials. G3 provided a variety of capacity development activities to government officials on public policy development and drafting, monitoring and evaluation, strategy development, improving inter- and intra-agency communication, human resources management, and program budgeting at the national and local levels. Which G3 training and capacity building efforts within national and local level government institutions have been most effective? To what extent have targeted institutions sustained results of G3 interventions and, when they did, what were the key factors that contributed to this? Which training and capacity building approaches would be most useful to adopt for future efforts to sustainably improve the capacity and skills of government officials? Government officials who received training under G3, other members of staff of target institutions as control group Interviews with counterparts, trainees, USAID, MSI staff Interviews, document review, survey of training participants, etc. Estimated timeframe The deliverables associated with this contract must be completed and accepted by USAID/Caucasus by January 31, 2014. Documents review: November 25-29 In-country work: December 2-December 20 Draft evaluation design and a work plan submitted to USAID: November 30 In￾brief at USAID: December 2 Out brief at USAID and outline of the final report: December 20 Draft final reports due to USAID: January 10 USAID comments due to the contractor: January 24 Final report: January 30, 2014 The last date of finalizing report: January 31, 2014 Logistics USAID will not be responsible for arranging logistics for the evaluation team. The COR will put the contractor in contact with its implementing partner and might provide help with a small number of meetings, such as meeting with USG agencies where needed. Other than that, the burden of logistics including meeting setting, transportation, interpretation, lodging and all other logistical issues will fall to the contractor. 49 The contractor must evaluate G3's work in some combination of Ajara, Rustavi, Kutaisi, Marneuli, Telavi, and/or Gori. Projects documents for review To the extent possible, relevant reports and other project documentation will be provided by the Mission to the contractor prior to travel to Georgia. These documents are: • Project Description as is stated in the award; • Implementing partner's Quarterly Reports; • Initial list of in-country contacts; • PMP indicator tables; • M&E plans submitted and approved by USAID; • Implemented monitoring reports; • Other deliverables (expert report, publications) produced by partner. USAID will provide the regular performance monitoring data that it has collected over the life of the G3 project. END OF SECTION C 50 ANNEX 2: EVALUATION MATRIX 52 Research Question Indicator Target Interlocutors Data Source Methodology Impact on Government Transparency and Accountability. G3 was designed to increase transparency and accountability at all levels of government. What short-term outcomes did G3 achieve both before and after the 2012 change of government? Outcomes disaggregated by before and after the 2012 change of government. KIIs with MSI; KIIs with public officials, CSOs, USAID officials and other donors. Background documents received Qualitative analysis of interviews; Developing case studies of successful or unsuccessful interventions where possible; Document review How have G3 efforts in e￾governance advanced transparency and effectiveness in the agencies targeted for assistance? How have G3’s efforts in advancing freedom of information compliance and facilitation of OGP Action Plan implementation process contributed to Amount and type of information available online (budgeting, expenditure, and procurement activities) Quality of information18 Pertinence and timeliness of information Level of ease in public access to information % of requests for information responded to; Difference CSOs that are part of the FOIA implementation committee (TI, GYLA, Jump Start, IDFI) and Open Government Program initiative; journalists Targeted gov. agencies (MoJ, GoARA) FGD with CSOs, students and journalists KIIs with central government officials and GoARA Institute for the Development of Freedom 2012 and 2013 data base on local governments request for info and number responded to; Qualitative analysis of interviews and FGD; quantitative analysis of FOIA data. 18 We will measure quality through the opinions of key stakeholders. In the focus groups with CSOs, the quality of information shared from the government will be measured through the opinion of those who have used or requested the information. Some of related questions from the FGD guide that will serve to measure quality include: a.. How good is the quality of the information shared by the government? b. Is it recent data? c. Is it reliable? d. Is it in a format that you are able to use for monitoring or research purposes? 53 Research Question Indicator Target Interlocutors Data Source Methodology transparency, openness or responsiveness of the targeted GOG institutions? between 2013 and 2012; Comparison of target municipalities to untargeted municipalities. MoJ and GoARA opinion of change in transparency, openness and responsiveness Use of the information to improve effectiveness of agencies (i.e. improve service delivery) On the local level, what contributions have G3 grants to regional media outlets and local NGOs made to citizen awareness and engagement related to transparency and accountability of local government? Quality of coverage of local affairs on local TV Power dynamics within CACs (how much are CSOs able to contribute and be heard) Level of activity of CACs Degree feel that civil society has a voice and is heard by local government CSOs and media operating at the local level Members of Civic Advisory Councils (CAC) Members of local governments KIIs with members of local governments; FGD with CSOs operating at local level and Members of CAC KIIs with some and FGD with other NGOs that received grants. Qualitative analysis of interviews and FGD; Which of the selected interventions/approaches are recommended for future programming? Sustainability of successful interventions Need for continued support Factors that led to successful interventions Factors that led to unsuccessful interventions Donor interests MSI and other implementation partners; government officials, CSOs engaged in FOIA and OGP Donors (GIZ, UNDP, NATO, European Union) KIIs with MSI; government officials, CSOs, USAID officials and other donors Qualitative analysis of interviews; Developing case studies of successful or unsuccessful interventions where possible; Document review; expert analysis and opinion 54 Research Question Indicator Target Interlocutors Data Source Methodology 2. Effectiveness of Government Mechanisms for Public Engagement. G3 has worked to build public engagement mechanisms at the national and local levels through direct work with government institutions and through/with CSOs. To what degree has the G3 approach been successful in building and sustaining effective mechanisms for public engagement at the national (GoARA, CSPA, MOJ) and local levels? On the municipal level, have partner local governments institutionalized mechanisms of public engagement (public hearings on budgets and other major functions, public calendars, bulletin boards, press releases, display boards or other fora for public hearings)? Number of public events that have taken place using the new mechanisms Regularity of events Mechanisms that worked and those that did not and why Contextual factors that led to effective mechanisms Regularity of events Mechanisms that worked and those that did not and why Contextual factors that led to effective mechanisms Central, regional and local government officials. CSOs that operate at the national and local level Members of CACs NGOs that received grants under G3 KIIs with central, regional and local government FGD with CSOs, students and journalists in Tbilisi and Batumi Reports KIIs with some and FGD with other NGOs that received grants under G3 and CAC members KIIs with members of the Advisory Council on Local Self-Government reform Reports Qualitative analysis of interviews and FGDs Quantitative analysis of PMEP data Document review Qualitative analysis of interviews and FGDs Quantitative analysis of PMEP data Document review 3. Sustainability and Impact of Capacity Development for Government Officials. G3 provided a variety of capacity development activities to government officials on public policy development and drafting, monitoring and evaluation, strategy development, improving inter- and intra-agency communication, human resources management, and program budgeting at the national and local levels. Which G3 training and capacity building efforts within national and local Capacity area that has improved the most (public policy development and drafting, M&E, Central, regional and local government officials; trainers; Interviews with counterparts, USAID, Qualitative analysis of interviews and document 55 Research Question Indicator Target Interlocutors Data Source Methodology level government institutions have been most effective? strategy development, communication, HR, program budgeting) trainees who received capacity building under G3, MSI staff, USAID, MSI staff KIIs with capacity building providers KII with GoARA on program budgeting capacity building; KII with PMO and CSB Survey of trained national and local government officials who received capacity building review Quantitative analysis of survey and PMP data To what extent have targeted institutions sustained results of G3 interventions and, when they did, what were the key factors that contributed to this? Retention of staff that was trained Key factors for successful sustained capacity building Risks to sustainability Government officials ; research institutions KIIs with government officials; survey of national and local government officials who received capacity building Quantitative analysis of survey data Which training and capacity building approaches would be most useful to adopt for future efforts to sustainably improve the capacity and skills of government officials? Current capacity building needs Most urgent capacity building needs Best mechanism for capacity building (training, mentoring, coaching) National and local government officials MSI staff in charge of capacity building Capacity building implementers Survey of national and local government officials who received capacity building KIIs with government officials; MSI KIIs with capacity building implementers Quantitative analysis of survey data Qualitative analysis of KIIs 56 ANNEX 3: LIST OF PERSONS INTERVIEWED 57 US GOVERNMENT OFFICIALS USAID/Tbilisi 1. Keti Bakradze, Senior Civil Socierty & Media Advisor, D&G 2. Lela Kerashvili, Program Development Specialist, D&G 3. Khatuna Khvichia, Project Management Specialist, D&G 4. Kirsten Michener, Senior Democracy & Governance Advisor, D&G 5. Lina Panteleeva, Project Management Specialist, D&G 6. David Stonehill, Deputy Director, D&G US Embassy/Tbilisi 1. Amy W. Diaz, Political Officer, 2. Maia A. Lyons, Political Officer USAID IMPLEMENTERS/PARTNERS 1. Maris Mikelsons, COP, G3, MSI 2. Sofia Gurgenidze, Central Government Component Head, G3, MSI 3. David Mikeladze, Local Government Component Acting Head, G3, MSI 4. Tengiz Chumburidze, Monitoring and Evaluation Specialist, G3, MSI 5. Vakhtang Gordeladze, Ajara Regional Coordinator, G3, MSI 6. Lika Chakhunashvili, Deputy COP, G-Media, IREX 7. Kakha Potskhishvili, G3 urban planner 8. Tiniko Khanjaliashvili, G3 former Coordinator for Telavi and Marneuli 9. Mariam Gorgadze, G3 Former Local Government Component Deputy Head 10. Tamuna Abdaladze, G3 Regional Coordinator for Kutaisi and Ozurgeti 11. Ketevan Tsibadze, CCE Coordinator, Kutaisi office 12. Kate Bibileishvili, CCE Deputy Coordinator, Kutaisi office 13. Zurab Tsertsvadze, CCE Coordinator, Batumi office 14. Teona Diasamidze, CCE Deputy Coordinator, Batumi office OFFICIALS OF THE CENTRAL GOVERNMENT OF GEORGIA 1. Irina Aghapsihvili, Senior Analyst at Civil Service Reform and Development Dept., Civil Service Bureau 2. Giorgi Dididze, Deputy Head of Reforms and Innovations Department, MRDI 3. Tengiz Gvelesiani, Head of Analytical Department, MOF 4. Tamar Jinchveladze, Head of HRM Department, Ministry of Justice 5. Maka Kurtanidze, Deputy Director, Civil Service Bureau 6. David Marghania, Director of IT Department, CSPA 7. Ketevan Samkharadze, Analyst of Civil Service Reform and Development Dept., Civil Service Bureau 8. Tengiz Shergelashvili, Deputy Minister, MRDI 9. Maya Tskitishvili, Head of Chancellery 10. Tamta Tsotskhalashvili, Head of Civil Service Reform and Development Dept., Civil Service Reform 11. Tato Urjumelashvili, Chairman, CSPA OFFICIALS OF THE GOVERNMENT OF THE AUTONOMOUS REPUBLIC OF AJARA 1. Raul Abuladze, Deputy Chairman, Ministry of Tourism and Resorts 2. Mamuka Berdzenishvili, Deputy Chairman, Department of Tourism and Resorts 3. Ramaz Bolqvadze, Deputy Minister, Ministry of Finance & Economy 58 4. Irakli Goradze, Head of International Affairs Department 5. Parmen Jalagonia, Head of Legal Department 6. Kakhaber Mekvabishvili, Head of IT Department 7. Ramaz Meskhidze, Head/IT Department, Ministry of Health 8. Natia Sirabdze, Head of Media and Public Relations Department 9. Nugzar Surmanidze, Minister of Health & Social Affairs LOCAL GOVERNMENT OFFICIALS Telavi 1. Dimitri Bochoidze, Telavi Council Chair 2. Tariel Galakhvaridze, Head of Apparatus, Telavi Council 3. Guram Papalashvili, Waste Management Department, Telavi Municipality 4. Alexander Tegerashvili, Former deputy Head of the Telavi Municipality Rustavi 1. Leila Aptsiauri, CAC Head, City Council Deputy Chair, Rustavi 2. Zaqro Datuashvili, Local Fee Department, Rustavo 3. Kakha Gurgenidze, Rustavi Council Chair 4. Khatuna Mishveladze, Head of Urban Planning and Transport Department, Rustavi 5. Giorgi Tabatadze, 1st deputy Mayor, Rustavi 6. Amiran Tsandishvili, Head of Property Management and Economic Issues Commission 7. Imeda Vardzelashvili, Head of Economics Department Kutaisi 1. Malkaz Chrelashvili, Formal Head of Economic department, now at Procurement Department, Kutaisi 2. Merab Meshveliani, Deputy Chair of Kutaisi Council Batumi 1. Tite Aroshidze, Deputy Head of economic Policy Department 2. Irakli Chavleishvili, Batumi Council Chair 3. Lasha Nakashidze, Head of Department for Strategic Planning, Investments and Economic Development, Batumi 4. Zurab Pataraia, Deputy Mayor of Batumi 5. Gia Ramishvili, the Lead Architect at the City of Batumi 6. Archil Vanadze, Head of Finance Department CIVIL SOCIETY ORGANIZATIONS Tbilisi 1. Eric Barrett, Executive Director, JumpStart 2. Florian Biermann, Assistant Professor of Economics, ISET 3. Sopho Chareli, GYLA 4. Lasha Gogidze, Senior Analyst/Project Coordinator, Transparency International 5. Tamar Gzirishvili, Researcher, Transparency International 6. Giorgi Kldiashvili, Director, Institute for Development of Freedom of Information (IDFI) 7. Eric Livny, MA Program in Economics Policy Institute, ISET, Director 8. Eka Ruadze, GEA Program Manager for G3 project in Marneuli, Gori and Poti 9. George Tsimintia, Head of the Board, Association of Young Economists of Georgia (AYEG) 59 Regions 1. Natia Abkhazava, Civil Society Institute (CSI), Batumi 2. Zviad Devdariani, CiDA, Rustavi, Director 3. Tamar Gvinianidze, Rioni TV, Director, Kutaisi 4. Nata Imedaishvili, Radio Freedom, Batumi 5. Paata Kldiashvili, The Association of Young Economists of Georgia, Head of Local Branch, G3 project Manager 6. Temur Lomsadze, NGO “For Universal Declaration of Human Rights”, visiting CCE 7. Gocha Maisuradze, Political Party Free Democrats, visiting CCE 8. Maya Merkviladze, 25th Channel, Batumi 9. Irakli Murghulia, CSI, Batumi urban planning, Team Leader 10. Sopho Pataraia, Civil Society Institute (CSI), Batumi 11. Malkhaz Sekhniashvili, Project Director, Tanamgzavri TV 12. Nino Tavalalashvili, GYLA, Batumi ACADEMIA & PRIVATE SECTOR: PROGRAM SERVICE PROVIDERS 1. Temur Bolotashvili, Ltd Art Studio Project, Rustavi Urban Planning, Legal Consultant 2. Nino Dolidze, Head of Public Policy Program, Georgian Institute of Public Affairs (GIPA) 3. Eka Gvinjilia, PMCGroup 4. Keti Jakeli, MSPA, Executive Director 5. Irina Khantadze, Executive Director, Center for Training & Consultancy (CTC) 6. Vazha Kukhianidze, Ltd Art Studio Project, Rustavi Urban Planning, Architect 7. Lasha Meskhia, PMCGroup 8. Davit Melua, NALAG, Secretary General 9. Dr. Eka Metreveli, Research Fellow, Georgian Foundation for Strategic and International Studies (GFSIS) 10. Gogi Sulaberidze, Ltd Art Studio Project, Rustavi Urban Planning, Head Architect 11. Vano Tavadze, PMCGroup INTERNATIONAL DONORS/ORGANIZATION 1. Joern Philip Eichler, Team Leader, GIZ 2. Irakli Kobakhidze, UNDP, Advisory Council member at MRDI 3. Archil Zhorzholiani, Advisor, GIZ INTERVIEWS ON SPECIFIC TOPICS CONDUCTED OVER THE TELEPHONE Web Development/IT and PR/Communication Trainees 1. Shota Abjandadze, IT Specialist, Kutaisi 2. Tamar Abuladze, Head of Administration, Akhaltsikhe 3. Tamar Akubardia, Head of Department, Rustavi 4. Nestan Archvadze, Press Department/Web Editor, Rustavi 5. Shorena Chokoraia, Poti City Council 6. Maia Ghurtskaia, Zugdidi 7. Ketevan Grdzelishvili, Kutaisi City Council 8. Natia Khachiuri, Head of Information and Protocol Department, Gori 9. Archil Mchedlishvili, Administrative Service Specialist, Telavi 10. Tamuna Sulkhanishvili, Marneuli 11. Nana Tavdumadze, Ozurgeti City Council Other 60 1. Diana Chachua, Media Analyst, TI Georgia 2. Girogi Girkelidze, Ozurgeti, Newspaper & Internet media Guria News 3. Kakha Jamburia, CSRDG, Telavi CSO grantee 4. Eliso Janashia, Poti, Chief Editor of www.tspress.ge – internet media 5. Kakha Lomadze, Ozugeti, Student Youth Council 6. Iza Salakaia, Poti, Journalist, worked at TSPress before 61 ANNEX 4: LIST OF DOCUMENTS REVIEWED 62 Assessment of GoARA E-Communication System DevelopmentReport Comparative study of FOI legislation of the US, Estonia, United Kingdom (Scotland), and Georgia CSPA DRB – Communications CSPA DRB Business proccesses review_final_04 10 2012 ENG - Final CSPA_Communication Strategy Decree of the Governemnt of Georgia 219 ENG Decree on GoARA Admin HRM Policy Approval Decree on GoARA PB_PR_E-comm Policies Approval Decree_Proactive Description of Procedures at SCPA Eighth Quarterly Report Final Report - CSPA Watchdog FOI Statistics Resume ENG G3 Contact List Evaluation G3 Fifth Quarterly Report 2012 G3 PMEP 1-24-13 Revised V2 G3 Seventh Quarterly Report 2013 G3 Sixth Quarterly Report 2012 GoARA E-Communication System Policy Document GoARA HR Manual GoARA PR PLAN 2013 GoARA-HRM Policy GoG Civi Servants Training June_2013 GoG Civi Servants Training Report_September_2013 GoG Civil Servants Training Report_August _2013 GoG_PR_Need Assesment Report HRM Manual for the Department of Tourism and Resorts of GoARA Human-Resource-Management￾Manual_GoG Manual of effective PR technologies for GoG institutions - EN Mission 1 Report Thomas Hart Mission 2 Report Thomas Hart Mission 3 Report Thomas Hart MOES_Recommendation Paper on Methods of Including Performance Appraisal into HR software MSPA, Service Improvement Plan for the City of Rustavi Municipal Service Providers Association (MSPA), Progress Report on G3 Activities 63 Ninth Quarterly Report PCMG, Training for Local Self-Government Management and Governance, Final Report, June, 2013 PMCG, Training Modules and Presentation Materials for modules on (1) municipal web portals, (2) media relations,(3) Asset Management, (4) State Procurement, and (5) Internal Audit Policy Drafting Training Report – MSI PR Training Report Report_Civil Servants Trainings Report_CSPA Dispute Resolution Board Singh, Rupinder, Assessment of Fiscal Decentralization, European Union, March, 2009 Study on Enhancing transparency and accountability of the CSPA Summary of the Analysis of the Best Provisions of FoIA Tenth Quarterly Report UNDP Georgia, Training and Development Framework in Support of Local Self-governments (Unpublished draft) Year Three G3 Work Plan (1) ინფორმაციის თავისუფლება - გზამკლევი პროგრამული ბიუჯეტირების სახელმძღვანელო Error! Not a valid link. 64 ANNEX 5: ILLUSTRATIVE QUESTIONNAIRES FOR KEY INFORMANT INTERVIEWS AND FOCUS GROUP DISCUSSIONS 65 Focus Group Recruitment Script Hello, my name is . I am calling you on behalf of the Mendez-England and Associates Evaluation Team of a USAID-funded Good Governance in Georgia (G3) program that began in February, 2011. I would like to invite you to participate in a focus group lasting about 1 hour in _[location] to obtain opinions from civil society about the amount and quality of information shared with the public by the government and the level and quality of the interaction of civil society with local government. The information gathered in these focus groups will be used to evaluate the success of the USAID G3 program. Your opinions are valuable to help USAID improve their support to civil society and to local and national government in Georgia for future programming. This evaluation is funded by USAID. You have been selected to participate in this focus group because we believe you are an important representative of civil society. Your participation in the focus group is voluntary and you may choose to end your participation any time without penalty. Your identities will be kept confidential. We would like to compensate you for any travel expenses to reach the focus group location and thank you for your time by providing you with a telephone card worth XX lari. If you have any questions about the evaluation, you can reach Giorgi Giorgadze at [phone number]. Would you be available on XX date at XX time? Focus group consent script Thank you for joining us today. I would like to remind you that we are conducting this focus group in order to obtain your opinions about the amount and quality of information shared with the public by the government and the level and quality of the interaction of civil society with government. The information gathered in these focus groups will be used to evaluate the success of the USAID G3 program. Your opinions are valuable to help USAID improve their support to civil society and to local and national government in Georgia for future programming. This evaluation is funded by USAID. Your participation in the focus group is voluntary and you may choose to end your participation any time without penalty. We will be audio and video recording the session. This is only so that we can accurately report your feedback and opinions. These recordings will be destroyed at the end of the study and will not be shared with anyone outside the research team. Your identities will be kept confidential and data will be reported in a manner to protect your identity. Your participation in the focus group is voluntary and you may choose to end your participation any time without penalty. If you have any questions about the evaluation, you can reach Giorgi Giorgadze at [phone number]. May I begin the session? Student and Media Focus Group Guide (National Level) Target: Students, journalists, NGOs that operate at the National Level (Tbilisi) and Republic of Ajara level (Batumi) Transparency and Accountability 1. What type of government information is of interest to you ? (warm up) 2. How difficult is it to obtain information about government budgeting, expenditures, and procurement or other matters of importance to you? a. Where do you obtain it from? b. How much information is available online? c. What hinders/enables you to obtain this information? 66 3. Do you feel sufficient information is made public related to the a. procurement process? b. expenditure of public funds? c. hiring procedures for public service employment? 4. Has this changed in the last couple of years? a. since Feb. 2011, the start of the G3 program? b. since the recent change in government (2012)? 5. How good is the quality of information made public by the government? a. Is it recent data b. Is it reliable c. Is it in a format that you are able to use for monitoring or research purposes? 6. How have you used this information? a. (some examples might be public expenditure monitoring, watchdog functions related to procurement for large government contracts) 7. Are you aware of citizens’ right to access information based on the Freedom of Information Act? a. Have you ever actively tried to request information from a national or local public agency? [In Batumui] the Government of Ajara? b. If yes, did you receive a response in a timely manner? Did you receive data of high enough quality to use? c. Do you feel there a need to improve legislation and regulations in respect to Freedom of Information? 8. What recommendations do you have for improving government transparency and accountability regarding… a. Information public agencies make readily available (without being asked)? i) Specifically information provided on the internet b. Responding to requests for information? c. Communication through the media? Public Engagement 9. How do you feel about the quality of citizen engagement from the government? a. Have you ever attended a public discussion of the budget? Are they well attended and worth the trouble? b. Does your local government hold public meetings and seek citizen input in other important policy and planning decisions? 10. Overall, what is the current status of civil society’s relationship with the (local/national) government? a. Do you feel that the government is engaging with you on important policy issues? b. Do you feel that your voice is heard? 11. What other recommendations do you have for improving government’s public engagement? 67 Focus Group Recruitment Script Hello, my name is . I am calling you on behalf of the Mendez-England and Associates Evaluation Team of the USAID-funded Good Governance in Georgia (G3) program that began in February, 2011. I would like to invite you to participate in a focus group lasting about 1 hour in _[location] to obtain your opinions and feedback about the Community/Civic Advisory Councils developed by the USAID G3 program and the interaction of civil society with local government. The information gathered in these focus groups will be used to evaluate the success of the USAID G3 program. Your opinions are valuable to help USAID improve their support to civil society and to local and national government in Georgia for future programming. This evaluation is funded by USAID. You have been selected to participate in this focus group because we believe you are an important representative of civil society. Your participation in the focus group is voluntary and you may choose to end your participation any time without penalty. Your identities will be kept confidential. We would like to compensate you for any travel expenses to reach the focus group location and thank you for your time by providing you with a telephone card worth XX lari. If you have any questions about the evaluation, you can reach Giorgi Giorgadze at [phone number]. Would you be available on XX date at XX time? Focus group consent script Thank you for joining us today. I would like to remind you that we are conducting this focus group in order to obtain your opinions and feedback about the Community/Civic Advisory Councils developed by the USAID G3 program and the interaction of civil society with local government. The information gathered in these focus groups will be used to evaluate the success of the USAID G3 program. Your opinions are valuable to help USAID improve their support to civil society and to local and national government in Georgia for future programming. This evaluation is funded by USAID. Your participation in the focus group is voluntary and you may choose to end your participation any time without penalty. We will be audio and video recording the session. This is only so that we can accurately report your feedback and opinions. These recordings will be destroyed at the end of the study and will not be shared with anyone outside the research team. Your identities will be kept confidential and data will be reported in a manners to protect your identity. Your participation in the focus group is voluntary and you may choose to end your participation any time without penalty. If you have any questions about the evaluation, you can reach Giorgi Giorgadze at [phone number]. May I begin the session? CSO Focus Group Guide (Local Government) Public Engagement Target: Members of the CACs and CSO grantees CACs 1. How was your Community/Civic Advisory Council created? a. Where did the initiative come from? b. What was the level of interest of the mayor (gamgebeli) in creating a CAC? 68 i. NOTE: Listen for if mayor was genuinely interested of cooperating just to comply with G3 requirements c. Who from the local community was invited to join? Was everyone invited or only select groups? i. NOTE: Listen to determine if recruitment was biased towards certain groups or others (eg did mayor only want his/her own supporters; does group include genuinely independent members) 2. Did you participate in the NALAG trainings? 3. How functional is your CAC? a. How often does it meet? b. Are CSOs active participants in discussions? c. Are local government officials active and engaged members of the CACs? (probe: how high are the officials, who leads the CAC agenda) d. What types of topics are discussed (are these topics of interest to civil society)? e. Is the CAC consulted prior to government decisions? 4. What has the CAC achieved? a. Are there any specific examples of how the CAC might have influenced the LG’s policy or actions? (compare monitoring budget implementation vs. input in advance to budget plans and other important decisions) 5. How much longer do you believe the CAC will function? a. Once the G3 program is over do you think it will continue? Why or why not? General 6. What are other mechanisms through which civil society regularly engages with the local government? a. Are there public hearings or public meetings to meet with residents? How often do they take place? i. How well attended and useful are Public Discussion on the budget? Are they genuinely participatory? In what ways are they participatory? Do they ever affect the final content of the budget plan? 7. What is the current status of civil society’s relationship with the local government? a. Do you feel that the government is engaging with you on important policy issues? b. Do you feel that citizens’ voices are heard? c. Do you feel you have enough information to be aware of the functioning of the government? 8. What recommendations do you have for improving government’s public engagement (engaging with civil society and taking civil society’s voices into consideration)? 9. Should the G3 program continue, what kind of support related to civil society engagement with the government do you feel is needed? a. What support for government? b. What support for civil society? 69 SEDP 1. Were any of you involved in the development of Social and Economic Development Plans or are aware of them? (If any are aware, continue with these questions) a. Do you feel that citizens were actively engaged in the development of these plans? b. How engaged were municipal (city) leaders? c. Do you feel that they were well done? 2. How was the Social and Economic Development Plan used? a. How much was the document used in the 2013 or 2014 budget preparation process? b. Do you think the document will be updated and used in the future after the G3 project ends? 70 Focus Group Recruitment Script Hello, my name is . I am calling you on behalf of the Mendez-England and Associates Evaluation Team of a USAID-funded Good Governance in Georgia (G3) program that began in February, 2011. I would like to invite you to participate in a focus group lasting about 1 hour in _[location] to obtain opinions from civil society about the amount and quality of information shared with the public by the local government. The information gathered in these focus groups will be used to evaluate the success of the USAID G3 program. Your opinions are valuable to help USAID improve their support to civil society and to local and national government in Georgia for future programming. This evaluation is funded by USAID. You have been selected to participate in this focus group because we believe you are an important representative of civil society. Your participation in the focus group is voluntary and you may choose to end your participation any time without penalty. Your identities will be kept confidential. We would like to compensate you for any travel expenses to reach the focus group location and thank you for your time by providing you with a telephone card worth XX lari. If you have any questions about the evaluation, you can reach Giorgi Giorgadze at [phone number]. Would you be available on XX date at XX time? Focus group consent script Thank you for joining us today. I would like to remind you that we are conducting this focus group in order to obtain your opinions about the amount and quality of information shared with the public by the local government. The information gathered in these focus groups will be used to evaluate the success of the USAID G3 program. Your opinions are valuable to help USAID improve their support to civil society and to local and national government in Georgia for future programming. This evaluation is funded by USAID. Your participation in the focus group is voluntary and you may choose to end your participation any time without penalty. We will be audio and video recording the session. This is only so that we can accurately report your feedback and opinions. These recordings will be destroyed at the end of the study and will not be shared with anyone outside the research team. Your identities will be kept confidential and data will be reported in a manner to protect your identity. Your participation in the focus group is voluntary and you may choose to end your participation any time without penalty. If you have any questions about the evaluation, you can reach Giorgi Giorgadze at [phone number]. May I begin the session? CSO Focus Group Guide (Local Government) Transparency and Accountability Target: CAC members and other CSOs operating at the local level (not Republic of Ajara) (both G3 grantees and non-grantees), journalists, and students. NOTE: Be very aware to make sure to distinguish in the conversation what happened before the 2012 change of government and what happened after the 2012 change of government? 1. What are the activities you’re involved to communicate with the LGs 2. How does the local government make information about its activities available to the public? How has this changed if at all since 2011? 71 a. TV, radio, newspapers, display/bulletin boards, public calendars? 3. How difficult is it to obtain information about government budgeting, expenditures, and procurement? Has this improved since 2011? a. From where do you obtain it? b. What hinders/enables you to obtain this information? 4. How much information about government budgeting, expenditures, and procurement is available online? a. Is the internet a good mechanism for the local government to provide information to the public? 5. How good is the quality of the information shared by the government? a. Is it recent data b. Is it reliable c. Is it in a format that you are able to use for monitoring or research purposes? 6. How have you used this information? a. (some examples might be public expenditure monitoring, to advocate for different budget allocations, etc) 7. What is your opinion of TV programming related to local public affairs? a. Has it changed in the last few years (since Feb. 2011, the start of the G3 program)? b. Is it informative? c. Has it contributed to citizens’ increased knowledge of government activities? 8. What is the current status of civil society’s relationship with the local government? a. Do you feel that the government is engaging with you on important policy issues? b. Do you feel that citizens’ voices are heard? c. Do you feel you have enough information to be aware of the functioning of the government? 9. What recommendations do you have for improving government sharing of information? 10. Should the G3 program continue, what kind of support related to government transparency and accountability do you feel is needed? 72 ANNEX 6: MINI-SURVEY QUESTIONNAIRE 73 Hello, my name is and I am calling you from the research organization, IRMS, which is part of the Mendez-England and Associates Evaluation Team. We are conducting a study of the USAID Good Governance in Georgia (G3) program that began in February, 2011 and is implemented by Management Systems International (MSI) and its partners. We are asking you to participate in a 15-minute survey in order to obtain feedback on your experience with the capacity building activities of the program. This study is funded by USAID and feedback will be used to improve similar USAID government capacity building programs in the future. Your opinions are valuable to us because they will help us obtain feedback on the G3 program and will allow us to communicate your future capacity building needs to USAID. You have been selected for this survey because you participated in a training organized by the program. Your participation in this survey is voluntary and you may choose to skip a question or discontinue your participation at any time. Your identity will kept confidential and will not be shared with USAID. If you have any questions about the survey, you may contact Giorgi Giorgadze at tel: 593 37 12 54, or the Mendez England and Associates Country Director, Marika Gorgadze tel: 577 72 53 12 May I proceed with the survey? Respondent ID Interviewer code: Level of Government 1. Central government 2. Government of Ajara 3. Local government 74 DEMOGRAPHIC INFORMATION ABOUT RESPONDENT Q1. Gender female 1 male 2 Q2. Age Q3. What is your highest level of completed education? [Only one answer] No formal education 1 Kindergarten 2 Elementary school (4-5 classes) 3 Incomplete secondary (5-9 classes) 4 Secondary (10-12 classes including general education, lyceum, gymnasium) 5 Secondary vocational (technical or college) 6 Higher education diploma (Bachelor, Master) 7 Advanced higher education 8 (Don’t know) -1 (Refuse to answer) -2 Q4. Where do you currently work? Government 1 Continue Parliament 2 Q7 NGO 3 Media 4 Academic 5 Consultant 6 Private sector business 7 Unemployed 8 Other, specify 9 (Refuse to answer) -2 Q5. What is your position? 75 [for GoARA or GoG] Ministry [for LG] Organization/Municipality Department Title Q6. For how many years have you been working… In your Current position Overall in the government [Go to Q11] Q7. Were you previously employed in local government or national government? National government of Georgia 1 Government of Ajara 2 Local government 3 No 4 Q11 Q8. For how many years did you work in the government? Q9. What year and month did you leave your last government job? Year Month Q10. What was your last position in government? [for GoARA or GoG] Ministry [for LG] Organization/Municipality Department Title 76 G3 CAPACITY BUILDING RECEIVED I will now ask you questions about the USAID Good Governance in Georgia (G3) program that began in February, 2011 and is implemented by Management Systems International (MSI) and its partners ([for Local Government respondents:]CSI, GEA, PMCG) ([for GoARA and GoG responents]CSI, DPA, GEA, GEPRA, Ikalto, PMCG, IDFI, GIPA, Smart Consulting, Robakidze University) Q11. Did you personally attend any training conducted by this program? If yes, what year did you attend these trainings # Training topic Attended Year Attended Yes No 1 Public Relations 1 0 2 E-Communication Training 1 0 3 Governance 1 0 4 HR Management 1 0 5 Legal Framework of Georgia 1 0 6 Policy Development 1 0 7 Presentation and Writing Skills 1 0 8 Process of Policy Development 1 0 9 Program Budgeting 1 0 10 Project Management 1 0 11 Public Policy 1 0 12 State Procurement 1 0 13 Strategic Planning 1 0 14 Design and Management of Municipal Web Portals 1 0 15 Investors Relations 1 0 16 Managing IT Assessments, Strategies and Capacity Development Plans 1 0 17 Project Design, Management and Evaluation 1 0 18 Skills facilitation 1 0 19 Legal Drafting and Juridical Documentation 1 0 20 Generating Recommendations for Citizen Participation in Local Self-Government Workshops 1 0 21 Civil Service Reform Workshops 1 0 22 Strategic Urban Planning Workshops 1 0 77 23 Action Plan Drafting Workshops 1 0 24 Establishing a PR Workshops 1 0 25 Open Government Partnership Action Plan Workshops 1 0 26 Other(please specify) 1 0 Q12. [GoARA and GoG respondents only] Did you receive any other capacity building support from the G3 program? [check all that apply.] Coaching: frequent contact with someone from the G3 program who supports the thinking or learning process for an individual or group Yes/No Facilitation: discussion facilitated by someone from the G3 program within a group, team, or organization eg. Roundtable with stakeholders on Civil Service Reform, Freedom of Information Act and Open Government Partnership (OGP) Yes/No OUTCOMES OF G3 CAPACITY BUILDING Q13. a. How would you rate your current competence in the following? 1=no competence, 2=little competence, 3=normal competence, 4=expert competence b. How would you rate your increase in competence in the following as a result of the G3 program capacity building activities?1=negligible increase, 2=modest increase, 3=large increase, 4=fundamental increase ; c. How would you asses the quality of training from scale 1 as being easy to understand to 5 – difficult to understand; d. Would you like to receive additional training on this topic? [Only applies to the trainings attended by participants Q11] Skill area a. Competence Rating (1-4) b. Increase in competence (1-4) c. quality d. additional training 1 – Yes 0 - No PublicRelations/Communication E-Communication Training Governance HR Management Legal Framework of Georgia Policy Development Presentation and Writing Skills Process of Policy Development 78 Program Budgeting Project Management Public Policy State Procurement Strategic Planning Design and Management of Municipal Web Portals Investors Relations Managing IT Assessments, Strategies and Capacity Development Plans Project Design, Management and Evaluation Skills facilitation Legal Drafting and Juridical Documentation Generating Recommendations for Citizen Participation in Local Self￾Government Workshops Civil Service Reform Workshops Strategic Urban Planning Workshops Action Plan Drafting Workshops Establishing a PR Workshops Open Government Partnership Action Plan Workshops Other Q14. Has there been an improvement in any of the following for your division due to the G3 program capacity building?[check all that apply] None Yes/No transparency Yes/No efficiency Yes/No quality of service Yes/No timeframe of the service Yes/No increased engagement with civil society Yes/No better communication Yes/No better planning Yes/No better management Yes/No 79 improved knowledge Yes/No other (specify) Yes/No FEEDBACK ON CAPACITY BUILDING Q15. How would you rate each of the capacity building mechanisms for their effectiveness in teaching new skills and building capacity? [For Local Government] How would you rate the G3 program’s training for its effectiveness in teaching new skills and building capacity? Excellent Good Fair Poor Training [ ] [ ] [ ] [ ] Coaching: frequent contact with someone from the G3 program who supports the thinking or learning process for an individual or group [ ] [ ] [ ] [ ] Facilitation: discussion facilitated by someone from the G3 program within a group, team, or organization eg. Roundtable with stakeholders on Civil Service Reform, Freedom of Information Act and Open Government Partnership (OGP) [ ] [ ] [ ] [ ] Q16. (For those rated good or excellent) What is it that made them so effective? Quality of the local expert that delivered the capacity building 1 Quality of the international that delivered the capacity building 2 Content /relevance of the training 3 Other, please specify 4 80 SECTION E: FUTURE PROGRAMMING (For those who answered 1 on Q4 and are still working in government) Q17. a. What are your priority areas of capacity building for yourself for the next five years? Please list. b. What form of capacity building assistance do you think would be the most effective mechanism of delivery? (choose all that apply) Priority areas of capacity building Delivery Mechanism Training Coaching: frequent contact with an expert who supports the thinking or learning process for an individual or group Facilitation: discussion within a group, team, or department facilitated by an expert 1. [ ] [ ] [ ] 2. [ ] [ ] [ ] 3. [ ] [ ] [ ] 4. [ ] [ ] [ ] Q18. If you had to choose one area of capacity building support for you to receive, which would it be and what form of capacity building do you think would be most effective? Area Delivery Mechanism (choose all that apply) Training Coaching Facilitation Q19. Do you have any other comments you would like to share about the G3 program’s government capacity building activities? 81 ANNEX 7: DISCLOSURE OF CONFLICTS OF INTEREST 82 83 Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Peter Eostein Title Organization ME & A Evaluation Position? DTeam Leader D X Team Member Evaluation Award Num ber (contract or other instmmenf) USAID Project(s) Evaluated (Incl:ude project name(s), implementer name(s) and award number(s), ifapplicable) I have real or potential conflicts of interest to disclose. D Yes D X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may incl:ude, but are not limited to: 1.Closefamily member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2.Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being eval:uated or in the outcome of the eval:uation. 3.Current or previous direct or significant though indirect experience with the project(s) being evaluated, incl:uding involvement in theproject design orprevious iterations of theproject. 4.Current or previous work experience or seeking employment with the USAID operating unit managing the eval:uation or the implementing organization(s) whose project(s) are being evaluated. 5.Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project (s) are being evaluated. 6.Preconceived ideas toward individuals, groups, organizations, or objectives of theparticular projects and organizations being eval:uated that could bias the eval:uation. NOTE: I do not believe I have any real or potential conflicts of interest. However, in the interest of full disclosure: 1 I learned on November 26 (two days ago) that the G3 CoP is someone who was a colleague of mine at the Urba n Institute from 1998 to 2004 where we worked together on some projects, To the best of my recollection we have not had any communication since that time-nearly 10 years ago. 2 I have been previously employed by the Urban Institute, ICMA, and Chemonics all of whom may compete for some projects with MSI. I am currently self-employed. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. IfI gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains propne. ctarv and refram. from usm. g the m. f,ormation f(or any purpose other than that f,or which1.t was furm. shed. Signatu re Date November 29, 2013 Peter Epstein 40 84 85 86 ANNEX 8: G3 GRANTEES MAJOR ACTIVITIES 87 City CSO Media 1. Rustavi Civil Development Agency (CiDA) Initial grant, March to Dec. 2012 One of two grants extended through 2013 ‐   Initial activities:  Improved city website in consultation with stakeholders;  brochure on Your City Government and You prepared and disseminated;  developed Citizens Electronic Monitoring System (CEMS),  support for CAC (founded May 2012) especially in budget process. ‐    In 2013 ‐   used CEMS for quarterly budget input ‐   Initiated replication of CEMS in Poti, Kutaisi, Gori. N/A 2. Batumi Civil Society Institute (CSI) Batumi branch April 2012 and extended to Dec. 2013 ‐  Dynamic CSO leader ‐   Supported post-election, reconstitution of CAC and 13 district councils ‐  Most active CAC with average of three meetings per month ‐   Proposals to council on waste management, tariffs, and handicapped access ‐  Budget discussions in 13 territories ‐  Created citizen web portal for CAC www.marte.ge ‐   developed TV spot on budget process ‐  CAC contributed to reduction in waste collection tariff for legal entities Channel 25. September 2012 to September 2013 ‐   12 direct call-shows ‐   12 educational telecasts ‐   86 coverage reports ‐  Unique feature: officials answer citizen questions live, face-to face, in studio ‐  Contributed to reduction in waste collection tariff for physical entities 3. Telavi “Center for the Strategic Research and Development of Georgia” (CSRDG) April 2012 to April 2013 ‐  SOW:  CAC assess and strengthen  improve website  educate petition and budget process ‐  Result: MSI assess concluded weak result in part poor CAC membership. Failed to make city website operational. MSI did not recommend continuing Tanamgzavri TV July 2012 to July, 2013 ‐   11 call-in talk shows ‐   11 educational programs ‐   55 reportages on LG affairs in news programs ‐   live streaming of 11 sakrabulo meetings ‐   60% growth in visitors to TV station’s website. 4. Akhaltsik he United Democrat Meskhs (UDM) May 2012 to May 2013. ‐   SOW:  CAC establish/strengthen  public awareness through media Imperia / Channel 9 (TV) May 2012 – May 2013 ‐   Promo for talk shows ‐  Reportages on LSG activities in the 88  hearings and public events calendar ‐  Result: MSI assess semi positive. CAC continuing with new Sakrabulo support; Budget input; weekly live web forum – was not feasible, thus activated municipalities’ existing Facebook; some media input. ‐  USAID turned down continuation cause of discontinuity. news program and TV program ‐  13 call-in talk shows on LSG issues with participation of LG representatives, CSOs and population 5. Kutaisi Association of Young Economists of Georgia (AYEG) May 2012 to May 2013 ‐  SOW:  Info material, including budget calendar and process  Hearings on budget and services survey  two town hall meetings  Two citizen report cards  TV company “Rioni,” and newspaper “Akhali Gazeti—semiannual budget implementation.  CAC strengthen. ‐  Result:  Did what promised.  Created AYEG Facebook page  4,5 of 10 written budget recommendations adopted  AYEG not in CAC. ‐  Grant extension turned down by USAID cause of political turnover TV & Radio Company Rioni April 2012 to April 2013 ‐  61 reportages – 11 in call-in shows; 50 in news programs on LSG issues ‐  12 call-in shows (11 live; 1 recorded finale call-in show) on LSG and citizen engagement ‐  11 viewer surveys to improve programming ‐  Recommendations (with G3’s CSOs grantees) to LSG based on analysis of citizens’ feedback from call-in shows ‐  Four 15 minutes quarterly TV & radio programs “Hour to Self￾Governance”- raising public awareness ‐  Promoting CAC and other G3 activities in TV 6. Gori, Marneuli and Poti Georgian Evaluation Association (GEA) July 2012 to July 2013 ‐  SOW:  Develop CSO capacity for CAC participation  Institutionalize participatory planning,  Citizen awareness  CAC and public hearings ‐  Result:  MoU among lead CSOs for cooperation  training on Advocacy Capacity Building for the CSO and Media  Quantitative and qualitative measures for monitoring budget hearings  900 brochures on municipality responsibility (300 in each town)  CAC meeting and training for members  Calendars of CAC and Sakrabulo meetings.  Article in newspapers - Azeri translation for Marneuli Marneuli TV July 2012 – June 2013 ‐  11 monthly, live call-in talk-show program on LSG issues in dialogue with citizens ‐  Pre-recorded educational-cognitive telecasts once per month ‐  Training for TV journalists on LSG functioning; approaches for engaging TV viewers ‐  Group of 10 volunteers (LG officials, NGOs, citizens) monitoring TV programs and talk￾shows ‐  Uploading all reports/programs/call-in talk shows on TV website Results: Found no reports on training results and 10 monitors’ activities 7. Zugdidi Association of Disabled Women and Mothers of Disabled (DEA) July 2012 to April 2013 ‐  SOW:  Train CAC and council in SEDP Poti. 9th Wave Broadcasting Company. May to May Reads as if most productive. City 89  Youth meeting with Sakrabulo  Project activities in Radio Atinati and Zugdidi Gazeti  Needs assessment CAC and city officials in 20 communities and village trustees  5 meetings at high school and 2 in university with youth on their concerns  Collected 1300 signatures petition to create youth council Solved problems raised ‐  96 reportages - 8 per month, including LSG re work – sessions, budget hearings, etc. ‐  24 live call-in shows - one for Sakrebulo, one - for Mayor’s office activities; ‐  12 monthly educational-cognitive telecasts ‐  36 promotional ads on planned TV programs ‐  Uploading programs/viewer feedback on the TV website and publicizing them via Facebook and Twitter. ‐  Results: citizen informed authorities about damaged road in live shows, solving this problem soon after; problem with canalization reported in live was solved; government stopped building of Methanol plant in Poti after protest; in Maltakva, near Poti, building of Gas station stopped after reporting population’s protest. 8. Ozurgeti Democratic Development Union of Georgia (DDUG) May 2012 to May 2013 ‐  SOW:  involvement in 5 territories, awareness on social assistance and village support  monitoring social assist budget  Published results in paper  City cooperated  CAC functioning  Council meeting in one village  24 articles in 17 issues of Guria News (partner for grant) on council actions  Budget calendar Broadcasting Company Guria (TV) May 2012 to May 2013 ‐  62 reportages on LSG events ‐  12, 45-minutes topical monthly call-in shows ‐  12 monthly educational telecasts aimed at raising public awareness ‐  With CSOs conducting quarterly TV viewer surveys to improve future programming and citizen￾engagement mechanisms of LSG Source: G3 internal Report on Media & CSO Grants, Dec. 16, 2013, and quarterly progress reports supplemented by KII interviews in four partner cities/municipalities. 90 ANNEX 9: ADDITIONAL TABLES 91 Table 1 Table 1. Leadership Changes in G3 partner LSGs since October 2012 LSG Month Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov SGC Kutaisi Mayor Council Chair Mayor Mayor Mayor Mayor Mayor SGC Poti Council Chair Mayor SGC Batumi Council Chair SGC Rustavi Mayor Mayor Ozurgeti Mun. Head Council Chair Zugdidi Mun. Head Gori Mun. Head Council Chair Akhaltsikhe Council ChairMun. Head Mun. Head Marneuli Mun. Head Telavi Mun. Head Mun. Head Source: Information provided by MSI staff, selectively confirmed through KII interviews. 92 Table 2. Review of Local Government Websites that Received G3 Support The Evaluation Team interviewed individuals responsible for websites updates as well as those who had attended trainings on design and management of municipal web portals. Several municipalities do not have an IT or web developer on staff and sent external consultants to the training. Major questions: 1. Did you attend PCMG training? 2. Whether and in what ways did G3 contribute to webpage development? 3. Why are G3 materials not updated on the web (SEDP, CAC, and SUP& SIP where applicable)? Chart #1 Municipality Were you involved (with LG) from beginning of G3. If not, when did you join? Usefulness of training re: (1) improving web page functionality? Usefulness of training re: (2) improving content, (3) re: proactive disclosure info? Have you improved your web page since G3 started? How? Are changes in response to G3 training or would they have happened anyway? Telavi No. Joined in June 2013 NA NA YES No connection to G3 Rustavi YES YES, adding few modules and making some information more visible by posting them on the front page NO YES, Some info became more visible and technically improved YES, to some extent, not radically Kutaisi YES YES, some categories has been added – rules of posting auctions YES, updates more frequently; Facebook page added YES, became more often updated, informative and user friendly. Yes, to some extent. Gori YES YES, changing location of information; finding lost information Akhaltsikhe YES YES, non￾functional web page became functional NO YES, improved, not significantly but just technically NO YES, G3 training stimulated municipality’s desire to make web page functional YES, because of G3. YES, gave a push to improvement process Zugdidi YES NO, before the training they have developed new webpage YES, more information was added – legal documents, updates, calendar, etc. YES, G3 training stimulated increase of content on the web YES, in terms of better and more often updating information Poti YES Interesting Generally, was interesting NO, she said that she works there for years and has NO, they were doing it for years 93 Municipality Were you involved (with LG) from beginning of G3. If not, when did you join? Usefulness of training re: (1) improving web page functionality? Usefulness of training re: (2) improving content, (3) re: proactive disclosure info? Have you improved your web page since G3 started? How? intensive experience. They update legislative acts regularly. Are changes in response to G3 training or would they have happened anyway? Ozurgeti YES Interesting Generally improved skills, not really using NO. They were doing all the necessary things even before – posting info on FB, on web, etc. NO, were doing without training as well Telavi: Although the G3 sub-grantee in Telavi had to improve the website of the municipality, CSRDG was not able to make its web-page operational. The main web-page person did not attend the G3 training. He is just a citizen of Telavi. He approached the Local Council and with the help of his friend, a Council member, and met with the Council Chair and voluntarily suggested to create a new website for the municipality. At that time, the Telavi web-page was not functional. He developed a new, functional website in July 2013 and is still continuing work on it. Later, he became a staff member of Gamgeoba as an Administrative Service Specialist. So, as he said, G3 did not contribute to website development at all; this guy is ready and happy to cooperate. I talked to a PMCG training participant as well who confirmed the abovementioned information and said that she is not a technical person, and that, for her, although the training was not very relevant and she did not use it in her work, it was nice to attend the training. Rustavi: The main website person attended a PMCG training on webpage development. She said that based on the training, they made minor technical changes to the website, although not really noticeable from outside, such as how to format information, etc. More changes were done with the help of CiDA – a Local Grantee. These changes included adding a few modules and making some information more visible by posting them on the front page. Kutaisi: After the PCMG training, the website is updated more frequently, some categories have been added to the site, the method for posting auctions was changed, and a Facebook page was created. Gori: The main web-page person attended a web portal training said that the training was very useful, but not enough for novices like her. Although she works in the information and protocol department and is responsible for website administration, she is not skilled enough in technical issues. (The staff only has an IT person helping to solve computer technical problems but not someone who is specialized in web development.) She assessed the PMSG training and trainer as very good. The trainer provided guidance after the training beyond its scope and helped find information that was after being uploaded on the website (a technical problem), since contact was lost with the company responsible for municipality website development. PMSG also changed location of some windows. Therefore, the training was really helpful. Akhaltsikhe: The head of the administration and a media person attended the web training and encountered the same problem as in most municipalities – they did not have a technical person responsible 94 for their website. Further, before the PMCG training, the website was not even functional. After the training, the necessity of a web-page was realized and so a new staff member, a journalist/web assistant, was hired to also update the website. In general, the web-pages are not informative but are functioning. There is no contact information and news updates do not include dates. Chart #2 Proactive Disclosure of Information on G3 Partner Websites Telavi Rustavi Kutaisi Mayor/Gamgebeli website http://telavi-gov.ge/ http://rustavi.ge/ http://kutaisi.gov.ge/ Council Website http://telavi-gov.ge/ http://rustavi.ge/ http://kutaisi.gov.ge/ Schedule of Council Meetings YES, it’s on bulletin board along with other announcements YES, it’s on bulletin board along with other announcements Calendar is empty, although there is news updates Agenda for Upcoming Meetings NO NO NO Minutes of Meetings In News Archive: general description of Council meetings In News Archive: general description of Council meetings Very short description, but last update is on December 2012 Schedule for other meetings YES, upcoming meeting announcement YES, upcoming meeting announcement NO Budget Calendar NO NO NO Budget Execution Reports (Prior FY 3rd Q of 2013?) Yes, hard to find – under Sakrebulo’s ordinances YES YES Planned FY 2014 Budget YES YES YES Audit Report for 2012 Budget NO NO NO Tender opportunities YES YES, hard to find. http://rustavi.ge/?page_id=3574& lang=ge NO. Linked to CSPA web-page Tender awards NO NO NO. Linked to CSPA web-page Land/Property Auctions YES YES YES, last doc is from August 2013 SEDP NO YES. One for 2013 and one for 2009 from UNDP. 2009 includes donor logo NO SUP (where applicable) NA NO, although there are news about Art Studio and related events NA SIP (Where applicable) NO NO NO Contact info for officials YES, including cell phones of all officials YES, officials as well as press service, hotline, citizens reception, web master Only hot line number and land line numbers of Mayor’s representatives in territorial entities. NO general contact information Guidance on making inquiries NO YES NO Links to CAC site NO, but there is info about Public Council (CAC) http://telavi￾gov.ge/?page_id=29 including composition and ordinance. NO, but there are news about Public Council (CAC) as well as Public Darbazi NO Link to CEMS (Rustavi, Gori, Kutaisi, Poti) NA YES NO 95 Bulletin Board Updates YES YES Last update was in 2012 Site Map NO YES NO Interactive Tools YES CEMS, Hot line YES Search Engine YES YES YES Kutaisi: The website is not very user friendly. Some information is repeated in different windows. For example, “Government” has three different windows, one which is empty and two which include the same information and “Mayor” is basically empty containing only the structure of the office with short biographies of personnel. Many documents, even one sentence announcements, are attached mostly as word files which are hard to find and not searchable. For example, a citizen’s house was burned down and there is bank account information in case anyone wanted to help financially. Other than front page news, most information on the website has not been updated since December 2012 (the most recent political changes). Even more general information is basically useless. For example, the list of museums does not include information on when they are open. Rustavi: Compared to other websites, Rustavi municipality has more updated and interactive information. A “Question to the Self-government” window features many questions and answers; an interactive question asks people their about opinions about park rehabilitation; announcements are updated regularly, although it is not very easy to find specific information; a bulletin board includes all statements and announcements, although not overly informative. The web-page has a “Proactive Disclosure” page which, although it is difficult to find, includes information, how to make an inquiry, a list of legal acts, budget, and planned tenders. Telavi: The website is not user friendly and has hard-to-find information. However, only Telavi has planned its FY 2014 budget. Chart #3 Proactive Disclosure of Information on G3 Partner Websites Batumi Poti Ozurgeti Mayor/Gamgebeli website http://batumi.ge http://www.poticity.ge http://ozurgeti.org.ge/ Council Website http://www.batumicc.ge/ http://www.poti￾sakrebulo.ge/ http://ozurgeti.org.ge/ Schedule of Council Meetings Yes, but last update is in July 2012 NO NO. There is an announcement, notifying upcoming Council meeting on November 20. Agenda for Upcoming Meetings NO NO NO Minutes of Meetings NO NO NO Schedule for other meetings NO NO NO Budget Calendar NO NO NO Budget Execution Reports (Prior FY? For 3rd Q of 2013?) No, only adopted 2013 budget NO, only adopted 2013 budget NO Planned FY 2014 Budget Yes YES YES Audit Report for 2012 Budget NO YES NO Tender opportunities Yes, on Mayor’s website, but linked to CSPA page. Specific tender announcements are also searchable. In total five in NO NO. The window “Auction￾Tender” includes one announcement from 2011 96 Proactive Disclosure of Information on G3 Partner Websites Batumi Poti Ozurgeti 2010-2013 Tender awards NO NO NO Land/Property Auctions NO – window assists, but is empty. Three tender results are searchable in June-July 2013 Two auctions are searchable in 2011 NO. The window “Auction￾Tender” includes one announcement from 2011 SEDP NO. It is not adopted NO Yes for 2013-2016, also Economic Development plan for 2008-2012 SUP (where applicable) NO, although info on SUP discussion is there NO NA SIP (Where applicable) NA NO NA Contact info for officials Just e-mail addresses for Council members; land line numbers for few deputy mayors Even mobile phone numbers for both – Sakrebulo and Mayor’s office YES, cell phones included Guidance on making inquiries NO NO NO Links to CAC site YES, www.marte.ge is CAC page and it’s lined to Sakrebulo site. Also Council Chair’s Decree on creation of CAC, 27 November, 2012 NO NO, no mention in news as well Link to CEMS (Rustavi, Gori, Kutaisi, Poti) NA NO NA Bulletin Board Updates News feed updates regularly (Geo and English) NO YES, but they are outdated and not informative until opening – all have the same title “announcement” Site Map YES for Council; NO for Mayor’s office NO NO Interactive tools Write an email to Sakrebulo: participate in budget planning; Citizens’ Service Center: hot line Write an email to Sakrebulo; hot line, letter to the Mayor Interactive question, how you like the site Search Engine YES YES YES Batumi: The Mayor’s webpage is somewhat proactive and includes: Participate in Your City’s Budget Planning and Citizens’ Service Center with special forms to apply for different requests. The Sakrebulo page includes writing an e-mail to the Sakrebulo and municipal programs for 2013. However, none of the sites include the most-demanded information. The CAC Facebook page is not part of the Sakrebulo website. Poti: Most of the windows are empty (Press Release, Reports, Legislation includes LSG law only) on both sites. Legal acts are updated regularly, but hard to search and not user friendly. Ozurgeti: The bulletin board includes announcements such as “Council meeting will be conducted….,” which, although already past activities, remain. The last announcement dated November 20, 2014 is still on the front page. 97 Chart #4 Proactive Disclosure of Information on G3 Partner Websites Akhaltsikhe Gori Marneuli Zugdidi Mayor/Gamgebeli website http://www.akhaltsikhe.ge/ http://www.gori￾municipality.ge/ http://marneuli.ge/ http://www.zugdidicity.ge/ Council Website http://www.akhaltsikhe.ge/ http://www.gori￾municipality.ge/ http://marneuli.ge/ http://www.zugdidi￾sakrebulo.ge/ Schedule of Council Meetings NO NO NO NO Agenda for Upcoming Meetings NO NO NO NO Minutes of Meetings NO NO NO NO Schedule for other meetings NO NO NO NO Budget Calendar NO NO NO NO Budget Execution Reports (Prior FY? For 3rd Q of 2013?) NO, just the draft 2013 budget NO, only 2013 budget and amendments to it NO NO Planned FY 2014 Budget NO NO NO, existing page is empty YES Audit Report for 2012 Budget NO NO NO NO Tender opportunities NO, just one announcement for 2010 NO NO, page exists split in ongoing and past tenders, but is empty YES, it is searchable, but not specific page for tenders Tender awards NO NO NO NO Land/Property Auctions YES, just three auctions in 2012 NO NO, just one announcement in News Feed dated by November 22, 2013 NO SEDP NO NO, only in news feed, named as Municipal Development Strategic Plan NO, page exists, but is empty NO SUP (where applicable) NA NA NA NA SIP (Where applicable) NO NO, only news on MSPA presenting SIP on February 20, 2013 NO NA Contact info for officials NO, not a single number. “Contact” is empty YES, only two general numbers and emailing option (with no email address) and cell phones of village governors YES, for Sakrebulo officials, including cell phones; NO for Gamgeoba, only info for village governors Yes, including cell phone numbers for both Sakrebulo and Gamgeoba Guidance on making inquiries NO NO NO NO Links to CAC site NO YES, Public Darbazi – its’ composition and rules of regulations and LoA copy NO NO, in news feed two info Link to CEMS (Rustavi, Gori, Kutaisi, Poti) NA NO NA NA 98 Proactive Disclosure of Information on G3 Partner Websites Akhaltsikhe Gori Marneuli Zugdidi Bulletin Board Updates NO NO NO NO Site Map NO NO NO NO Interactive tools NO NO NO Citizens’ Reception; Write Letter to Sakrebulo Chair Search Engine NO YES YES YES Akhaltsikhe: There is a special window for NGOs with detailed information including contact info, sector of expertise, etc. It is hard to navigate because there is no search engine. The site mostly includes general information about the municipality such as history, agriculture, culture, and sports, etc. The only updates are news updates on the front page, but they lack dates. Gori: This is the only website including all memorandums with donor organizations, including LoA with G3. It contains information about CAC, the so-called public Darbazi. The Window “Contact” is a bit confusing: there are two windows with the same title; from one – more visible one - you get only an emailing opportunity with no email address and no phone numbers; in the other, there are two numbers and an email address available. Marneuli: Information is only in the Georgian language, although a major portion of the local population are ethnic Azeris, who are not fluent in Georgian. In part of Gamgeoba, even names of officials are empty, only village governors’ info is available. Zugdidi: The only updated information is the daily updates of Sakrebulo/Gamgeoba and contact information of all officials.