EVALUATION FINAL REPORT: ENERGY SECTOR CIVIL SOCIETY INITIATIVES IN GEORGIA MAY 15, 2006 This publication was produced for review by the United States Agency for International Development. It was prepared by the Academy for Educational Development. EVALUATION FINAL REPORT: ENERGY SECTOR CIVIL SOCIETY INITIATIVES IN GEORGIA Contract No. DOT-1-00-04-0002-00 Task Order No. 2, Sub-Activity 9 Disclaimer The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. © Academy for Educational Development 1825 Connecticut Ave, NW Washington, DC 20009 (202) 884-8000 (202) 884-8400 www.aed.org Prepared for: Prepared by: USAID/EGAT/I&E/E Academy for Educational Development Version: 1.0 2 Evaluation of Energy Sector Civil Society Initiatives in Georgia Table of Contents EXECUTIVE SUMMARY 5 Background 5 Evaluation Objectives and Methodology 5 CENN Program Evaluation 6 Liberty Program Evaluation 7 Summary of Key Findings and Recommendations 8 SECTION 1. INTRODUCTION 12 1.1 Background 12 1.2 Objectives of the Evaluation 12 1.3 Methodology 13 SECTION 2. CENN PROGRAM EVALUATION 13 2.1 CENN Project Partners 13 2.2 Program Objectives and Activities 14 2.3 Activities Completed by CENN 17 2.4 Evaluation Findings 25 SECTION 3. LIBERTY PROGRAM EVALUATION 34 3.1 Liberty Institute Project Partners 34 3.2 Program Objectives and Activities 34 3.3 Activities Completed by Liberty 37 3.4 Evaluation Findings 44 SECTION 4. CONCLUSIONS AND RECOMMENDATIONS 47 4.1 Barriers Faced by Both Projects 47 4.2 Support for the USAID IRs 47 4.3 The Projects: Successes and Limitations 48 4.4 Role of Civil Society Initiatives in the Energy Sector in the Future 49 4.5 Recommendations 51 LIST OF TABLES Table 1. CENN Project Outcomes, Outputs and Indicators 15 Table 2. CENN Project Activities 16 Table 3. Planned and Implemented Activities for CENN Project 17-18 Table 4. Statistical Information on ESCA Activities 23 Table 5. Output indicators and achievements for Objective 1 25 Table 6. Output indicators and achievements for Objective 2 28 Table 7. Output indicators and achievements for Objective 3 31 Table 8. Outcomes, Outputs, and Output indicators for the Liberty Project 35 Table 9. Tasks, M&E Criteria, and Activity Indicators 36 Table 10. Tasks, Activities, and Achievements 37 3 Table 11. Hotline Operation Data 40 Table 12. Journalist Training Sessions 43 Map of Districts in Tbilisi 30 LIST OF APPENDICES Evaluation Scope of Work 53 List of Interviews Conducted 55 List of Documents Reviewed 56 Summary of Hotline Calls Received in Tbilisi and Regions 57 Focus Group Summary 61 List of Abbreviations 74 4 EXECUTIVE SUMMARY Background The US Agency for International Development (USAID) has been actively promoting civil society engagement in transforming the energy sector, with the objective of improving sector transparency and accountability. USAID’s Office of Energy and Information Technology, Energy Division (EGAT/EIT/E) issued a global Annual Program Statement (APS) in March 2003 to solicit proposals from non-governmental organizations (NGOs), academic institutions, and other organizations working to strengthen civil society. The APS encouraged creative approaches in support of USAID’s energy sector governance program that could provide models for replication and scale-up. USAID received two proposals from NGOs in Georgia in response to the APS: The Caucasus Environmental NGO Network (CENN) and Liberty Institute. Both the CENN and Liberty programs were designed to “increase public understanding of, and participation in, decisions regarding delivery of energy services”, one of the objectives of the APS. The Liberty Institute’s program, “Towards Public Accountability in the Energy Sector”, was implemented in the period January 2004 to June 2005. CENN’s program, “Development of Local Capacities and Public Awareness for Better Energy Governance”, November 2003 until August 2005. USAID contracted with the Academy for Educational Development (AED) under the People, Energy and Development IQC mechanism (Contract #DOT-I-00-04-00002-00) to conduct an evaluation of the two NGO programs. Evaluation Objectives and Methodology The evaluation’s primary objective is to assess the impact of the two NGO projects, including outputs and outcomes, their effectiveness and potential for increased effectiveness, barriers, and the benefit of civil society involvement measured against the cost. The impact assessment considered whether or not these two programs provided support for USAID/Georgia’s IRs. The evaluation also explores the need for an independent consumer rights/advocacy organization within the transition process in the energy sector. Finally, the evaluation team was also asked to comment on the replicability of these two programs within Georgia and the value of similar programs in other countries. The AED evaluation team consisted of two US consultants (Ms. Mary Worzala and Dr. Reehana Raza) assisted by a Georgian firm specializing in field research (GORBI Research). Prior to the field work, the team collected project documents, clarified evaluation questions with USAID, developed a list of stakeholders, and defined a framework for evaluating the outputs and impact of the programs. Interviews were conducted with USAID/Washington staff and several stakeholders prior to the field research. The evaluation team spent two weeks in Georgia conducting interviews with 19 different organizations in Tbilisi, Kutaisi and Akhaltsikhe (see Appendix B). Four focus groups were conducted with project beneficiaries, including two in Tbilisi on the CENN program and one each in Kutaisi and Akhaltsikhe on Liberty’s program. A meeting was also conducted with Georgian media representatives, and was attended by 16 journalists. USAID, CENN and Liberty all reviewed and commented on the draft version of the report. The two programs were largely evaluated separately; however, there was a significant amount of collaboration between the organizations and an overlap in the stakeholder group. In addition, both programs were designed in support of the same USAID objective. 5 CENN Program Evaluation The CENN project, Development of Local Capacities and Public Awareness for Better Energy Governance, sought to ease existing tensions in the Georgian electricity sector through the strengthening of energy sector governance. The objective of the project was to enhance transparency and establish bottom-up accountability through community mobilization, public education, and consumer representation in Tbilisi. The project established Energy Services Consumers’ Association (ESCAs) in six districts1 in Tbilisi to serve as liaisons between consumers, state institutions, and private companies at the local level. The goal of these organizations was to empower citizens, and to provide the necessary skills for them to effectively serve as consumer advocates in the energy sector. Consumers were also made aware of their responsibilities as consumers of electricity in a market economy. CENN’s activities were focused in Tbilisi in the service territory of Telasi, the electricity distribution company that is owned by the Russian company, RAO UES. Between the period of the project proposal and its implementation, the distribution company was sold to RAO UES. CENN Program Key Findings The Evaluation Team found that the ESCAs played a successful mediating role for consumers and provided a focal point for registering complaints as well as mechanisms to address and resolve them. The ESCA’s advantage was that it was perceived by consumers as non-partisan, with access to key stakeholders, and direct access to solve their problems. In a transitional power sector, the ESCAs played a key role in helping the community take ownership of the problems they were facing, and realize that they could have a role in solving them. Reliance on the state for solving problems and maintaining community spaces resulted in community needs not being addressed properly. Through CENN’s program, residents came to realize that they could affect changes in their communities. Telasi also recognized the value of ESCAs as well￾informed intermediaries that were and deeply involved in their communities, which helped to increase collections and reduce Telasi’s commercial losses. The strong relationship between CENN and the distribution company contributed to the success of the project. CENN’s natural leadership approach worked through extensive mobilizing efforts and allowed natural community leaders to come forward, bringing a community network and established relationships to the project. Recruitment of “natural” leaders was one of the key elements in the success of CENN’s activities. The ESCA leaders were already involved in their communities, and the CENN program provided the tools and skills that they needed to be more effective. Providing minimal compensation for the ESCA leaders meant that their commitment was genuine and not motivated solely by financial compensation. There was great flexibility in how each of the ESCAs was set up. Mobilizers were encouraged to think independently and respond to the situation on the ground. Leaders were based in their communities and communicated with residents on a daily basis to help resolve problems. ESCA offices were established in locations that made sense to each leader – in a school, apartment building, or community building. By September 2004, all six ESCAs were established. Training was provided to ESCAs on specific energy subjects and development of operational plans. The ESCAs also worked together to identify the problems common to all their districts. These included: 1 The six districts were Nutsubidze, Varketili, Didube, Okrosubani, Avlabari and Nadzaladevi 3 The six districts were Nutsubidze, Varketili, Didube, Okrosubani, Avlabari and Nadzaladevi 6 • Old debts • Flat rates despite the existence of meters • Electricity theft • Poor communication between Telasi and Consumers • Seasonal tariffs • Problems related with the installation of new electric meters Mobilization around a concrete issue in one sector helped the mobilization process and provided a focus for ESCA activities. Support for the ESCAs increased with the achievement of concrete results. Resolving small problems effectively and addressing community concerns built credibility in the effectiveness of the ESCAs. CENN, with the ESCAs, worked closely with the municipality to resolve neighborhood issues, including street-lighting and other community￾wide concerns. Over time, the ESCAs have had a number of successes. These include: • organizing condominium associations to solve illegal connections, electricity theft, and problems related to malfunctioning elevators; • mobilized neighborhoods to pressure municipalities to solve the lack of street-lighting; • negotiated with Telasi to post rights of electricity service consumers in the Telasi Business Center; • regular public consultation meetings are held by ESCAs where consumers can discuss their problems; and • debts were recalculated and paid and electricity service was re-assumed for individual consumers. The consolidation of the six ESCAs into a national ESCA with an expansion of its role both functionally (from electricity to a range of services) and geographically (from six neighborhoods to all of Tbilisi) may be too rapid. There are questions about how the new ESCA structure will affects its ability to continue to solve problems at the neighborhood level. The new structure of the association envisions activity both at the grassroots level and the policy level. The ESCAs’ success may be diluted as a national organization without grassroots mobilization or extensive interaction with key stakeholders in sectors other than electricity. Finally, the ESCA is not yet well-established enough for people to pay for their services, and most of the people who used their service are disadvantaged. Financial viability going forward is a concern. Liberty Program Evaluation The Liberty Institute project, Towards Public Accountability in the Energy Sector, was designed to raise awareness of journalists, politicians and the general public concerning energy sector issues, and to promote transparency and accountability in the energy sector. This objective was to be accomplished by raising public awareness of energy issues, monitoring developments in the energy sector for corruption, increasing the accountability of politicians and political parties on energy sector issues, and increasing media capacity to provide coverage of the energy sector. The Liberty project worked mostly at the national level in raising awareness through its media activities, but also worked regionally through a program of town hall meetings, information dissemination and the establishment of consumer hotlines. The media capacity-building program strengthened contacts between journalists and key players in the energy sector. The journalism program was highly regarded by journalists and stakeholders who participated in the training and awards programs. The Liberty newsletters 7 and energy user’s guides were distributed to those who expressed an interest in energy issues and to stakeholders in the energy sector. Media monitoring provided good information for energy stakeholders, was used in planning the winter energy campaign conducted by the Ministry of Energy and Fuels (MEF), and was useful to the other energy stakeholders. Energy stakeholder meetings were held jointly with all key players in the energy sector. Stakeholders felt that these were useful forums, and because they were held regularly, there was the necessary continuity to tackle important issues. The Liberty program created space for public dialogue through radio programs, radio bridges, town hall meetings, roundtables and other activities. Liberty had a strong partnership with MEF and the USAID-funded energy sector public outreach and participation project. Together, they collaborated on town hall meetings, public service announcements, radio programs, and the March 2005 survey was revised to include questions that were important to MEF. It is difficult, however, to concretely measure the effectiveness and impact of these information and outreach activities. Liberty’s town hall meetings successfully diffused tensions in a highly politicized environment; however, they were not sustained interactions that allowed citizens to resolve their problems. Town hall meetings brought together representatives from the United Energy Distribution Company (UEDC), MEF, and NGOs to discuss problems and provided a forum for consumer complaints to be expressed. The town hall meetings were held at a critical juncture in the regions, during the period of installation of communal meters. Despite UEDC publicity campaigns, residents in the affected communities were not adequately prepared for this change, and were outraged and confused. Liberty provided a forum in the town hall meetings to vent this anger and clarify consumer concerns. Changes in project management and partners may have affected the continuity of the Liberty project’s activities. Regional staffs assigned to the project were working on other Liberty projects at the same time. While Liberty provided visibility and authority to citizens’ concerns in energy sector issues, the multiple staff roles may have diluted their influence in solving energy sector concerns. The hotlines created another mechanism for citizens to voice their concerns; however, inadequate institutional structures were in place for the hotlines to solve consumers’ complaints. Liberty never received strong support or interest from UEDC; their involvement and support was variable, especially at the beginning. It appeared from comments made that Liberty and UEDC had a somewhat antagonistic relationship at times. UEDC managers were often changing, on average every six months, which made it difficult for Liberty to liaise between consumers and UEDC. Liberty did not have much contact with GNERC, the Georgian energy regulatory body. Liberty could potentially have advocated for consumers in the communal meter hearings held by GNERC, however, these hearings were held after the project ended. Summary of Key Findings and Recommendations to USAID ESCAs have proved effective as grassroots organizations which offer consumers some recourse for their problems in the electricity sector, and other sectors, particularly for the poorest consumers. Through this interaction, consumers feel more confident about resolving problems with the distribution company and are empowered. ESCAs have also represented consumer interests at government bodies, specifically at GNERC. However, the evaluation team found little evidence that they have been particularly effective in the GNERC context. This may be the result of two factors: one that the ESCAs are at an early stage of their development and as yet are not capable of playing such a role and second, that the regulatory process in Georgia is 8 weak and there is little real opportunity for consumer interest groups to make meaningful contributions to policy making. Nevertheless, the ESCAs clearly have acquired a national presence. Although the Liberty project did not plan to play an active intermediary role between consumers and the UEDC, circumstances (communal metering) made intermediation necessary. Despite the lack of preparedness for this activity, Liberty’s interventions were often beneficial. Local Liberty offices represented consumer interests in resolving some of these problems. There is clearly a need to consider the establishment of community-based organizations like ESCAs in the regions. Communal meters are still causing tension and are not widely understood or accepted. A planned tariff increase planned will inevitably prove contentious both in Tbilisi and the regions. Even more challenging is the upcoming shift from communal meters to individual meters and the planned sale of UEDC to a private investor in the summer of 2006. All of these changes will be challenging for consumers who have already had to accept significant changes in their electricity supply. In such a context, a third party intermediary would prove useful. The ESCA approach pioneered by CENN is at a crucial stage. The recent formation of the national ESCA and its expansion both geographically and functionally may undermine its very strength as grassroots organizations. Geographically, ESCA leaders continue to be located in their local neighborhoods and have a presence in their local communities. However, the centralization of the ESCA offices as well as ESCA leaders’ involvement at a more national level may undermine the local links. ESCAs were always active in a number of sectors beyond electricity. However, their official expansion in these areas without the necessary lead up training—particularly familiarity with each sector and their key stakeholders—could lessen their effectiveness. CENN justified the formation of the association for sustainability, as well as, for the purpose of moving ESCAs centre stage on the public policy arena as the consumer organization. Although these steps are very justifiable, much more attention needs to be given to how ESCAs will manage these multiple roles, and whether the existing proposed structure is the only one to ensure the ESCAs’ effectiveness and sustainability. ESCA type organizations clearly serve two functions and are relevant both in transitional and in more stable electricity environments. They can be a conduit for public information but more importantly, they can represent organized consumer interests, essential in an efficient democratic process. Existing outreach programs serve an informational need or a social marketing approach rather than actively resolving problems that consumers face. UEDC and Telasi have customer service departments, but from the comments heard in the evaluation, it was clear that there remains a lot of distrust and lack of satisfaction and a need for a consumer representative body. Although a public advocate exists at GNERC, it is not an easy forum to access. The consumer advocate is there to help resolve problems with different arms of the sector, but is a forum of last resort. Resolution at a lower level, i.e. the neighborhood levels—the level at which ESCAs’ work may be more efficient and less contentious. Recommendations 1. Support for the ESCA should continue for at least another year. During this period, a means of self-financing needs to be found to allow the ESCA to be institutionally and financially sustainable. Currently, the ESCAs are trying to establish a user fee model. However given their 9 client base, which is mainly consumers at the lower income spectrum, this is not likely to be a viable approach. 2. The ESCA’s role and the level at which it can be most effective as a lobbying organization needs to be reviewed and analyzed. Currently, the ESCA Association is covering multiple roles including condo/neighborhood association as well as functioning as city-wide advocacy organization. It is likely that it cannot fulfill all of these roles successfully. 3. There needs to be a carefully consideration of the expansion of the ESCAs to the UEDC area. One of the possible suggestions has been to utilize the ESCAs themselves to lead the formation of similar organizations in the UEDC area. However given the current limited capacity of the ESCAs, it is recommended that they not serve as the primary drivers of this activity. Furthermore, this may also not be sensible as it contradicts a central platform of CENN’s success in establishing the ESCAs in Tbilisi. The success of the ESCAs has largely been due to local mobilization and the process of selecting ‘natural leaders’. Knowing the community and building confidence in the project at the local level have been central elements of the ESCAs’ success and will play a key role in their success in the UEDC regions. For the UEDC regions, the formation of the ESCAs will need to follow a similar confidence-building approach. The ESCA leaders from Tbilisi can play a role in training or sharing experiences, but are unlikely to be the leaders in the regions. 4. ESCAs need access to specialized resources, both technical and legal, to strengthen their effectiveness and scope. ESCA leaders mentioned that they need support on internal building issues (elevators, hallway lighting, stairways, roofs, etc.), electricity consumption in internal networks, and legal support to deal with the specific legislation on consumer rights and to strengthen their ability to defend consumer rights. 5. ESCAs can also play a watchdog role at the municipal level, ensuring transparency of municipal services contracts and municipal expenditures on neighborhood improvements. This role needs to be strengthened but must be considered in light of the overall assessment of ESCAs’ role (Recommendation 2). 6. Despite USAID and other donor investments in the regulatory process in the past, there is a continued need to strengthen such activity in order that a public space is created for public participation at the policy level. To this end, training and institution building for GNERC, ESCA, and citizens needs to continue. Currently ESCA leaders participate in the policy process at various levels, particularly with the municipality and GNERC. To strengthen the regulatory process, more training is required both for ESCAs and GNERC. ESCAs need to be taught how to approach regulatory issues and how to participate in the regulatory process. The regulatory body, despite its function, also has a limited understanding of the role of civil society in the regulatory process. The current view is that ESCAs are a medium through which public information can be shared rather than a valid means through which consumers ‘voices’ on electricity service delivery can be heard and incorporated in the policy-making process. New government legislation which will decentralize local government will have implications for how ESCAs can influence the local policy and decision-making process. Consumer organizations like ESCAs can participate in the policy making process at a decentralized level. All these aspects of training and institutional building will strengthen ESCAs’ (and other future consumer organizations) effectiveness. 7. A fully functioning, democratized energy sector requires public participation at various levels. There is a need to develop the capacity of think tanks, academics, and journalists who 10 can be well-informed voices to help ensure transparency and accountability in the sector, and ensure that alternative views are expressed and accounted for in the policy-making process. 11 1.0 INTRODUCTION 1.1 Background The US Agency for International Development (USAID) has been actively engaged in promoting the role of civil society organizations in transforming the energy sector, with the objective of improving sector transparency and accountability. In this context, USAID’s Office of Energy and Information Technology, Energy Division (EGAT/EIT/E) issued a global Annual Program Statement (APS) in March 2003 to solicit proposals from non-governmental organizations (NGOs) and academic institutions. The APS was intended to support creative approaches within the NGO and academic community to promote the energy sector governance program. The areas of special interest in the APS included: • Improving policy, legal, and regulatory frameworks to establish necessary conditions of energy service delivery; • Increasing institutional ability to provide or deliver energy management services; and • Increasing public understanding of, and participation in, decisions regarding delivery of energy services. USAID received two proposals from Georgia and, based on favorable evaluation, awarded cooperative agreements to the Caucasus Environmental NGO Network (CENN) and the Liberty Institute. Both CENN and Liberty have been active members of the NGO community in Georgia for many years. For the most part, both NGOs were involved in problems outside of the energy sector, with CENN focused on environmental problems in the region and Liberty focused on human rights. Prior to the APS issuance, the NGO community in Georgia became more aware of the problem of energy sector corruption through a presentation made by one of the USAID contractors working in the sector. USAID staff working in the Democracy and Governance sector also attended the presentation, and became convinced that the NGO community had a role to play in increasing the transparency and promoting good governance in the energy sector. When the APS was released, there was a convergence of interest and opportunity which resulted in the CENN and Liberty proposals. USAID staff and contractors working with the NGO community made them aware of the opportunity and helped them to think through the objectives and strategies of their programs. This support was an important factor in the eventual success of these two programs obtaining funding. Both the CENN and Liberty programs were designed in support of the third area of interest: increasing public understanding of, and participation in, decisions regarding delivery of energy services. The Liberty Institute’s program, “Towards Public Accountability in the Energy Sector”, was implemented in the period January 2004 to June 2005. CENN’s program, “Development of Local Capacities and Public Awareness for Better Energy Governance”, was initially scheduled to take place over the same time period, but was given a no-cost extension to continue work until August 2005. USAID contracted with the Academy for Educational Development (AED) under the People, Energy and Development IQC mechanism to conduct an evaluation of the two NGO programs. 1.2 Objectives of the Evaluation Based on the Scope of Work for the evaluation (included as Attachment A), and discussions with USAID, the primary questions which were addressed in the evaluation include: 12 1. Evaluate the impact of CENN and Liberty projects, including outputs and outcomes. 2. Evaluate: a) effectiveness of projects; b) potential for increased effectiveness; c) barriers to achieving objectives; and d) benefit of civil society involvement measured against the cost. 3. Do these two projects support USAID/Georgia's IRs? If so, how do they support the IRs? The primary focus was on IR 1.51.1, "Improved Commercial Performance of Energy Sector Entities, Sub-IR 1.51.1.2, "Increased Public Trust in the Energy Sector". 4. Within the transition process in the energy sector, how necessary is it to have an independent consumer rights/advocacy organization as opposed to customer service departments within the utility and/or ministry? To what extent were these projects able to organize political voice, whether through the ESCAs or other activities, and how does the customer department within the distribution company play a role in this, if at all? 5. Should the Georgia Mission continue to fund civil society projects in the energy sector? What particular elements were either successful or unsuccessful in these two projects? How replicable are these two projects in other areas of the world? 1.3 Methodology The methodology used in the evaluation of these two programs was to employ multiple research tools for information gathering and to pursue a verification process through triangulation. The research tools used to gather information included: • focus groups with beneficiary groups • semi-structured interviews with key stakeholders • document review of the two NGOs and additional relevant materials • data from the distribution companies on key measurable outputs (as available) The two programs were largely evaluated individually with the focus on the merits of each program. However, there was a significant amount of collaboration between the programs and an overlap in stakeholder groups. In addition, both programs support USAID’s objective of increasing transparency in the sector. The evaluation report presents the activities and accomplishments of each program separately, with the CENN and Liberty programs included in Section 2 and Section 3, respectively. Section 4 addresses the role of civil society organizations in the energy sector more broadly, and contains recommendations that have resulted from the evaluation work. 2.0 CENN PROGRAM EVALUATION 2.1 CENN Project Partners The CENN project, Development of Local Capacities and Public Awareness for Better Energy Governance, sought to ease existing tensions in the Georgian electricity sector through strengthening energy governance. In the past, tensions in the sector have been unusually high. Georgians have had to accept new market rules in the electricity sector, which have not always been accompanied by improvements in the quality of service. The objective of the project was to enhance transparency and establish bottom-up accountability through community 13 mobilization, public education, and consumer representation. The project established ESCAs (Energy Services Consumers’ Association) in six districts3 to serve as a liaison between consumers, state institutions, and private companies at the local level. The goals of these organizations were to empower citizens, and to provide the necessary skills for them to effectively serve as consumer advocates in the energy sector. Consumers were also made aware of their responsibilities as consumers of electricity in a market economy. All six of the ESCAs are based in the Georgian capitol, Tbilisi. The project was implemented by an NGO coalition led by CENN. The coalition included CENN, Green Alternatives, and Eco-Vision. The project team also collaborated with the Liberty Institute and Green Wave at different points of the project. The project manager was based at CENN and was responsible for coordinating with each of the coalition partners. Each NGO in the coalition had discrete tasks: CENN was primarily responsible for local capacity building, mobilizing, establishing the ESCAs and preparing aspects of the public awareness activities; Eco-Vision was responsible for the school campaign; and Green Alternative prepared and broadcast the TV and radio programs. The first phase of the project lasted from November 2003 to April 2005, with a no-cost extension granted by USAID in April until August 2005. Levan Tavartkiladze was the project-manager in the first phase and staff turnover was limited during this period. Levan Tavarkiladze left at the end of the first phase, and Nika Malazonia was hired as the new project manager. A number of surveys and analyses were undertaken in the project preparation phase to inform the coalition of the state of affairs in the sector. Local capacity building and mobilization for laying the groundwork for the ESCAs began in summer of 2004 and the ESCAs were established in September 2004. Several training sessions were held both to train the mobilizers as well as to build up the capacity of the ESCAs. Parallel to this, a number of activities took place to build up public awareness. Public awareness activities included implementing a school campaign for energy awareness, preparing and broadcasting TV and radio programs, and putting together publicity materials and other public information documents related to the ESCAs, consumer rights and energy sector governance. 2.2 Program Objectives and Activities 2.2.1 Anticipated Outcomes/Results According to the project proposal, the CENN project aimed to: … increase the transparency and efficiency of performance and consumption in the energy sector, through community mobilization, confidence building, increasing public awareness and citizens’ responsibility over the resources’ use, creation of ESCAs (Energy Services Consumers’ Association), increasing public participation in the decision-making processes at al levels and launching a nation-wide constructive dialog between all stakeholders on the energy sector governance” (extract from project proposal). There were three overriding objectives for the project: 1. Creating a more aware and responsible consumer 2. Establishing performing ESCAs 3. Responsible and accountable energy entities 14 The ESCAs also had specifics objectives. These included: • Building public education and responsibility; • Monitoring decisions and performance in the energy sector, and sharing the information with consumers (watchdog system); • Mediating disputes between private consumers and electricity distributors; • Lobbying for reforms that lead to better and more equitable electricity provision for consumers; and • Acting at grassroots level as consumers’ right protection units (advocacy). 2.2.2 Project Objectives and Indicators Prior to the conducting the field work, the evaluation team prepared the following table showing program objectives/outcomes, project outputs and output indicators. The anticipated outcomes, outputs and indicators are taken from CENN’s proposal and the project documents. Table 1. CENN Project Outcomes, Outputs and Indicators Outcomes/Objectives Outputs Output indicators More aware and responsible consumer Existing tensions in the sector will be reduced Establishment and performance of ESCAs ESCAs recognized by all stakeholders as the legitimate representative of consumers Responsible and accountable energy entities Existing tensions in the sector will be reduced Changed public behavior and increased public awareness and participation in 6 pilot districts in Tbilisi Number and capacity of established and functioning ESCAs Energy entities that are responsible and accountable to the public -payment rates -number of non payment cases -amount of illegal consumption -change in attitude as per public opinion surveys -participation in public meetings -public pressure and lobbying at different levels -quality and quantity of public input in decision making -number of organized meeting with concerned parties -number of public requests and visits to ESCAs -number of disputes resolved at ESCAs' level -number of proposals/recommendations presented -number of recommendations developed by ESCAS and utilized by the state -number of disputes resolved at the level of GNERC -number of roundtables and public consultations meeting organized by the MEF -number of public meetings held by the Energy Committee of Parliament -number of proposal submitted by ESCAs on legislative initiatives considered by the Energy Committee of Parliament -Improved electricity services in the 6 districts -Functioning ESCAs mimicking condominium management units 15 2.2.3 Planned Activities of the CENN Project Similarly, the evaluation team prepared a table linking the broad objectives set by the project with key activities and activity indicators. These activities were included in the CENN’s proposal, as were the activity indicators. Table 2. CENN Project Activities Outcome Objective Planned Activity to Achieve Objective Activity Indicator More aware and responsible consumer Existing tensions in the sector will be reduced Establishment and performance of ESCAs ESCAs recognized by all stakeholders as the legitimate representative of consumers Responsible and accountable energy entities Existing tensions in the sector will be reduced -Organization of general multi-stakeholder meeting -Implementation of small grants program -Delivering on-going training and training the trainers course -Preparation and publication of the best examples and practice of good governance -Broadcasting of TV and Radio programs -Implementation of school campaign -Development of district specific strategy -Community mobilization -Creation of ESCAs -Election of the council of ESCAs -Creation of database of ESCAs in Tbilisi -Implementation of small grants program -Delivering on-going training and training the trainers course -Organization of workshops for ESCAs and municipality -Preparation and publication of general materials for ESCAs -Development and transfer of recommendations to the relevant authorities -Preparation of TV and Radio programs -Organization of general multi-stakeholder meeting -Organization of experts' (trimestrial) meetings -Organization of workshops for ESCAs and municipality -Development and transfer of recommendations to the relevant authorities -Implementation of purposeful lobbying of rules and regulations directed toward sustainable development of the energy sector 500 copies of the leaflet describing best example are published and distributed 10,000 copies of leaflets, brochures on rights and responsibilities distributed 50 broadcast on local radio and TV database of ESCA is created 16 2.2.4 Monitoring and Evaluation Plan The monitoring and evaluation (M&E) plan for the project was developed around the three key objectives, with outputs and activities linked to these objectives. Both the output indicators as well as the activity indicators were included in the M&E work plan outlined by the CENN coalition. The M&E framework however raises a number of issues: • Objectives 1 and 3 were ambitious particularly given the timeframe of the project. It would have been beneficial for the project if the objectives were broken down more carefully into more concrete outputs • The output indicators were not always appropriate indicators for the output desired the timeframe of the project • The activity indicators were not always clearly related and many were left undefined Given what was achieved in the projects, focusing solely on the indicators given in the M&E framework may under-represent all that was done by CENN and its partners. In our evaluation, we examine these indicators but reflect beyond them to highlight the strengths of the project. 2.3 Activities Completed by CENN The evaluation team verified the completion of the tasks as reported by the CENN project team through interviews and document review. In this section, we will note areas where CENN’s work was particularly successful, where questions remain about the completed work or where the work could have been more effective. Table 3 lists CENN’s activities and achievements, and notes particular accomplishments. Following Table 3 is a narrative on the implemented activities. Table 3. Planned and Implemented Activities for CENN Project Planned Activity Number of Stakeholder Meetings Implementation of Small Grants Program Deliver ongoing training and training of trainers (TOT) course Preparation and publication of best practices of good governance Broadcasting TV and Radio programs Implemented Activity 9 multi-stakeholder meetings organized 2004-2005 6 ESCA (12 leaders total) given $220/month for transportation and expense reimbursement 8 trainings conducted. 2 for teaching community mobilization skills; 6 for building ESCA capacity Public Participation in the Energy Sector Governance published rather than leaflets -5 PSAs prepared -Radio shows prepared by Liberty and Green Wave Radio -Monthly articles published Comments Stronger awareness among stakeholders of CENN’s project and ESCAs Perception among ESCA leaders that the grant was in lieu of salary. Raises question of sustainability -Concrete training; operational and action plans around which ESCAS structure their activities were developed -There was a view that -CENN published additional materials as need arose -Focus group raised issue that ESCA clients did not use the publications 17 training should be offered to ESCAs on how to conduct themselves and prepare for forums. -Publications were praised by government stakeholders Objective 1 & 2 1 & 2 1 & 2 1 1 & 2 continued, Table 3. Planned and Implemented Activities for CENN Project Planned Activity Implementation of the School Program Development of District-specific strategy Community Mobilization Creation of ESCAs Creation of database of ESCAs in Tbilisi Implemented Activity -Prepared program on energy safety -Textbooks and workbooks prepared -Teachers trained in 6 districts -Calendar, poster, and painting competition; trips to Gardabani and Telasi conducted for children Individual strategies developed in each district -4 mobilizers trained -12 ESCA leaders identified and trained 6 ESCAs established ESCA database created; maintains information on clients served and problems solved Comments Well received project. Ministry of Energy is seeking funding to scale up the program to create a national curriculum. CENN staff viewed 6 neighborhoods as quite ambitious Selecting natural, independent activists in the community jump-started the building of trust and credibility for ESCAs ESCAs seen as legitimate intermediaries between consumers and other energy sector stakeholders. Question of sustainability remains. A mobile population and booming property market make database difficult to maintain. Names in database are often incorrect. Objective 1 2 2 2 2 Planned Activity Organization of workshops for ESCAs and municipalities Development and transfer of recommendations to authorities Organization of experts’ (trimestrial) meeting Implementation of purposeful lobbying of rules and regulations directed toward sustainable development of the energy sector Implemented Activity No separate workshops held between ESCAS and Tbilisi Municipality. Tbilisi Municipality was present at a monthly stakeholder roundtables 29 recommendations and proposals presented by ESCAs to Telasi and GNERC A regular part of the project was to hold meetings with key experts. ESCAS systematically put bottom-up pressure to promote rules and regulations directed toward sustainable development of the energy sector Comments Municipality viewed their partnership as with CENN. ESCAs without CENN would be weakened ESCAs have been successful with implementation of their recommendations by the Municipality and Telasi, but not GNERC To date they have not been able to influence system-wide change Objective 2 & 3 2 & 3 3 3 18 2.3.1 Activities under Objective 1: Creating a more aware and responsible consumer Stakeholder meetings The objective of the stakeholder meetings was to share information, assess stakeholders’ concern and build greater transparency. In total, nine multi-stakeholder roundtable meetings were organized through 2004-2005 in collaboration with Liberty. These were monthly events held on the last Friday of every month; the first one was held in October 2004. As a precursor to holding stakeholder meetings, CENN interviewed stakeholders in the sector to identify their legal obligations and responsibilities and determine how they could contribute to the project. Overall, one indication of the success of this strategy of intensive interaction and information sharing is the greater awareness among the key stakeholders in the electricity sector of CENN’s project as compared to Liberty’s. The Evaluation Team met a range of stakeholders who were aware of the project and were positive overall about the project’s activities. Small grants program The small grant scheme was designed to offer the ESCAs financial support based on the priorities they identified. This was not how the grant scheme was implemented. Rather then being one-time payments for specific activities, the funds were used for sustained monthly payments to the ESCA leaders. ESCA leaders viewed these payments as a stipend which, during the project period, strengthened their perception that they were employees of CENN. The monthly payments will need to be eliminated to ensure that the ESCAs are sustainable in the long run. Overall, the project was very good at keeping the ESCAs’ costs low for purposes of enhancing sustainability potential. Training and train the trainers courses Training sessions were offered in two areas: training the mobilizers, and capacity building for the ESCAs. The training for the mobilizers was conducted in January 2004. Since the training material was available only in Georgian, the evaluators were not able to assess the quality of the material. However, during the evaluation, the CENN project manager emphasized the importance of the training for developing the mobilization strategy for the ESCAs. Previously, CENN had no experience in community mobilization and this training enhanced CENN’s ability to design a successful mobilization strategy and plan. ESCA training focused on both energy issues and skills to develop operational and action plans. ESCA leaders were already active members of the community, but the training helped them to be more persistent about solving problems. One ESCA leader that noted problems had always existed but often remained unsolved. Now, she does not give up easily as she has learned to approach the problem and where to apply to solve these problems. Training topics included: • Legal framework of electricity supply and cut-off • The contract between customers and Telasi • Ownership of electricity meter • The different billing systems from 1999 onwards 19 • Electricity theft and non-payment of electricity bills • Social subsidies • Working with municipalities on communal problems The training on billing systems, electricity theft and non-payment of electricity and social subsidies were critical for undertaking debt restructuring. Particularly useful was the debt calculation matrix for calculating debt based on rooms and water capacity, which ESCA leaders used to negotiate with Telasi. In the Evaluation Team’s meetings with key stakeholders, particularly at GNERC and Telasi, there was a view that ESCA leaders needed to be trained to prepare for and conduct themselves in a public forum. This training would be useful in the next phase of building the ESCAs’ capacity. Preparation of best examples and practices of good governance This activity was part of the public outreach/education strategy. Originally, leaflets on the practice of good governance were planned. Instead, CENN put together a publication entitled “Public Participation in Energy Sector Governance.” CENN also put together a number of publications in response to particular gaps in their public outreach strategy. These included “The Citizen’s Guide for Electricity Consumers and Tariff”, which was distributed during meetings and to relevant stakeholders. The documents were generally well received by stakeholders within the government. However, customers who participated in the focus groups admitted that although they received these publications, they rarely looked at them; it was much easier to get ESCAs to solve their problems. It may have been useful to offer consumers a brief training on how they could use these documents, or to offer this information in a more accessible form. Media program Other key elements of the public outreach programs were the TV and radio programs. In total, four short (30 second) and one long public service announcements (PSA) were created. These programs were aired from November 2004 to March 2005. A regular monthly radio program on the ESCAs activities was put together in collaboration with Liberty and aired on Radio Green Wave. A documentary on the project’s activities was also produced that was approximately 20 minutes long and focused on ESCA activities. The documentary was aired at a local cinema during the time of the Evaluation Team’s visit in Georgia. CENN is currently in negotiations with local TV channels to air the program on mainstream television.4 The PSAs did not come up in the focus group discussions. Primary school energy education program To respond to citizens’ demand for more information on energy efficiency and electricity safety issues, the CENN project implemented an educational program targeted at six schools in the ESCA districts. The energy education was extremely successful and well-received by the public, Telasi, and the government, particularly the materials and classes on energy efficiency. The Energy Efficiency Book and Energy Efficiency Handbook describe energy saving practices and offer practical exercises on saving energy. To accompany the teaching materials, 12 4 Following completion of the evaluation, the CENN project team reported that the documentary has been aired on two TV channels: Stereo 1 and Channel 202. 20 teachers (two from each school) were trained from the six schools to deal with the pedagogical material. The classes were jointly taught by the project team and ESCA leaders, and key input was provided by the stakeholders. Telasi in particular was very involved, and made presentations on the following topics: • Georgian power system and its components; • Telasi the company; • Organizations responsible for electricity cut-off; and • The impact of non-payment of bills. A total of 120 students between the ages of 13-18 participated in the program. To accompany the lessons, excursions were organized to Telasi and to the Gardabani thermal power plant. A painting contest was held entitled “Winter in our Homes.” The success of this school program can be measured by a number of factors. First, the First Deputy Minister Alexander Khetaguri from the Ministry of Energy of Georgia expressed great admiration for the school program. He is currently encouraging the Ministry of Education to mainstream the program for all 7th and 8th graders across the country and is looking for donor funding to do this5 . Second, the sub￾contractor Eco-vision, in collaboration with CARE International, continues to use these textbooks and curriculum in the regions. Third, in the school visited by the evaluation team, the teachers expressed an interest in continuing the program with or without CENN’s support. They felt that they could utilize the existing materials for the current year’s students. 2.3.2 Activities under Objective 2: Establishing performing ESCAs Developing district-specific strategies A central element of establishing ESCAs was to identify the most suitable neighborhood in each district to focus ESCAs activities. In each district, a number of neighborhoods were examined and measured on a point system. Consideration was given to whether there was a need for such an organization in a particular neighborhood, and whether there were people there who could potentially take the ESCA idea forward. The point system was based on a number of variables including: where transformers were located within a neighborhood; focusing on neighborhoods of 200-300 households; level of commercial losses; the number of cooperative buildings within each district; good locations for ESCA/mobilization meetings; and the number of homes owing debts to Telasi. Based on these criteria, neighborhoods within each district were chosen. Supplementary surveys and analysis were then carried in each district to identify socially active people and local issues around which mobilization could occur. The CENN team did mention that working with six neighborhoods was difficult and next time, they would limit activities to fewer neighborhoods. Community mobilization Community mobilization was structured around intensive interaction with all households in the neighborhood, and interventions in the community to build up good-will and credibility. Community problems, which the project team addressed, included problems related to street￾lighting, debt restructuring, and often included projects unrelated to electricity. Through interactions with the community, potential community members were identified as leaders, and 5 After the period of the evaluation, the project team met with the Ministry of Education to discuss integrating an energy component into the curriculum. The idea was subsequently approved by the Ministry, and the energy materials are planned to be included in the 2006-2007 school year. 21 22 approached by the project team to see if they would be interested in participating in an organization like the ESCA. The CENN project team made a conscious decision to build up “natural leaders” who evolved from within the community rather than working with existing organization and leaders of organizations. This was because established leaders already have links with different political and non-political bodies, which may make them less acceptable to the community. The project deliberately selected people who were active in the community on their own accord and were known to seek out opportunities to help people in their communities. Another important characteristic the project team focused on was individuals who had a “fighting spirit.” Overall, choosing natural leaders proved effective in providing the necessary energy to jumpstart the ESCAs. Inevitably, some ESCA leaders were stronger than others and were more effective. However, the community of ESCAs proved strong, and as one ESCA leader mentioned, they learned by working together. Creation of ESCAs By September 2004, all six ESCAs were established. A total of six training sessions were offered on specific energy subjects followed by training on how to develop an action and operational plan. At the end of the training, each ESCA produced their own action and operation plan with a breakdown in actions, timetable and responsible people, and possible partners to achieve their goals. The ESCAs also worked together to identify problems common across districts. These included: • Old debts • Flat rates despite the existence of meters • Electricity theft • Poor communication between Telasi and consumers • Seasonal tariffs • Problems related with the installation of new electric meters Over time, the ESCAs have had a number of successes. These include: • organizing condominium associations to solve illegal connections, electricity theft, and problems related to malfunctioning elevators; • mobilized neighborhoods to pressure municipalities to solve the lack of street-lighting; • negotiated with Telasi to post rights of electricity service consumers in the Telasi Business Center; • regular public consultation meetings are held by ESCAs where consumers can discuss their problems; and • debts were recalculated and paid and electricity service was resumed for individuals. Table 4 on the next page offers a more detailed list of ESCA activities. Table 4. Statistical Information on ESCA Activities DISTRICTS # Type of Activity Unit Varketili Avlabari Okrosubani Nutsubidze Didube Nadzaladevi Total 1 Time period ESCAs have been working Months 10 10 10 10 10 10 10 2 Number of households in the ESCAs neighborhood Households 200 300 150 300 200 180 1330 3 Number of households ESCAs helped with debt Households 52 228 97 0 0 59 436 4 Number of customers who received other types of consultations from ESCAs Customer 126 553 40 31 180 155 1085 6 Number of mistakes found in Telasi debt calculations Mistakes 0 175 45 0 0 52 7 Amount of incorrectly billed debt cancelled GEL 0 20,000 5,000 0 0 16,050 41,050 8 Amount of debt restructured GEL 4,876 22,800 5,000 0 0 17,000 49,676 9 Repair of sewerage system Building 5 0 0 0 0 3 8 10 Social subsidies on communal services and electricity Households 0 65 7 0 0 0 72 11 Repair of elevators Building 0 0 0 5 0 0 5 12 Street 5 0 0 0 5 3 Outside lighting 13 13 Entrance lighting Building 5 0 0 5 5 0 15 14 Repair of damaged meters Meter 5 20 3 10 7 0 45 15 Installation of new meters Meter 10 25 0 0 0 0 35 16 Replacement of obsolete electric cables Street 0 5 0 0 3 0 8 Creation of ESCA database An ESCA database has been established and resides in the ESCA central office. The database lists the different types of consultations that the ESCAs have offered. The information that is maintained includes: the district where the activity occurred; type of case (electricity or other); a case-number (abonent); name of person helped and their contact address; debt, if any; type of action provided. The focus group subcontractor, GORBI, used this data base to select participants for the focus groups that were conducted. The database was not without problems. In Avlabari district, many of the names (and addresses) proved useless since many of the houses had been sold or properties destroyed sue to a booming property market. In the second district, Varketili, GORBI found that many of the addresses visited were not aware of the ESCAs or their activities (7 out of 17). The problems are not necessarily related to how the database is managed. There is little that ESCAs can do to keep the database current in a transitional environment. In cases where people were not aware of ESCAs, this may reflect that the household member who interacted with the ESCA was not present, or that they are not familiar with the organization ESCA, but the ESCA leader by name. The latter suggests a greater need to brand ESCA. Organization of workshops for ESCAs and Tbilisi municipality The Tbilisi Municipality was one of the many stakeholders involved in the monthly roundtables held by CENN. The monthly roundtables were important for introducing the ESCAs to these stakeholders, including the Tbilisi Municipality. Following theses roundtables, a number of problems were brought to the attention of the municipality. These included issues related to: street-lighting; power supply cut-offs due to accidents; water supply issues; and sewage problems. Members of the Municipality stated that CENN played an important role in bringing neighborhood issues to the attention of the municipality. They also stated that CENN’s activities had played an important role in educating people. The Evaluation Team noted that although the Municipality staff was aware of the ESCAs, their discussions centered on CENN rather than the ESCAs, which suggest that the brand identity is CENN rather than the ESCAs. The Municipality staff emphasized that ESCAs need to function as a group to be more effective. They also stated that without CENN the ESCAs would be weakened. ESCA leaders, too, mentioned that their interactions with the Municipality had been on a one-to-one basis and they had represented themselves as CENN members. All of these factors have weakened the ESCA brand identity, which needs to be developed for long term sustainability and for the period when the ESCAs have been weaned off CENN support. The Municipality staff continued to see an important role for the ESCAs in a number of areas outside electricity, including heating. They also emphasized the importance of lobbying for ESCAs to be effective in getting their issues on the table. Development and transfer of recommendations to the authorities Developing recommendations and proposals for greater efficiency of the electricity sector is an important task of the ESCAs. In total, 29 proposals were put forward by the ESCAs to these bodies. In addition, ESCAs regularly put forward recommendations in the stakeholder meetings. Where ESCAs have had relatively more success has been at the Tbilisi Municipality and Telasi, where concrete actions have been taken such as the repairing and reinstalling the street lights. They have had limited success at GNERC. The ESCAs have also been more successful on resolving individual customer problems rather than having an influence on system-wide change, either in legislation or processes. 2.3.3 Activities under Objective 3: Responsible and Accountable Energy Entities Organization of experts’ meetings A central element of the project was to organize regular meeting with energy experts for the benefit of the project staff and the ESCAs. These meetings kept the project team and ESCAs abreast of key issues in the sector and led to specific pieces of research being produced. The objective was also to use the research to cross-check the situation on the ground. The following areas were of specific interest to CENN: procedures of licensing; the legal framework; analysis of privatization agreement; legal status of condominiums; opportunity for public participation in governance; etc. Elements of the research were incorporated into the CENN publications. Rules and regulations for sustainable development of the energy sector ESCAs need to be more effective in ensuring that rules and regulations are implemented that contribute to sustainability in the energy sector. Although the ESCAs have been lobbying and putting grassroots pressure on the relevant stakeholders, as yet they have not been successful in pushing through systematic reform. 2.4 Evaluation Findings This section addresses the overall evaluation findings for the CENN Project, including effectiveness, efficiency, program management, and barriers faced by the project. Findings relating to support to USAID IRs in included in Section 4. 2.4.1 Effectiveness of CENN’s project Objective 1: Creating a more aware and responsible consumer The expected output of Objective 1 was “changed public behavior and increased public awareness and participation in 6 pilot districts in Tbilisi”. The output indicators are presented in the Table below, along with the evaluation team’s findings on whether the output was achieved. As we have noted elsewhere, focusing solely on these objectives would underestimate the actual achievements of the project. Table 5. Output indicators and achievements for Objective 1 Output Indicator Achieved or Not? Debt payments increases In Avlabari district, for example, with the ESCA’s help, 1500-2000 GEL are paid to Telasi every month, approximately 5% of Avlabari district’s total payment. Number of non-payment cases decreases Increased payment rates automatically decrease the number of non-payment cases Amount of illegal consumption ESCAs were active in identifying local theft cases in their neighborhoods Change in attitude as per public opinion surveys Difficult to say based on surveys at hand Participation in public meetings -ESCAs introduced amendments in the electricity related legislation for public discussion and review -ESCA members regularly attended public meetings organized GNERC Public pressure and lobbying at different levels Difficult to say if it had a significant impact Quality and quantity of public input in decision-making Difficult to say if it had a significant impact 25 How effective was the CENN project in creating a more aware and responsible consumer? As noted at the outset, this objective is an ambitious one. Moreover, underlying CENN’s indicator is an assumption that a more aware consumer automatically changes his/or her behavior regarding payments and theft, which may not necessarily be the case. Two surveys undertaken by BCG (March 2004 and 2005) for Liberty and PA Consulting (June/July 2005) for USAID can offer some insight into consumers’ awareness and behavior about the electricity sector towards the end of the project period. However, the changes, or lack of change, cannot be attributed to the CENN project directly. The numerous changes that were happening in the sector, particularly the rapid improvement in service delivery over this period due to investments made in the sector by the government, are more likely to be significant influences on consumer behavior. In the BCG surveys there were few questions that were the same in the two surveys, limiting trend analysis.6 The PA survey was of citizens and enterprises in all regions of Georgia, including Tbilisi (the CENN project area), and its objective was to ascertain reliability of service, and attitude, belief, and knowledge about the energy sector. The survey reveals the following about electricity consumers in Tbilisi: • 30.6% of Tbilisi citizens still do not know the name of their local distribution company. • Compared to the previous year, 37.8% of citizens felt electricity supply was much better; 39.4% felt it was a little better; 37.2% of enterprises felt electricity was much better, and 28.4% felt it was a little better. • Tbilisi citizens had an awareness of the Ministry of Energy, and only half that were surveyed were aware of GNERC, GWEM and GSE. Only 17% knew of the existence of the Energy Public Advocate. Of those who knew these institutions, very few knew what they did. • Only 11.2% of those surveyed in Tbilisi knew of the ESCAs, and of those 8.3% admitted to being friends or relatives of ESCA members. • 53.4% and 48.6% of citizens and enterprises in Tbilisi, respectively, were felt there was corruption in the electricity sector, and a significant number saw the problem as being in the MEF. Nevertheless, 43% of citizen and 56.5% of enterprises felt corruption was lower than the year before. In terms of solutions, they also saw the corrupt MEF as being the solution to the problem. (PA Consulting Survey 2005) The survey sample included districts other than the six districts of the CENN project. Hence, these cannot be directly correlated to CENN and ESCA activities. Nevertheless, the results do indicate that although people are feeling more positive about the energy sector, they still seem to lack the most basic information about the sector, and there is still a need for public information. Although the percentage of people who knew about the ESCAs was small, it is respectable given the geographical scope of the ESCAs’ work. It is difficult for the evaluation team to determine how effective the ESCAs were in changing people’s behavior. The indicators, although premised on a strong assumption, do not suggest so. It is perhaps more appropriate to say that those customers who have interacted with the ESCAs and have successfully been helped have again became more responsible customers, and local monitoring by the ESCA leaders did result in electricity theft being identified. Where ESCAs have been successful is in their ability to solve consumers’ real problems; they have generated good will and eased tensions and suffering within these communities. 6 See Section 3 on the Evaluation of the Liberty Program for more details on this. 26 Nevertheless, in terms of easing tensions, a more significant factor has been the better supply of electricity. These are some of the comments about the state of electricity service before the ESCAs were established: • Consumers felt less powerless. Focus group participants expressed how they frequently visited key stakeholders, particularly Telasi, but were not able to get answers to their problems, both because the staff was unresponsive and because there were frequent changes in the staff. • Consumers felt that their rights were violated by Telasi. One focus group participant stated “…Everyone has rights and responsibilities, whereas Telasi only accentuates the responsibilities of the customer…” ESCAs have been able to play an important intermediary role between customers and the different stakeholders in the sector. Some of the comments from the focus group include: “I had no electricity for 5 years because of arrears and I could not pay the enormous debt I was asked to cover. I asked for debt restructure but the only answer I got was, ‘We do not care about your problems.’ When I learned about ESCA, I called their office and was told to leave my data at the end of December. They settled everything and never asked to be paid for the service. Now I have already paid part of my debt.” - Unemployed women from Varketili, 71 “As a neutral organization this association is more trusted than other organizations. Moreover, it has never refused to help people. The Association helps people who are unable on their own to protect their rights in dealing with TELASI.” -Woman from Avlabari, NGO Member, 47 In any real sense ESCAs are the only body that represents customers’ interest in the sector. Although there is a Public Advocate at GNERC whose responsibility it is to represent the interest of consumers, very few people have actually heard of it. It is also likely that individual consumers would have a harder time dealing with bureaucracy involved in accessing the Public Advocate than a body like the ESCAs, who function as a collective and have knowledge of the system Objective 2: Establishing performing ESCAs The expected output for Objective 2 was “…established and functioning ESCAs.” A number of specific goals were also set for the ESCAs including: • Building public education and responsibility; • Monitoring decisions and performance in the energy sector, and sharing the information with consumers (watchdog system); • Mediating disputes between private consumers and electricity distributors; • Lobbying for reforms that lead to better and more equitable electricity provision for consumers; and • Acting at grass root level as consumers’ right protection units (advocacy). Table 6 below assesses the achievement of the expected output during the project period. 27 Table 6. Output indicators and achievements for Objective 2 Output Indicator Achieved or Not? Number of organized meeting with concerned parties -On average 3 monthly meetings were held with GNERC -20 meetings were held with Telasi every month -Multi-stakeholder roundtables were held every month which included ESCAs -ESCAs held regular public meetings at their offices Number of public requests and visits to ESCAs -Approximately 300 public requests are made per month -On average 3 customers come in to the ESCA offices daily -There has been a demand for ESCAs to expand their services Number of disputes resolved at the ESCAs’ level -1085 people were provided with consultations -436 families have been assisted with debt problems -67 bills were found with incorrect calculations -41,050 GEL were cancelled; 49,676 GEL were restructured Number of proposals/recommendation presented -At the roundtables, 29 recommendations and proposal were presented by ESCAS to different parties -Regular suggestions and recommendations were presented to Telasi Number of recommendations developed by ESCAs and utilized by the State -5 recommendations given by the ESCAs were used by Telasi -Based on the lobbying by the ESCAs, 13 streets were provided with street lighting Overall, the project has successfully met this objective. The ESCAs are functioning bodies and are accepted as such by key stakeholders in the energy sector. Those customers who have had interactions with them also believe that the ESCAs are effective intermediaries who work on behalf of consumers. ESCAs clearly have had an important role in delivering public education on consumer rights and other electricity sector information, have been successful in mediating disputes between private consumers and electricity distribution companies, and have played an advocacy role. Perhaps where they have not been as successful is lobbying for reform. To qualify, they have had some success at Telasi and the municipality level in resolving specific customer concerns and resolving street-lighting; thirteen streets in these six districts have been provided with street lighting. They have had less success in changing policy or legislation at GNERC. The ESCAs have also not played a significant role in monitoring the performance of the electricity sector and sharing that information with consumers. However, ESCAs may not be the most appropriate body to undertake such a task as this may require a body that has more technical skills and authority to undertake this more complex task. The ESCAs have, however, been successful in moving into offering intermediation in areas other than electricity. The following are some key observations on the establishment and functioning of the ESCAs: • There was a general consensus from all parties interviewed that ESCAs played a successful mediating role. The ESCAs were able to channel anger and distrust for more constructive purposes and better dialogue. People were organizing before, but the ESCAs provided a focal point for registering complaints and mechanisms to address/resolve them. • CENN’s natural leadership approach worked through extensive mobilizing efforts. This was successful in a highly politicized context where there was little public trust in existing organizations. By allowing natural community leaders to come forward and be involved in the program, each of the leaders was able bring to the program a community network 28 and established relationships. Recruitment of “natural” leaders was one of the key elements in the success of CENN’s activities. • Providing minimal compensation for the ESCA leaders meant that their commitment was real and not motivated by financial compensation. The ESCA leaders were already involved in their communities and the CENN program provided the tools and skills that they needed to be more effective. • There was great flexibility on how each of the ESCAs was set up. CENN let these organizations evolve organically. No standardized approach was followed and mobilizers were encouraged to think independently and respond to the situation on the ground. Leaders were based in their communities and were talking with residents on a daily basis to help resolve problems. ESCA offices were established in locations that made sense to each leader – in a school, apartment building, community building at little or no cost. • In a transitional electricity sector, the ESCAs played a key role in helping the community realize that the problems they were facing were their own and they could have a role in solving them. Reliance on the state for solving problems results in community needs not being addressed. Residents came to realize that they could affect changes in their communities. • Mobilization was around a concrete issue in one sector. This helped the mobilization process and provided a focus for ESCA activities. Support for the ESCAs also built up after they were able to achieve concrete results. By resolving small problems effectively and addressing community concerns, they built up credibility in the community are their representatives. While electricity/energy sector concerns were their organizing focus, ESCAs dealt with other community problems as well. • The ESCA’s advantage is that it is seen as a non-partisan actor, and because of their access to key stakeholders, has better direct access to resolve problems. • Telasi recognized the value of ESCAs. This can be seen by the following factors: o When ESCAs bring a problem it is given more weight since Telasi has come to understand that the ESCA representatives have researched the customer’s problem and come with the necessary documentation to find a resolution. o Telasi would take ESCAs with them to explain key issues to consumers. This built up Telasi’s credibility with consumers as well. o ESCAs played an important monitoring role in their neighborhood by cross￾checking incidents of theft and fraud for Telasi. This signified a significant working relationship between the two. o Telasi, too, has undertaken greater community outreach, taking lessons from their interactions with the ESCAs and local communities. o ESCAs have helped increase collections for Telasi by restructuring debt, explaining to customers how bills are calculated, and verifying whether an incurred debt is correctly calculated and if the customer is liable to pay. • Knowledge of the ESCAs tended to spread through word of mouth and through the publicity they received in the media. One indicator of the demand for such services is to see the geographical location of ESCA clients. Besides those clients that they serve from within their districts, ESCAs have been helping consumers from other districts. The map below graphically demonstrates the scope of the ESCAs activities beyond the six districts in which the ESCAs were formed. 29 Map Legend -Red dots indicate the neighborhoods within 6 initial districts of Tbilisi where ESCAs initiated their services (Avlabari, Nadzaladevi, Varketili, Oqrosubani, Nutsubidze, Didube). The black circles in these areas mark the reach of ESCAs' services provided. -The green circles indicate the seven districts where ESCAs also provided services. These districts include: Vake, Digomi, Didi Digomi, Saburtalo, Gldani, Chugureti, Lotkini The ESCAs have evolved in their organizational structure in a way that may have implications for its future effectiveness. In June 2005, ESCA developed a charter and registered as an independent non-profit association. In the latter part of 2005 (and outside the project period), the six ESCAs were consolidated into one association with the intention of serving all of Tbilisi. The ESCAs are now housed in one office and have a “hotline” to serve all citizens of Tbilisi. CENN has been working with the ESCAs to develop organizational structures, laying out internal procedures, membership fee, payment structure for different types of members, establishing a new accounting system etc. The real concern is that the expansion of the ESCA from a grassroots body to a national association with the mandate to resolve all types of consumer problems beyond electricity may be too rapid. The ESCAs’ success seems to have been based on the fact that they were grassroots bodies with links to the local communities and the local stakeholders. They built up credibility because of their successes at the local community level. It is unclear whether the ESCAs will be able to offer that same kind of service now that the ESCA leaders function as members of a national association. A number of issues arise: i) Will ESCAs be able to have the same presence and same level of commitment to their local districts? ii) Will their expansion into new areas (sewage, heating etc.) be as successful given that they have not had the same level of interactions with local stakeholder in these different areas? 30 iii) Will ESCAs be able to offer the same level of intervention in other Tbilisi districts where they do not know the local stake-holders or where they are not part of the local community? There are also concerns about whether ESCAs will need to adjust how they function in a more decentralized local government setting that is currently being proposed and implemented in Georgia. A singular body in a time of greater decentralization may not be the most effective institutional match. What may be more appropriate is a number of bodies that serve different purposes. This includes grassroots bodies like ESCAs that interface with local utility stakeholder and consumers to resolve consumer problems on specific issues. A representative body like the ESCA Association could function at the national level with the task of lobbying and monitoring government. There may also be a need for condominium associations that organize for their own communities and interface with a national ESCA to resolve problems. Different types of organizations may be required to ensure an effective governance structure in the utility sectors. Currently, the amalgamation of the ESCAs into the ESCA Association without adequate consideration of all these roles suggests that the ESCA Association should be responsible for all of these tasks. This is not possible, nor is it effective. Objective 3: Responsible and accountable energy entities The third objective of the project was establishing more “responsible and accountable energy entities.” The secondary aim here was to contribute to easing tensions in the sector. The project indicators, and an assessment of progress towards achievement, are given below in Table 7. Table 7. Output indicators and achievements for Objective 3 Output Indicator Achieved or Not? Number of disputes resolved at the level of GNERC Resolved one dispute. ESCAs worked with ex￾ombudsman at GNERC/public advocate to resolve a case in Nadzaladevi district where 50 families were billed incorrectly by Telasi Number of roundtables and public consultations organized by MEF Members of MEF participated in the multi-stakeholder roundtable meetings organized by CENN. In total 9 meetings were held. Number of public meetings held by the Energy Committee of Parliament Energy Committee of Parliament was abolished after Rose Revolution Number of proposals submitted by ESCAs on legislative initiative considered by Energy Committee of Parliament -See Above￾Improved electricity service in six districts ESCAs helped restructure debt, through which electricity services were improved for particular consumers. Helped resolve problems of theft that would have had an effect on utility supply, although marginal given the scale involved. Functioning ESCAs as emulator of condominium management unit ESCAs helped with the formation of three condominium associations in Nutsubidze district. The ESCAs’ impact on different entities has varied depending on the stakeholders. There is no doubt that through outreach activities and roundtables the ESCAs have been established and are seen as credible intermediaries. Their greatest impact has been on Telasi. ESCA leaders have had the greatest opportunity to interact with Telasi to solve specific consumer problems, particularly in restructuring debt and in helping with the theft and fraud-related issues. Most of 31 the ESCA recommendations have targeted Telasi and the Tbilisi municipalities. CENN’s chosen indicators do not account for this and do not adequately reflect the responsiveness of Telasi and the municipalities to ESCA demands. ESCAs have not been as successful with GNERC. Besides the help they received from the previous ombudsman in solving the Nadzaladevi case, the ESCAs have generated little response from GNERC. ESCA leaders now regularly attend public hearings and participate but have had limited impact. Before the recent attenuation of GNERC, the ESCAs were part of a GNERC working group that was looking at internal networks in multi-family residential buildings; this group has now been disbanded. One reason for the limited impact ESCAs have had at this level may be the narrow view that GNERC staff members hold of public participation in the regulatory process. For GNERC, the role of NGOs and bodies like ESCAs is of public education. GNERC staff expressed the view that public participation was not significant for purposes of the regulatory process itself. Clearly, there is a limited commitment within GNERC regarding the role of public participation in the regulatory process and its centrality to better energy sector governance. This lack of understanding would contribute to the limited impact ESCAs had, and will have in the future, on regulation and regulatory reform. 2.4.2 Efficiency of CENN Projects Efficiency here refers to how resources, both human and financial, have been planned and expended. The following observations can be made from the evaluation: • CENN efficiently used the resources provided, paying careful attention to allocation between various activities. • Tasks were efficiently allocated among partners. Discrete tasks were given to each partner, which evolved from complying with USAID’s operational requirements. This proved effective, allowing each partner to have ownership over their project and execute it independently. • For sustainability, a key objective of CENN’s project was to ensure that the newly formed ESCAs had limited dependence on financing from CENN. Hence, ESCAs incurred limited expenses for accommodation and salaries. Accommodation costs were limited by relying on the availability of community space, unless absolutely necessary and no salaries were paid to ESCA leaders. However, a grant scheme was established to offer each ESCAs financial help on specific items that they requested. The grants became regular monthly payments and were give to all the ESCAs on a regular basis for their current expenditures. Many ESCA leaders saw this in lieu of a salary, which contributed to the view that they were CENN employees. • The school textbooks continue to be used both by the schools in the initial project as well as by CARE International in the regions. It is possible that these textbooks may be used again if the MEF is able to convince the Ministry of Education to include an energy component in the national curriculum. 2.4.3 Program Management and Relations with External Partners As previously discussed, CENN worked closely with partners in implementing the project. Partnerships of note include: 32 • USAID operational procedures helped clarify roles for different partners and helped CENN set up elements of the administrative and MIS systems which previously did not exist. CENN used this experience to set up the administrative and financial infrastructure for the ESCA Association as well. Save the Children helped CENN undertake these tasks. • CENN was very active in developing relationships with all the key stakeholders in the sector. They worked hard at establishing relationships with all the stakeholders in the sector including GNERC, MEF, the consumers, and Telasi. They were also very active in ensuring that these stakeholders were aware of their activities. • Telasi and CENN have built up a strong working relationship. Telasi helped CENN identify appropriate neighborhoods for establishing the ESCAs; CENN and partners involved Telasi in the schools program and worked together to produced the materials and teach the program. • Despite CENN and ESCAs extensive interactions with stakeholders, including GNERC, CENN was not aware of the meeting held at GNERC on the communal meters on July and August of 2005. • CENN, with the ESCAs, worked closely with the municipalities to resolve neighborhood wide issues, specifically street lighting and other community wide issues. 2.4.4 Barriers Faced by the Project Sectoral Barriers that are common to both projects reviewed are included in Section 4. Barriers going forward for the ESCA include: 1. There are questions about whether the new structure accompanying the ESCA Association will affect their effectiveness in the future. Currently, the ESCAs are carrying a multiple of tasks that should be carried out by different bodies or arms of the ESCAs. This lack of clarity and alignment of functions could prove to be a barrier in the future. There is also concern that this new structure may undermine the very features—grassroots mobilization and local links—that have made them effective in the past. 2. As part of the new structure of the ESCA Association, and to ensure its long term sustainability, there is now a membership fee for people using the ESCA services; initial help is offered for free. Although the fee is minimal (3 GEL for six months, approximately US$1.50), there are questions about whether people will be willing to pay and how this will affect the poor and disadvantaged who are the key clientele of the ESCAs. This in turn could affect the sustainability of the ESCAs. In the focus group, people indicated a willingness to pay but offered various suggestions about how to encourage payment. They offered the following suggestions: people would be more willing to pay if there was more publicity about ESCA activities; there would be a greater willingness to pay if ESCAs offered help in all services and not just electricity; and paying for each delivered service would be better than a flat membership fee. 33 3.0 LIBERTY PROGRAM EVALUATION 3.1 Liberty Institute Project Partners Liberty Institute formed a consortium of NGOs to implement a project to raise the awareness of journalists, politicians, and the general public on energy sector issues, and promote transparency and accountability in the energy sector. The NGOs in Liberty’s proposal included: Liberty Institute, International Society for Fair Elections and Democracy (ISFED), Radio Green Wave and Research and Development Association (RDA). While implementing the project, Liberty Institute experienced some changes in its partners. ISFED was actually composed of two organizations – Fair Elections Foundation (FEF) and ISFED. FEF was involved in the project until December 2004, and was primarily responsible for regional work. FEF had representatives in Kutaisi, Rustavi, Akhaltsikhe, Poti and Gori. In January 2005, the regional coordinator from FEF worked with Liberty to transfer information and contacts in regions. In February 2005, Liberty Institute hired new regional coordinators and renewed regional work in Kutaisi, Akhaltsikhe and Gori. In Akhaltsikhe, FEF's representatives were rehired; in Gori and Kutaisi, Liberty Institute used its own representatives. In Rustavi and Poti, there were no Liberty representatives. In March 2005, a new region, Batumi, was added. Besides regional work, ISFED was providing mass media monitoring reports until December 2004. Since January 2005, mass media monitoring was conducted by the same group of people under a direct contract with Liberty Institute. Research and Development Association was aided by BCG Research to complete the analytical tasks. However, BCG didn't participate in the project as a formal partner. At the beginning of the project, it became clear that RDA did not have enough human resources to conduct polling in six cities of Georgia. An MOU between Liberty Institute and BCG was signed in which it was stated that BCG would provide the resources necessary to conduct all of the research in the project and guarantee the high quality of research. RDA representatives were involved in research work, but the research work was primarily conducted by BCG staff. Liberty Institute has local offices or representatives within other NGOs in all of the locations for the project activities, and Radio Green Wave had partner radio stations in the following cities: Akhaltsikhe – Green Wave, Meskheti FM 107.7; Kutaisi – Dzveli Kalaki FM 107.9; Zugdidi – Atinati FM 105.9; Poti – Poti + FM 100.5; Lagodekhi – Hereti FM 102.8. In addition to changes in project partners, Liberty Institute experienced changes in project management over the 18 month period of implementation. The first Project Manager, Mr. David Kakabadze, was in place from the project’s inception until June 2004. Mr. George Bakradze was Project Manager from July to December 2004, and Mr. David Lezhava came on board in January 2005 and continued to the completion of the project in June 2005. Several project staffs were on board for the entire period, including the Communications and Outreach Officer, Ms. Veronica Chkadua, and the head of Liberty Institute, Mr. Levan Ramishvili. 3.2 Liberty Program Objectives and Activities 3.2.1 Anticipated Outcomes/Results The stated objectives of Liberty’s program were as follows: 34 “The goal of the program is to foster transparency of the energy sector and promote good governance, accountability of both industry and policymakers to customers, as well as "corporate citizenship" of gas and electricity companies. The program aims to bring increased transparency to governmental processes and actions, though strengthening legal and official mechanisms of accountability and more active involvement of civil society in the decision-making processes. The program will strengthen media's capacity to cover energy reform issues; public outreach campaign will raise citizens' awareness about challenges to be met and through increased understanding will build public confidence into reform process; and finally the program will set up mechanism of public accountability of politicians and parties for their policies and activities in energy sector through catalyzing political party’s positions and debates on energy related issues.” (Excerpt from Liberty Implementation Plan) While Liberty’s proposal identified desired program outcomes and outputs, the proposal did not clearly define specific indicators that would be used to measure outputs. Prior to the evaluation, the team prepared the following table showing program objectives/outcomes, project outputs, output indicators and possible data sources. Table 8. Outcomes, Outputs, and Output indicators for the Liberty Project Outcome/Objective Outputs Output Indicator Source of Data Mobilize civil society around energy reforms and increase public participation in the energy sector Change in public attitude per surveys March 2004 and March 2005 surveys Increase public sector awareness in energy sector reforms: * Improved understanding of the market approach * Propose policy alternatives for enhanced public debate Increase public awareness (Task 1) Reduced public cynicism (Task 1) Change in public understanding Increase in public participation in energy forums March 2004 and 2005 surveys GNERC/MEF and project records More transparency in the formulation and implementation of energy sector reforms Reduced corruption in the sector (Task 2) Change in public perception March 2004 and March 2005 surveys New framework for dialogue and increased understanding between stakeholders in the energy sector Media monitoring Parliamentary energy committee Articulated party positions and enhanced debate; greater understanding of policymakers on energy reform issues Greater accountability of politicians and political parties in energy sector issues (Task 3) Increased political debate on energy sector issues Political party platforms Increased media coverage; enhanced journalist capacity to cover energy issues Increased media capacity to cover energy issues Quantity and quality of media coverage of energy issues increased Media monitoring 3.2.2 Planned Activities of the Program Liberty Institute proposed four areas (Tasks) in which to work to accomplish the overall program objectives. Shown in Table 9 below are the planned activities of the Liberty Institute program as described in their proposal, along with the monitoring and evaluation criteria which were proposed to measure the impact/effectiveness of the project, and activity indicators. The proposed M&E criteria were included in the Liberty’s proposal, while the activity indicators were developed by the evaluation team. 35 Table 9. Tasks, M&E Criteria, and Activity Indicators Task Areas Proposed M&E Criteria Activity Indicators Task 1. Raising public awareness, raising public confidence • Baseline study • Town Hall meetings/ Energy Round Tables • Newsletter • Information sheets • Web site • Alliance of the entities will serve as a brilliant means to implement advocacy campaigns for issues of significant public interest. • Round tables will stimulate discussion on issues. Active policy makers will be more accountable to consumers. • 10 Town Hall meetings will involve people from different regions of Georgia and from Tbilisi in the reform process. • Website will serve as permanent source of information for additional public involvement initiatives after completion of the project. • Newsletters will provide information for interested parties about ongoing trends in energy issues. # Town Hall meetings # Round tables held # newsletters produced Current Web site # alliances for energy issues Task 2. Anti-corruption • Hotline • Monitoring • Support for policy-making, rule-making and legislative drafting and lobbying. • Monitoring and FOI audit promotes good governance and set the principle of Rule of Law. • Project will identify necessary legal documents and amendments to create proper environment for reforms. # FOI requests # policy papers and advocacy Task 3. Accountability of political parties • Vote Smart • Political awareness test • Civic ratings • Vote Smart will offer information to voters that will enable them to judge the skills and positions of individual politicians. We will monitor how many citizens have accessed Vote Smart information. • By means of Political Awareness Tests the volume of meaningful feedback from candidates will be evaluated; the number of people filling tests and polls as well as the consistency of party positions on energy and corruption issues will be evaluated. Where no feedback was solicited research of the relevant reasons will be carried out. # citizens accessing VS info # people responding to tests and polls Task 4. Increasing media capacity • Press Club • Radio Programs • Reporters’ Resource Center • Training and awards program for journalists • 18 Press Clubs will be conducted, which will encourage media representatives to work in energy-sector. • Results of media monitoring will accumulate necessary information disseminated in media. Monitoring will expose deficiencies in media coverage and show the direction for better journalism. Media monitoring will trace the quantity and quality of press and mass media coverage of related issues. • Reporters' Resource Centre will enable interested persons to get necessary information and keep pace with ongoing development in sector. # Press Clubs Media monitoring # of times resources were accessed # trainings Journalist awards program held 36 3.3 Activities Completed by Liberty Table 10 gives a summary of the activities that were planned and conducted by the Liberty Institute under the programs. Each of these is discussed in more detail in this section. Table 10. Tasks, Activities, and Achievements Task Activity Planned Conducted 1 General Public Survey 2 surveys of 1200 Households 2 surveys of 1200 Households 3 Civic Rating 200 100 3 Political Awareness Test 500 200 3 Survey of Energy Experts 0 405 (formed basis for database) 3 Focus Groups 3 3 1 Web Site Maintenance 18 months 18 months 4 Media Monitoring 17 months 17 months 4 Press Club 18 sessions 18 sessions 4 Radio Programs 72 (weekly programs) 73 1 Advertisement Production Yes Yes 1 Social Advertisement (public service announcements) 16 18 1 Promotional Materials 2500 Approximately 900, including pens, t-shirts and notebooks 2 Hotlines 5 regions * 16 months = 80 months operation 65 months (in regions) + 15 months (in Tbilisi) = 80 months 1 Newsletter 6 issues 7 issues 1 Information Sheets Yes 1500 (user guides) 2 Support for Policy Making 16 months 4 months 2 Support Legislative Drafting 16 months 4 months 4 Awards Program for Journalists Yes Yes 1 Town Hall Meetings 10 9 * Note: The evaluation team verified the completion of the tasks as reported by the Liberty project team through interviews and document review. In this section, we will note areas where Liberty’s work was particularly successful, where questions remain about the completed work, or where the work could have been more effective. Task Area 1: Raising public awareness, raising public confidence Town hall meetings/energy round tables A total of nine town hall meetings were held in the regions of Georgia. The objective of the town hall meetings was to “create a coalition of community-based organizations and active citizens, NGOs, media and energy stakeholders in order to foster the process of cooperation and to bring coalitions to constructive dialogue” (project final report). According to Liberty staff, the most active and well-attended town hall meetings were held in Kutaisi and Alkhaltsikhe following the introduction of communal meters in both cities. Citizens were dissatisfied with communal meters and were not well informed prior to their installation. Both the Ministry of Energy and UEDC participated in the meetings, and were able to present their views and listened to consumer complaints. Findings from the focus groups in Akhaltsikhe and Kutaisi provided additional insight into the tensions that exist in the regions regarding power sector issues. Focus group participants were 37 quite outspoken in their dissatisfaction with information provided by UEDC and the way that customers are treated. Although collections in the regional areas of Georgia have increased from 20% to 80% in the past year customer service has not improved—customers are upset that communal meters have been installed without their knowledge and agreement. Focus group participants also expressed their frustration with the ability of UEDC customer service representatives to solve problems, stating that only management was authorized to resolve any problems. Typical of comments from the focus groups are the following: “We have to pay 60 GEL each month according to the meter’s readings. Even if we pay in full, but our neighbors do not, then the power is cut for everyone. And with each cut in the supplied energy, our debt continues to grow as the cost os the energy is evenly divided among all customers. This system of payment has pitted one neighbor against another.” An employed woman, 50, Akhaltsikhe “Service center does not provide adequate information, customers are not satisfied with the quality of received information. There is an information deficit”. Female student, 21, Kutaisi “We are only given two reasons for cuts: poor payment and if we object, then breakage. It’s no use asking for information”, Woman, 31, Kutaisi In this environment of distrust and inadequate information, the town hall meetings organized by Liberty brought the parties together and provided a forum for them to voice their complaints. The Liberty project ended shortly after the widespread installation of communal meters, but was able to address the immediate concerns of residents in both Kutaisi and Akhaltsikhe through the town hall meetings, and ease tensions in the short-run. However, there is still a strong need for an intermediary in the regions, particularly in regards to billing and collections issues related to communal meters. While UEDC is planning to install individual meters in the future, there exists today a distrust that needs to be addressed through a program of information and assistance to residents so that they can understand their electricity bills. Baseline study and follow-up survey A baseline survey was conducted in March 2004, and a second survey was completed in March 2005. Both surveys included 1200 households and were conducted in Tbilisi and six regional cities (Gori, Rustavi, Poti, Zugdidi, Akhaltsikhe, Kutaisi). A comparison of the two questionnaires and the survey results available in English shows that there were some similar questions, but that the two surveys are not identical and have somewhat different approaches. The first survey was broad and included many energy sector issues, not only electricity. The second survey was a joint effort between CORE (supporting the public outreach program of the MEF) and Liberty. While some survey questions are similar, the survey results can not be used to make any definitive statements about the impacts of Liberty’s program. The surveys were intended to be the primary output indicator. In reviewing the draft evaluation report, the Liberty team commented that nearly all of the questions in the March 2005 survey were the same as the March 2004 survey, however, the evaluation team was unable to verify this from the English version of the survey questions made available. While the surveys could have provided important information to inform the evaluation, we were unable to compare them. Liberty produced a summary of the two surveys (available in Georgian), which was published in one of the project newsletters and distributed to energy stakeholders through the regional Liberty offices. It is not possible to directly attribute any changes in answers to questions 38 relating to corruption or public perception of the energy sector to Liberty’s activities; rather, any changes are more likely due to a combination of Liberty activities and other developments in the energy sector. Newsletters Liberty produced seven issues devoted to energy sector concerns. It was not possible for the evaluation team to comment on the quality or content of these, since they were only available in Georgian. However, the distribution channel may not have been the most effective – the newsletter was included in Liberty’s monthly magazine and distributed through those channels, and to energy sector stakeholders. While the newsletters likely provided good coverage of development in the sector, their effectiveness is not possible to measure as a stand-alone activity. Web site The Liberty Institute web site was maintained for the whole period, and according to Liberty staff, the project newsletters and survey/focus group research results were available on the site. A search of the English version of the web site could not verify this, possibly because the materials are only available in Georgian. The audience for project information on the web site is primarily external or one that is already aware and informed on energy issues. Integrating web site content into the journalist training could have enhanced its effectiveness and use. In the meeting conducted with journalists who had participated in the media training, journalists did not seem to be aware of the energy sector content on Liberty’s web site, or how they could use the material to inform and improve their reporting. They were especially interested in the database of energy experts, which they felt could be a valuable resource for research and understanding the issues involved in energy sector reform. Promotional and information materials Liberty produced a variety of promotional materials and information guides. The focus groups were aware of the t-shirts, which were distributed at the town hall meetings. The slogan chosen by Liberty was remembered, as indicated by the following comment made in the focus group: “The energy company representatives told us to take the old meters outside, and to organize payment in order to have power. People had fulfilled their obligation: while previously only 20% had paid, now it is 80%, but there were still no results. People learned of the Liberty Institute slogan: ‘Forget the dark past, work for a brighter future.’ People do what they can to save money by economizing on everything, and if they pay their bills, then the second party in such an arrangement must live up to its obligations as well.” Focus group participants were aware of the consumers’ guide produced by Liberty, but were not willing to study it and felt that the guide would not be effective as a resource for helping them to solve problems with the distribution company. Task Area 2: Anti-corruption Hotline 39 Table 11 below shows the months that the hotline was in operation. As can be seen in the table, the hotline operated in only three of the seven districts on a continuous basis. In addition, it is unclear whether complaints or calls to the hotline were resolved or dealt with on a systematic basis. There were many calls to the hotlines, and a summary of some of the issues is included in Appendix D. There did not appear to be a detailed record of all calls to the hotline or whether callers concerns were addressed. The Liberty team expressed their experience, backed up by the focus group findings, that UEDC did not appear to have a consistent or reliable mechanism or approach for dealing with customer complaints. Liberty also provided feedback in their review of the draft report that many customer concerns or requests for information were addressed during the call, and others were passed along to UEDC. Few of the focus group participants were aware of the hotlines. Monitoring Liberty’s proposal refers to monitoring corruption in the energy sector, but it’s not clearly indicated what is meant by this or how the results of the monitoring are to be used. In the opinion of the evaluation team, this activity seems to have been de-emphasized because of the Rose Revolution and the ensuing changes both within the government and in Liberty Institute. Support for policy-making Liberty participated in the drafting of bankruptcy law relating to the energy sector that provides for protection from creditors while the companies are restructuring. This law extended the period that energy sector companies’ assets were protected from three months to one and a half years. Liberty reported that many energy companies made use of this law in the past two years. Table 11. Hotline Operation Data April 2004 May 2004 June 2004 July 2004 August 2004 September 2004 October 2004 November 2004 December 2004 January 2005 February 2005 March 2005 April 2005 May 2005 June 2005 Axaltsikhe Kutaisi Lagodekhi Zugdidi Puti Batumi Gori Task Area 3: Accountability of political parties Vote Smart The Vote Smart campaign was designed to help citizens make well informed decisions during elections. The parliamentary elections were held before the project started, so the original 40 design of this task was no longer useful. However, Liberty attended meetings at the parliamentary committee where energy issues were discussed and Liberty staff monitored main political decisions in the sector. Information on important issues was distributed by e-mail and/or published in Liberty’s newsletter. According the Liberty Project Manager, the survey of politicians, which was conducted under the political awareness test activity, should be considered an integral part of the Vote Smart campaign. Political awareness test The political awareness test aimed to study the attitude of politicians toward certain energy issues and to establish the degree of their fairness and sincerity. A survey of 200 politicians was conducted in Tbilisi, and the survey results were reported in Liberty’s newsletter. Civic ratings Civic ratings aimed to reveal the opinion of different layers of society concerning energy issues and politicians. A survey of 100 representatives from 37 NGO's was conducted. The objective of the survey was to study the vision of NGO's regarding the reforms, the current situation, and expectations in energy sector. The results of the survey were shared with energy sector stakeholders and published in the Liberty newsletter. Task Area 4: Increasing media capacity Press Club The Press Club was a roundtable discussion by energy sector stakeholders that was broadcast on radio. Important topics such as rehabilitation of Georgia’s power plants, privatization of the energy system, energy consumer rights, and social safety nets for consumers were discussed and debated. Radio programs Weekly radio programs were held on different energy sector issues. The format for these programs allowed for audience call-ins. According to project participants and others interviewed, one of the major areas covered was the popularization of public participation in the energy sector reform process. Reporters’ Resource Center Liberty maintained its project reports and other resource materials that were available for journalists. Unfortunately, this did not seem to be a well-publicized or utilized resource. Training and awards program for journalists This activity consisted of a series of training sessions that were held for journalists on energy sector issues. The first two sessions provided general information on developments in the energy sector in Georgia, which were followed by other sessions devoted to specific issues. Table 12 below provides the list of training sessions that were held. Both the trainings and awards programs were highly regarded by journalists, as evidenced by their comments in a journalists’ meeting organized as part of the evaluation. 41 Media monitoring Two radio stations and five leading newspapers were monitored over a 17 month period to track their coverage of energy sector issues. Each month, a report was produced (in Georgian) that summarized the coverage. This information was relayed to energy sector stakeholders, including the MEF, and was used to develop outreach strategies by the MEF. 42 43 Table 12. Journalist Training Sessions Training Date Main Invited Expert Name Institution Thematics 1 5-Dec-04 Buba Tsirekidze & Vasil Khorava CORE Int & Young Energy Experts Association Main Mechanisms of functioning of the Georgian electricity system and its problems 2 29-Jan-05 Buba Tsirekidze & Vasil Khorava CORE Int & Young Energy Experts Association Main Mechanisms of functioning of the Georgian electricity system and its problems 3 26-Feb-05 Aleksander Khetagury Ministry of Energy Future plans for reforms 4 2-Apr-05 Liana Jervalidze Independent Expert Central Gas Pipeline 5 16-Apr-05 Paata Lomsadze and Ketevan Kinkladze Center for Enterprise Restructurisation and Management Assistance Benduqidze’s Model of Energy Sector Privatization 6 14-May-05 George Tavadze and his staff GNERC (Regulatory Commission) GNERC’s achievements and problems 7 21-May-05 Kaplja and his staff Telasi Telasi Achievements and Problems 8 11-Jun-05 UEDC’s staff UEDC/PA staff UEDC/PA Achievements and Problems 9 Joseph Natroshvili GWEM (Georgian Wholesale Electricity Market) Possible reform directions and GWEM’s problems 10 David Morchiladze and his staff Tbilgazi (Gas distribution company) Reforms and prospects of distribution company 3.4 Evaluation Findings 3.4.1 Effectiveness of Liberty’s Program The Liberty Institute program pursued many different types of activities that ranged broadly both geographically and by intended target. While each of the activities could play a role in increasing transparency of the sector and reducing corruption, the impact and effectiveness of individual activities is impossible to measure with the available information. At best, anecdotal evidence is presented. Comments on specific activities that were made as part of the evaluation include: • The media capacity-building program strengthened contacts between journalists and key players in the energy sector. The journalism program was highly regarded by journalists and stakeholders who participated in the training and awards programs. The database of energy sector experts that was created was not well publicized or utilized; journalists were not aware of the existence of the database. • Web site, newsletters and energy user’s guides did not have a clearly defined target audience or distribution channels. Materials were distributed to subscribers of regular Liberty publications, and stakeholders in the sector with an interest in energy issues. Focus group participants were not generally aware of the publications. • Media monitoring provided good information for energy stakeholders, and was used in planning the winter energy campaign of MEF. Media monitoring is being continued under the CORE contract, and so must have had some usefulness to the stakeholders. • Energy stakeholder meetings were held jointly with all key players in the energy sector. Several people mentioned that these were useful forums, and because they were held regularly, there was the necessary continuity to tackle important issues. • The Liberty program created space for public dialogue through radio programs, radio bridges, town hall meetings, roundtables, etc. However, it is difficult to measure the effectiveness of this outreach. Call-in shows received a lot of calls, as did the hotlines, but it is unknown whether the complaints and concerns expressed in these forums were resolved. If not, over the long-run, consumers could experience higher levels of distrust in the forums, and perhaps also in energy sector institutions. • Liberty’s town hall meetings would immediately diffuse tensions in a highly politicized environment, but these were not sustained interactions that allowed citizens to resolve their problems. Often, Liberty became the intermediary for consumers who began to come to them for help. However, this role seemed to be taken on as an urgent situation arose, rather than on a sustained basis. Town hall meetings brought together representatives from the distribution company, MEF and NGOs to discuss problems and provided a forum for complaints to be expressed. The town hall meetings were held at a critical juncture in the development of the electricity sector in the regions; during the period of installation of communal meters. Despite UEDC publicity campaigns, residents in the affected communities were not adequately prepared for this change, and were outraged and confused. Liberty provided a forum in the town hall meetings to vent this anger and clarify issues. 40 3.4.2 Efficiency of the Program The question of efficiency that is addressed in the evaluation is one of efficient use of resources. Given the monetary and human resources available to Liberty Institute, were they used efficiently? Prior to award of the cooperative agreement, Liberty underwent a financial audit to assess whether the record-keeping systems were adequate to track expenditures in compliance with USAID guidelines. Concern was expressed about the tracking of expenditures on this project separately from other Liberty projects. Findings of the evaluation concerning efficiency include the following points. • Changes in project management and partners affected the continuity of the Liberty program’s activities; Liberty’s first Project Manager left to work more closely with the government. Impact of ISFED suspending operations in Georgia was that new partners needed to be identified in the regions. This may have contributed to some of the political work not being done as designed (Vote Smart, Civic ratings, policy work), as did the change in government following the Rose Revolution. • Staffs in the regions who were assigned to the project were working on other Liberty projects at the same time. This may have led to some confusion about the Liberty role in energy sector since the same people were also working on criminal justice, education and communications. Liberty staffs were well known from their work in these other areas, and for their commitment to public participation, transparency and dialogue regarding changes that are occurring not just in the energy sector. USAID expressed some concern regarding the allocation of staff resources between projects, and Liberty staff in the field were unable to state how much time staff spent on the USAID project vs. other projects. • Transfer of information from hotline calls appeared to be bureaucratic (from regions to project management in Tbilisi, which sent information to UEDC in Tbilisi and then back to regions). It was not clear if clients got a direct response to their query, although likely some inquiries were addressed directly in the calls. A more efficient mechanism would have been to have direct communication and transfer of information to UEDC offices in the region or town, to be dealt with by the local UEDC office. 3.4.3 Program Management and Relations with External Partners Both the Liberty and CENN projects ended up collaborating on activities such as radio programs, public service announcements and publications. USAID did not expect this collaboration, but it was mutually beneficial. Both programs collaborated with the MEF in stakeholder forums, which were rated as an effective activity by several stakeholders interviewed. Comments relating specifically to Liberty’s relations with external partners include: • Partnerships with different NGOs in the regions may have undermined Liberty’s brand identity. In one region visited by the evaluation team, Liberty was well-known and respected. In the second region visited, Liberty’s activities were known under the umbrella name of a different organization, although the specific individual responsible was the same under both NGO names. The activities undertaken by Green Wave in the regions may not have always been associated with Liberty Institute. While we can not 45 categorically state whether this had a positive or negative influence on project outcomes, the lack of brand identity may have led to confusion and the lack of a unified message. • Liberty never received strong support or interest from UEDC; their involvement and support was variable, especially at the beginning. It appeared from comments made that Liberty and UEDC had a somewhat antagonistic relationship at times. UEDC felt that Liberty was sometimes acting as a strong consumer advocate, not taking the impartial role of mediator. Towards the end of the project, the relationship between Liberty and UEDC improved when Liberty helped to diffuse anger in the regions where communal meters were installed. One complicating factor for Liberty was that UEDC managers were often changing, on average every six months. This made it difficult for Liberty to liaison between consumers and UEDC. • Liberty had a strong partnership with MEF and the CORE public outreach and participation project. Together, they collaborated on town hall meetings, public service announcements, radio programs, and the March 2005 survey was revised to include questions that were important to MEF. • Liberty did not have much contact with GNERC, potentially because regulatory issues seemed to have less relevance in rural areas. Liberty could potentially have advocated for consumers in the communal meter hearings held by GNERC, however, these hearings were held after the project ended. 3.4.4 Barriers Faced by Liberty Barriers relating specifically to Liberty’s project include: • Liberty project objectives were quite broad and diverse. Activities were widely spread and difficult to merge into a cohesive program whose impacts are measurable. • Liberty was working in a difficult environment in the regions, covering a large geographic area. The issues faced by consumers in the regions were also more difficult. Liberty was too thinly staffed in the regions to expect major results or impacts from the regional work. • UEDC’s interaction and support was variable throughout the project, which hampered Liberty’s ability to influence either the flow of information that consumers received or to address consumers concerns. In addition, UEDC’s policy of continually changing managers in the local distribution offices made keeping them abreast of the project difficult. • Liberty’s political affiliations may have been a barrier to how people perceived them and their effectiveness as independent representatives or mediators. Conversely, the visibility and high profile of Liberty Institute may have been a positive influence on being able to deal with consumer complaints. It is difficult to judge the impact of Liberty’s reputation on program activities. • Liberty staff leaving to join the government may have undermined the continuity in the program both from the view of immediate staff vacancies and planned activities. When the first program manager left Liberty, policy advocacy was essentially dropped from the program’s activities. 46 4.0 CONCLUSIONS AND RECOMMENDATIONS 4.1 Barriers Faced by Both Projects The transitions within the energy sector in Georgia placed significant barriers in place to achieving the USAID objective of “increasing public understanding of, and participation in, decisions regarding delivery of energy services.” Some common barriers that both projects faced include: • There is a lack of understanding of the role of public participation within government. o GNERC sees public participation as being about information sharing rather than listening and collectively solving consumer problems. Ombudsman is seen to be “an island rather than connected” to the system of problem resolution. o In the Government there is a wariness about how public participation can undermine the Government’s broader policy agenda, since the public lacks technological knowledge about the sector. o Distribution companies view public participation as being about disseminating information. • Public participation in the new government was also given a low priority as the current government is “fighting fires.” o Transitions within the sector have affected the priority given to public participation. The Government’s priorities are constantly changing and its focus has been on privatization, new metering strategies, and dealing with perception of corruption of the sector. o The Rose Revolution has led to a higher public expectation of government, with high potential for failure and disappointment. The current government may discourage public participation in its urgency to show progress on important issues. 4.2 Support for the USAID IRs A variety of groups organized from throughout civil society, with an interest in electricity issues, can offer a system of checks and balances for the power sector. This can potentially improve efficiency of the sector; building up public trust is an important part of the latter. Both programs supported an increase in public trust in the energy sector through outreach and information activities (radio programs, energy roundtables, PSAs, consumer guides, regular multi￾stakeholder meetings). In addition, the CENN project contributed to developing public trust in the following manner: • The project ensured that information about different elements and processes of the electricity sector were easily available to the public. • The ESCA activity eliminated customer frustration by helping them solve long term problems, specifically issues of debt and incorrect billing, which has prevented them from having regular access to electricity. • There was greater transparency about billing and collection process. • The ESCA monitored theft and fraud issues more effectively, which gave customers more confidence in the service they were receiving. • Telasi gained greater visibility and now has plans to undertake more public outreach activities that would improve public trust. 47 • Customers saw real problems solved such as the problem of electricity lighting which also helped build public trust. The Liberty program supported an increase in public trust particularly through the journalist training and media capacity-building program. Liberty also played an important role as intermediary in UEDC areas where tensions were high due to communal metering. The resolution of consumer complaints was not assured through the establishment of either the hotlines or UEDC’s customer service departments; however, Liberty provided a forum for diffusing tensions. 4.3 The Projects: Successes and Limitations The successes of the two civil society projects included: • Strong collaboration between the EGAT/EIT/E team based in Washington and the Democracy and Governance team in USAID Georgia. Coincidentally, at the time the APS on energy governance was announced, USAID/Tbilisi was already thinking about issues of electricity governance due to the local situation brought to the Mission’s attention by PA Consulting. To address this issue, Save the Children took the initiative to start brain￾storming with different NGOs. Save the Children was in a good position to help CENN and Liberty with their proposals, contributing to their success in winning these international awards. • The ESCAs have been established as viable intermediaries that offer essential services. They inform consumers, resolve problems, and represent consumer interests in various public settings. The ESCAs’ success can be gauged by a number of factors including: key stakeholders view them as a legitimate representative of consumers’ interests; there is demand for similar organizations in the regions outside Tbilisi; and consumers who have interacted with ESCAs believe that they are honest brokers between them and electricity sector corporations. • CENN’s “natural leadership approach” and the extensive community mobilization leading up to the establishment of the ESCAs was a critical element in their success. ESCA representation at stakeholder roundtables gave ESCA leaders the opportunity to interact with other key stakeholders in the energy sector on equal footing. This improved their credibility and facilitated their ability to be effective intermediaries. • The ESCAs’ strategy of solving concrete problems, specifically the issue of electricity debt with Telasi, helped them achieve credibility and legitimacy within the broader community. • CENN’s school campaign on energy awareness was extremely popular and successful. Discussions are underway to develop this component so that it can become part of the national curriculum. • Through concrete grassroots actions, the ESCAs were able to ensure some accountability and transparency at the local level, and through that process alleviate some distrust. Their role as consumer advocates at the local level also paid dividend; at the national level they have not yet been tested. 48 The limitations of the projects included: • As compared to CENN’s project, the Liberty project was overly ambitious, particularly considering the more challenging environment that Liberty confronted in the regions. Liberty failed to predict the need for mediation between consumers and the distribution company, UEDC, in the face of new reforms under PA Consulting’s management contract. Consequently, although it did at times play an intermediary role, it was often not adequately prepared to undertake this role • The formation of the ESCAs into a national association with an expansion of its role both functionally (from electricity to a range of services) and geographically (from six neighborhoods to all of Tbilisi) may be too rapid. It is likely that the ESCAs will not be as successful, as there has not been grassroots mobilization, or extensive interaction with key stakeholders in sectors other than electricity.7 4.4 Role of Civil Society Initiatives in the Energy Sector in the Future At the onset of these two projects, Georgia faced a unique situation in the electricity sector. A decade of transition and uncertainty in the electricity sector – a reflection of a broader economic and political transition occurring in Georgia – produced an atmosphere of poor transparency and corruption, generating suspicion, distrust, and anger among consumers. Between 1997 and now substantial reforms have taken place.8 The reform process, as expected, has been fraught with difficulties. Uniquely, Georgia also faced economic transitions at multiple levels, ethnic conflicts on the border, and a decade of economic hardship resulting in limited to no electricity service. Consumers’ experience with privatized electricity in Tbilisi through AES Telasi was incongruous. Increasing tariffs (2.1cents per kWh in 1999 to 6.4 cents per kWh in 2003) and the introduction of disconnections for non-payment accompanied poor to non-existence electricity service and unreliable billing. This produced a real crisis of confidence, often manifested in people on the streets of Tbilisi. Outside Tbilisi reform arrived more slowly and the project period (November 2003-June 2005) corresponded with UEDC operations being placed under a management contract. This brought with it substantial changes for consumers. Often these reforms were poorly communicated and had limited public involvement, generating a similar crisis of confidence. It was within this context that the CENN and Liberty projects were introduced. Although things have improved substantially with the Saakashvili government, particularly in Tbilisi, there are still challenges ahead. Most immediate is a 50% tariff increase. In order to smooth the transition, what is required is energy sector governance that offers consumers transparency, accountability and interaction with electricity sector entities. ESCAs have proved effective as grassroots organizations that offer consumers some recourse for their problems in the electricity sector (and in some other sectors as well). This has been 7 Since November 2005, the six ESCAs have been united into one association. The organization has achieved some success in solving communal problems other than electricity. According to CENN, in the period November 2005 to March 2006, the ESCA has solved 106 electricity, 51 waste management and 57 water related problems. In addition, the ESCA has begun to establish relationships with other key stakeholders in water (Tskalkanali) and waste management (Civil Services Department). Also according to CENN, the ESCA will work at both the grassroots level and the policy level, with the ESCA’s personnel working with both decision-makers and the target groups. 8 The industry had been unbundled; there has been privatization of the Tbilisi distribution system (AES Telasi) which has also been re-sold (RAO Telasi); a regulatory commission had been formed and new electricity legislation passed. The reform process continues with frequent adjustments. The most recent is the government’s intention that UEDC will be sold to private investors by the summer of 2006. 49 particularly so for less well-off consumers in neighborhoods. At the neighborhood level, ESCAs work to resolve people’s problems, help them restructure debt (resulting in the reconnection of electricity), and provide consumers a means through which to transverse the maze of Telasi bureaucracy. Through this interaction, consumers feel more confident about resolving problems with the distribution company and are empowered. ESCAs have also represented consumer interests at government bodies, specifically at GNERC. However, the evaluation team found little evidence that they have been particularly effective. This may be the result of two factors: that the ESCAs are at an early stage of their development and as yet are not capable of playing such a role; and the regulatory process in Georgia is weak and there is little opportunity for consumer interest groups to make meaningful contributions to policy making. Nevertheless, the ESCAs clearly have acquired a national presence. Even though the Liberty project did not plan to play an intermediary role between consumers and the UEDC, circumstances – in particularly the communal metering issue – made intermediation necessary. Despite the lack of preparedness for this activity, Liberty’s interventions were often beneficial. Local Liberty offices represented consumer interests in resolving some of these problems. A common suggestion that we heard in the field was the need for ESCA-type organization in the regions. This suggestion came not only from Dean White, Chairman of UEDC, but is also self-evident by the number of communications CENN received from the regions asking for their help in organizing similar organizations. As Dean White stated: “ESCA stands a better chance of success in this area through providing improved information and an organizational structure for customers that can collectively pursue an objective. The utility and regulator are more likely to be influenced positively by an organized group of consumers, with well crafted arguments, than various individuals trying to do so separately. Secondly, the ESCA can be useful for communal metering/billing. For instance, there is a service agreement between the utility and the communal grouping that should describe the roles and responsibilities of the parties. Under the regulatory guidelines, customers are responsible for allocating usage amongst themselves, or the utility can use an equipment inventory for each customer to allocate the bill if the customers are unable to agree. This is difficult for customers to comprehend and support could be useful in this area. Further, an ESCA could assist with ways for each community to better design their own allocation scheme. For instance, in some areas, there is old metering equipment which although too inaccurate for individual billing purposes could nonetheless be "cleaned up" for a couple of GEL per meter and provide a basis for an allocation. Or, the community could pursue some further submetering (to reduce the number of customers served by a single meter). Thirdly, it appears that the GoG will pursue a major tariff increase later this year (even 50% or higher). This will be a shock to many and adjusting to this will take time. Perhaps an ESCA could also work with communities to identify localized options to moderate the impact, such as ways to improve energy efficiency, or looking for areas to promote fuel substitution (email communication).” There is clearly a need to consider the establishment of community-based organizations like ESCAs in the regions. As noted above, communal meters are still causing tension and are not widely understood or accepted. There the tariff increase planned will inevitably prove contentious. Even more challenging is the upcoming shift from communal meters to individual meters and the planned sale of UEDC to a private investor in the summer of 2006. All of these changes will be challenging for consumers who have already had to accept significant changes in their electricity supply. In such a context, a third party intermediary would prove useful. 50 The ESCA too is at a crucial stage. The recent formation of the national ESCA and their expansion both geographically and functionally may undermine their very strength as grassroots organizations. Geographically, ESCA leaders continue to be located in their local neighborhoods and have a presence in their local communities. However, the centralization of the ESCA offices as well as ESCA leaders’ involvement at a more national level may undermine their local links. ESCAs were always active in a number of sectors beyond electricity. However, their official expansion in these areas without the necessary lead up training – particularly familiarity with each sector and their key stakeholders – could lessen their effectiveness. CENN justified the formation of the association for both purposes of sustainability, as well as for the purpose of moving ESCAs center stage on the public policy arena as the consumer organization. Although these steps are very justifiable, much more attention needs to be given to how ESCAs will manage these multiple roles, and whether the existing proposed structure is the only one to ensure the ESCAs’ effectiveness and sustainability. ESCA-type organizations clearly serve two functions and are relevant both in a transitional and in more stable electricity environments. They can be a conduit for public information, but more importantly they can represent organized consumer interests, essential in an efficient democratic process. Such consumer organizations are common in the United States and other well developed electricity sectors and they are supplementary and necessary institutions to other bodies, like utility companies’ customer services and public relations departments, government and regulatory ombudsman. As we saw in the evaluation, existing outreach programs serve an informational need or a social marketing approach rather than actively resolving problems that consumers face. UEDC and Telasi have customer service departments, but from the comments heard in the evaluation, it was clear that there remains a lot of distrust and lack of satisfaction. Although there is clearly a demand for information – PA survey and BCG surveys consistently demonstrated that the public wants more information about the energy sector – there is a need for a consumer representative body. Although a public advocate exists at GNERC, it is not an easy forum to access. The consumer advocate is there to help resolve problems with different arms of the sector, but is a forum of last resort. Resolution at a lower level (i.e. the neighborhood levels, the level at which ESCAs work) may be more efficient and less contentious. The effectiveness of an organization like the ESCA will depend on the environment for public interaction in the electricity sector. Currently, this environment has been weakened. The Government has ambitious and contentious plans for the sector and is wary that their policy agenda will be undermined. They worry that the public lacks the necessary knowledge to be constructive in the policymaking process and, by participating, will slow the pace of reform. Under this government, GNERC has been weakened, with more power shifting to the Ministry. At the same time, although the Government is open to public participation – as stated under the new Amendment to the Electricity Law on market rules – it is unclear how this participation will be structured. 4.5 Recommendations 1. Support for the ESCA should continue for at least another year. During this period, a means of self-financing needs to be found to allow the ESCA to be institutionally and financially sustainable. Currently, the ESCAs are trying to establish a user fee model. However given their client base, which is mainly consumers at the lower income spectrum, this is not likely to be a viable approach. 51 2. There needs to be a review of the ESCA’s role and the level at which it can function most effectively as a lobbying organization. Currently, the ESCA Association is covering multiple roles- condo/neighborhood association and city-wide advocacy organization. It likely can not fulfill all of these roles successfully. 3. There needs to be a carefully consideration of the expansion of the ESCAs to the UEDC area. One of the possible suggestions has been to utilize the ESCAs themselves to lead the formation of similar organizations in the UEDC area. However given the current limited capacity of the ESCAs, it is recommended that they not serve as the primary drivers of this activity. Furthermore, this may also not be sensible as it contradicts a central platform of CENN’s success in establishing the ESCAs in Tbilisi. The success of the ESCAs has largely been due to local mobilization and the process of selecting ‘natural leaders’. Knowing the community and building confidence in the project at the local level have been central elements of the ESCAs’ success and will play a key role in their success in the UEDC regions. For the UEDC regions, the formation of the ESCAs will need to follow a similar confidence-building approach. The ESCA leaders from Tbilisi can play a role in training or sharing experiences, but are unlikely to be the leaders in the regions. 4. ESCAs need access to specialized resources, both technical and legal, to strengthen their effectiveness and scope. ESCA leaders mentioned that they need support on internal building issues (elevators, hallway lighting, stairways, roofs, etc.), electricity consumption in internal networks, and legal support to deal with the specific legislation on consumer rights and to strengthen their ability to defend consumer rights. 5. ESCAs can also play a watchdog role at the municipal level, ensuring transparency of municipal services contracts and municipal expenditures on neighborhood improvements. This role needs to be strengthened but must be considered in light of the overall assessment of ESCAs’ role (Recommendation 2). 6. Despite USAID and other donor investments in the regulatory process in the past, there is a continued need to strengthen such activity in order that a public space is created for public participation at the policy level. To this end, training and institution building for GNERC, ESCA, and citizens needs to continue. Currently ESCA leaders participate in the policy process at various levels, particularly with the municipality and GNERC. To strengthen the regulatory process, more training is required both for ESCAs and GNERC. ESCAs need to be taught how to approach regulatory issues and how to participate in the regulatory process. The regulatory body, despite its function, also has a limited understanding of the role of civil society in the regulatory process. The current view is that ESCAs are a medium through which public information can be shared rather than a valid means through which consumers ‘voices’ on electricity service delivery can be heard and incorporated in the policy-making process. New government legislation which will decentralize local government will have implications for how ESCAs can influence the local policy and decision-making process. Consumer organizations like ESCAs can participate in the policy making process at a decentralized level. All these aspects of training and institutional building will strengthen ESCAs’ (and other future consumer organizations) effectiveness. 7. A fully functioning, democratized energy sector requires public participation at various levels. There is a need to develop the capacity of think tanks, academics, and journalists who can be well-informed voices to help ensure transparency and accountability in the sector, and ensure that alternative views are expressed and accounted for in the policy-making process. 52 Appendix A. Evaluation Scope of Work I. Introduction and Background: USAID’s Energy Team (EGAT/EIT/E) has taken a leading role in supporting new approaches for increased public involvement in the sector, with the aim of improving public trust in energy institutions and in collaborative problem solving. Innovative approaches for including the public are especially called for when utilities and regulatory bodies are themselves in a transformative moment. The Energy Team has hypothesized that civil society organizations could play a supportive role in this transformation, assisting energy institutions in identifying gaps in governance processes and service delivery from the public perspective, and improving transparency and accountability in the sector. In this context, the Energy Team awarded cooperative agreements to two civil society organizations in Georgia: The Caucasus Environmental NGO Network (CENN) and the Liberty Institute. These cooperative agreements were cost-shared by USAID’s Energy and Environment Office in USAID/Georgia. These contracts ended in July 2005. II. Scope of Work 2.1 Program Objectives USAID would like an independent assessment of recently completed projects in Georgia in order to monitor the results of its civil society program, make improvements where necessary, and document successes and potential for scale-up in country and replication in others. 2.2 Sub-activity Objectives The research should focus on primary, on–site research, and include interviews with all relevant stakeholders. Stakeholders include the energy institutions that CENN and Liberty Institute interacted with in the course of carrying out project activities. These will include (i) Telasi Distribution Company in Tbilisi, (ii) United Distribution Company in other regions of Georgia, (iii) Ministry of Fuels and Energy (iv) Georgia National Energy Regulatory Commission, (v) relevant municipal office (vi) civil society organizations and the segments of the general public (through focus groups). Secondary documents should be reviewed as necessary, including those pertaining to USAID objectives in Georgia. Interviews with USAID staff from the E&E Bureau are also desirable. 2.3 Tasks The contractor will undertake the following tasks: Task 1: Propose a methodology for undertaking project evaluations. The methodology should specify how impact will be measured and should provide specifics on how the contractor will define and distinguish between project outputs and outcomes. Task 2: Work with USAID’s activity manager to apply this methodology to evaluate the impact of the project’s undertaken by CENN and the Liberty Institute. A research agenda should be developed around the specific problems these organizations were attempting to address, including but not limited to: collections, theft, remetering, input into policy decisions, input into service rules and customer service contracts. These sectoral problems should be distinguished 53 from the institutional objectives of building awareness, credibility and legitimacy with the public, and the public outreach and participation strategies used to address them. Successes should be recorded even if these were not explicitly articulated as goals of the project. Task 3: Carry out an assessment of (i) the effectiveness of the CENN and Liberty projects (ii) the potential for increased effectiveness (iii) barriers in achieving objectives (iv) the benefit of civil society involvement measured against the cost. III. Working Relationships The contractor should develop working relationships with CENN and the Liberty Institute. IV. Functional Labor Requirements Energy specialist with evaluation experience. Political scientist/political economist with energy sector experience. Deliverables: • A fully comprehensive assessment with documented successes, identification of weaknesses, and potential for scale-up (in Georgia) and replication (globally). A final report should be submitted to USAID/Georgia and USAID/Washington – EGAT/I&E/E • Presentations to USAID staff in Washington, DC. 54 Appendix B. List of Interviews Conducted Organization Person (s) USAID/I&E/Energy Team Ms. Davida Wood USAID/ENE and I&E Mr. Walter Hall USAID/Tbilisi Ms. Dana Kenney Ms. Keti Bakradze Perspectives Group Mr. Doug Sarno Caucasus Environmental NGO Network Ms. Nana Janashia Ms. Tamar Pataridze Mr. Nika Malazonia Ms. Irina Kitiashvili Mr. Goga Japaridze Mr. Levon Tavarkiladze (former CENN Project Manager) Liberty Institute Mr. David Lezhava Mr. Levan Ramashvili Ms. Veronica Chkadua Ms. Nino Chava (Kutaisi) Mr. Irakli Bachudadze (Kutaisi) Mr. Gia Andguladze (Alkhaltsikhe) Representatives from BCG Research and Radio Green Wave Ministry of Energy Deputy Minister Khetagury Ecovision Mr. Gia Sopadze TELASI Mr. Valeri Pantsulaia Mr. Levan Murusidze Ms. Nino Tivadze (Varkheteli Districti) Mr. Temuri Tsintsadze (Avlabari District) United Energy Distribution Company Main Office Mr. Dean White Mr. Irakli Elashvili Mr. Avto Dvalishvili United Energy Distribution Company Akhaltsikhe Mr. Nika Tsiklauri United Energy Distribution Company Kutaisi Ms. Teona Geguchadze CORE International Ms. Buba Tsirekidze Ms. Inga Pkhaladze Energy Service Consumers Association ESCA Round Table with all 12 ESCA leaders Georgia National Energy Regulatory Commission (GNERC) Ms. Irma Kavtaradze Ms. Nino Asatiani Ms. Nia Tushkia School #84 Teachers who participated in the CENN educational program PA Consulting Mr. Bob Baumgartner Open Society Mr. Irakli Khodeli Tbilisi Municipality Mr. Iago Dvalishvili Mr. Otar Orgobiani Kutaisi Municipality Mr. Giga Shushania Akhaltsikhe Municipality Mr. Shalva Dalalishvili Journalist Meeting 16 representatives from the media: television, radio and print IREX Media Ms. Lika Chakhunashvili 55 Appendix C. List of Documents Reviewed CENN Proposal and Annexes 1. CENN-ESCA final proposal 2. Questionnaire 3. Stakeholder Inventory and Analysis 4. Mobilization Plan a. 2004 Work Plan b. 2005 Work Plan c. No Cost Extension Work Plan 5. Sample Zoning of Project District 6. Community Selection Criteria 7. ESCA Leaders and Contacts 8. ESCA Work Plan 9. Painting Action in Schools 10. EE Books for Schools 11. Materials for Open Classes 12. Excursion in Telasi 13. Excursion in Gardabani 14. Publication on Electricity Tariff 15. P2 Publication 16. Energy Services Consumer Guide in Tbilisi 17. ESCA’s Brochures 18. Calendar 19. Posters 20. Booklet on Electricity Safety Rules 21. Newspaper Articles 22. Training in Community Mobilization 23. Training in Energy 24. ESCA Training Materials 25. Roundtables Additional Materials 1. Sample questionnaire for rapid assessment in 6 Tbilisi pilot areas 2. Result of Rapid Assessment 3. CENN ESCA budget 6/18/03 4. CENN Financial Report 11/1/05 – 1/15/06 5. Socio-economic Effectiveness of Structural Adjustments in the Energy Sector 6. Interim report 1/13/06 7. PMP 8. Work Plan 2003-2004 9. Final Report 10. Letter of Support to the Caucasus Environmental NGO Network (CENN) by Nick Gilauri, Ministry of Energy Georgia, July 21, 2005 56 Liberty Institute 1. Liberty Institute proposal for ESG RFP - 6-18-03 2. Liberty Institute proposal budget 3. Liberty Work Plan – General 4. Liberty Work Plan 12-03 5. Survey cover questions 6. Electricity Supply Consumer Survey statistical report 7. Liberty Focus Groups – analytical report 8. Final Report Weekly and quarterly reports 1. Weekly Report February 10, 2005 2. Weekly Report February 17, 2005 3. Weekly Report February 24, 2005 4. Weekly Report March 3, 2005 5. Weekly Report March 10, 2005 6. Weekly Report March 17, 2005 7. Weekly Report March 24, 2005 8. Weekly Report March 31, 2005 9. Weekly Report April 7, 2005 10. Weekly Report April 14, 2005 11. Weekly Report April 21, 2005 12. Weekly Report May 19, 2005 13. Weekly Report May 25, 2005 14. Weekly Report June 2, 2005 15. Weekly Report June 9, 2005 16. Weekly Report June 16, 2005 17. Weekly Report June 24, 2005 18. Performance Report #4, October – December 2004 19. Performance Report #5, January – March 2005 PA Consulting 1. UEDC Public Outreach report – “GESI study report on communications with minorities”. 2. 2005 Citizen and Enterprise Baseline Survey – Responses from Telasi Customers 3. 2005 Citizen and Enterprise Baseline Survey – Select Responses from Georgian Customers 4. Communications Strategy for the UEDC, Georgia Energy Security Initiative, PA Consulting, November 2005 CORE 1. “Energy Attitudes and Opinions of Georgia’s Citizens and Enterprises” 2. Gender and Energy Case Study 57 3. White Paper: Issuing “Electricity (Capacity) Market Rules” under Public Administrative Proceeding as Provided by the “General Administrative Code of Georgia” 4. Advisory Assistance to the Ministry of Energy of Georgia: Public Outreach Strategic Plan, Amendment No. 1, April 27, 2005 5. Comprehensive Public Participation and Outreach Plan, Year 2 (2006), December 30, 2005 6. Summary of GNERC Meetings: a. July 22, 2005 b. January 17, 2006 c. July 15, 2005 58 Appendix D. Summary of Hotline Calls Received in Tbilisi and Regions Tbilisi The concerns of residents in Tbilisi were primarily about theft, meters, electricity shortages, and poor service by Telasi representatives. Numerous cases of electricity theft by the customers were identified as one of the major problems concerning electricity issues. Inhabitants of Tbilisi’s various districts expressed their dissatisfaction regarding illegal consumption/theft of electricity by neighbors. The belief is that when neighbors install illegal transmission lines, it affects the supply to paying customers, leaving many without electricity. Eleven of 35 recorded hotline complaints were about theft. Those who do make timely payments to Telasi do not have better service than those who steal electricity, which provides little incentive for those who are legal consumers to continue to pay their bills. There is suspicion surrounding meters in Tbilisi. Customers feel that they have been forced to install the meters at their own cost without the guarantee that electricity will be supplied to their homes. Some believe that Telasi representatives are tampering with the meters to increase electricity charges. Systematic electricity shortages and inconsistent voltage are frequent complaints. There is a belief among callers that supply is determined based on an unfair system of favoritism for the wealthier communities, commercial buildings and businesses. Zugdidi From March until October 20, 72 people called to log complaints at the Hotline. The most acute problem is that the time-table of electricity supply is not being observed. The inconsistent and unplanned supply of electricity causes many problems. In some districts, residents may go weeks without electricity. Callers do not understand why after so many years of attempted reform and promises made, the electricity supply problem has still not been solved. Residents recognize that electricity has become a political issue, and that each election beings promises from government officials of a solution to the supply problem. Once the elections have passed, the issue is no longer addressed. Callers are also concerned about the dangers of illegal cables and poles. The illegal connections not only present dangers due to falling wires and electrocution, but it is also decreasing the quality/voltage of the electricity that paying customers receive. Some of the residents in the region purchase their electricity from commercial suppliers. These callers are dissatisfied with the quality of the electricity, and feel that the money that is being collected for payment is not being credited to their accounts. Poti The major electricity problems in Poti concern permanent shortage of electricity supplied, installation of meters, and the provision of electricity by commercial energy companies rather than the United Distribution Company (UEDC). 59 Some callers feel that they have been forced into purchasing electricity from commercial suppliers by energy company representatives, or by UEDC, which has almost entirely cut off their electricity supply to the region. Those who purchase electricity from commercial suppliers suffer from high electricity costs and poor electricity quality due to un-serviced transmission lines. Residents are unwilling to install expensive meters if they will not be guaranteed electricity supply from UEDC. Kutaisi Meters are a major problem in Kutaisi. Meter installation has not been uniform; some families and entire communities have not been supplied with meters. Some customers have installed the meters, but are still not receiving electricity. Others are hesitant to pay for meter installation if electricity supply cannot be guaranteed. Public protests have occurred in the region, provoked by the distrust that citizens have of the energy company. Residents were told that if they installed meters and make their payments, the electricity supply will improve. Despite the increased installation of meters, and the significant increase in payment collection, the electricity supply has not improved, and the supply time-tables are not being observed. Commercial energy suppliers play an important role in Kutaisi. Despite the high cost and violations in security standards, customers find it in their best interest to purchase electricity from these suppliers, because they provide more permanent electricity than Telasi is able to. Lagodekhi Residents are very concerned that the more privileged areas of town are receiving better electricity service than the less prestigious half of town. A local radio station has received a petition signed by 168 residents who are concerned about this situation. Some residents have been without electricity for months. One caller was concerned because neighbors are using electricity from a hospital, and are receiving consistent electricity supply, while those who do not have the special wires needed to use the electricity from the hospital must do without. 60 Appendix E. Focus Group Summary Top Line report A Group Discussion Survey was carried out by GORBI on behalf of the Academy for Educational Development (AED). The focus group study covered the towns of Tbilisi, Akhaltsikhe and Kutaisi. The study was designed in conjunction with AED experts with the aim of assessing the effectiveness of projects implemented by The Liberty Institute NGO and the Caucasus Environmental NGO Network (CENN). The main objectives of the survey were: • To determine the relationships between electricity distribution companies and customers • To identify problems in the energy service sector • To assess customer knowledge about the energy service sector. An additional topic for research in Tbilisi was: Evaluation of the effectiveness of the Association for the Electric Services Customers (ESCA) Qualitative methodology focus group study was used for collecting information. A total of four focus groups were conducted within the research framework: one each for Kutaisi and Akhaltsikhe respectively, and two for Tbilisi, the capital city. Focus group guides and instructions for survey and criteria for recruiting focus groups participants were mutually agreed upon with AED. Focus group participants in Kutaisi and Akhaltsikhe were selected from people who had participated in town hall meetings conducted by the Liberty Institute. Recruiters were instructed not to invite journalists, politicians, and employees of local electricity distribution companies or representatives of local self-government or NGOs to participate in focus group discussions. In Tbilisi the Association of the Electric Services Customers has been founded in six city districts, which have been divided accordingly into “poor payer” and “good payer” districts. Representatives for the two groups in Tbilisi were selected from a ”good” district (Varketili) and a “poor” one (Avlabari). In both districts the selected participants had interacted with the “Association”/ESCA. Ages of participants ranged from 21 to 71 years. Most participants were unemployed women, in the regions as well as Tbilisi. The profile of the focus groups was nothing out of the ordinary, and reflective of the high level of unemployment in Georgia. Moreover, more represented since they are a socially active part of the population and they handle most problems of communal service ‘utilities’ payments. On average, each focus group lasted for 90 minutes. All group discussions were videotaped. The main survey findings are provided below and categorized with particular headings, supporting research findings and accompanying commentary. 61 MAIN FINDINGS: KUTAISI AND AKHALTSIKHE FOCUS GROUPS: • According to group participants Kutaisi is the city with the most acute energy problem, as indicated by frequent power outages due in part to downed lines. Payment for supplied electricity does not mitigate this problem. Uninterrupted electricity is not guaranteed even when bills are regularly paid. • The regional, state-owned United Electric Distribution Company (UEDC) prefers to use a tough approach with customers. Relations are generally poor because the company usually takes one-sided decisions; customers are left with few options and they can only comply. • The stance is plain: if the bill is not paid then the electricity is shut off. The company is reluctant to postpone payment or negotiate about restructuring payment plans. • Focus group discussions show that payment has improved in the regions. Previously only 20% of the population paid for consumed electricity but now the payment rate has increased to 80%. The power distribution company does nothing to assist consumers, as it is reluctant to take on the responsibility of dealing with energy theft. Rather, it prefers to deal with loss through communal electric meters. The cost of this stolen electricity is then billed to other customers, which naturally angers them and further complicates customer relations; there are also complications and disputes between neighbors as a result of such problems. • The company has no standard policy for dealing with customers. Quite often contradicting decisions are taken on similar requests. Sometimes the distribution company takes the customer’s side, but then acts differently toward other customers in similar situations. • There is no delegation of responsibility within the distribution company. No one at the company service centers is authorized to make decisions or provide any information to customers. People must address the company's management to solve their problems. Arranging meetings with the management (director) at the distribution company is much easier in Akhaltsikhe than in Kutaisi. However, neither in Kutaisi nor in Akhaltsikhe were customers provided reliable answers to their questions. • Local municipalities fail to protect customer rights. As Kutaisi’s mayor stated, “The distribution company is a state within a state.” While many lawsuits against the distribution company have been won in the regions, no implementation on the part of the company followed the court orders. • According to the focus groups, the Liberty Institute and the Akhaltsikhe-based Union of the Democrat Meskhs are the only organizations that have clout with the distribution company. Only with the help of these NGOs can meetings of the company, the municipality and the public takes place. The public would like such meetings to be held on regular basis, as it will help protect their rights should the company fail to keep its promises. 62 • In some cases the Liberty Institute protected customer’s rights and acted as a mediator between the distribution company and the population. However, such instances are rare and, in the opinion of the participants, these instances do not sufficiently alter the existing situation. • According to the focus groups, the public has no information about the existence of the Customer Hotline. Some people said they had seen a copy of the “Guide for Electricity Energy Consumers,” but were unwilling to study the book, being sure it would not help them resolve problems in relations with the distribution company. • Public involvement in the ongoing regional energy sector reform process is negligible. There are no surveys of public opinion or interactive TV programs dedicated to the reform. The public is poorly informed about the guidelines for energy sector development. No useful information is available through press or radio. The information provided by TV channels is concerned only with energy supply problems of the capital, and never addresses regional problems. • Focus group participants think the only solution is the installation of private electric meters, which the distribution company promises in time. Installation was begun in Kutaisi but was suddenly stopped; no one seems to know when the installation will be complete. The public believes that the distribution company is trying to deliberately prolong the process, as in the existing situation the public cannot control the amount of energy they use and that enables the company to overcharge clients. • Some focus group participants think that privatization of the distribution company could be a solution. At the same time, they are aware that unless the state defends the rights of its citizens, the new private company’s conditions may be worse. TBILISI FOCUS GROUP: • The public is trying to solve problems in the electric service sector, but the task is difficult, as institutions remain indifferent to the public and it is difficult to find a common language. • The public cannot defend their rights because Telasi’s public relations service is ineffective. There are too many complicated bureaucratic circles, incompetent clerks, and a lack of transparency. • Customers think that Telasi and Energy State Regulating Company (ESRC) officials are advising customers on their responsibilities rather than helping them understand their rights. • The public perceives ESCA as an NGO that protects their rights; they are also aware that ESCAs suit Telasi’s interests as well. Customers trust ESCAs as a neutral mediator between them and the energy supply utility. They believe that ESCAs study each particular case carefully, clarify the situation for customers, and resolve inadequacies in operational procedures. After the ESCA resolves a customer problem, customers are willing to fulfill their actual obligations to Telasi. 63 • According to customers, ESCA’s activity help save time, as customers are no longer obliged to search for information on various issues of energy or rights for themselves. ESCAs professionalism ensures the positive resolution of cases in favor of their clients. • Customers think that ESCAs are important for people who, for lack of time, adequate health, or age-related problems are unable to address problems in the utilities sphere themselves. It should be noted no such organization previously existed in Georgia. • The public should be better informed of the existence of ESCAs, as most people learn about it from informal contacts. If ESCA’s sphere of competence increases to include other utility sectors, and more people learn of its benefits, then a portion of the public will surely want to join the association, while others will be willing to pay for legal advice and seek help in protecting their rights. DETAILED FINDINGS REGIONAL FOCUS GROUPS The state-owned United Energy Distribution Company (UEDC) supplies Georgia’s regions with electricity. The situation is the most acute in Kutaisi, where most customers pay but the electricity supply is still inconsistent. According to the focus groups from Kutaisi and Akhaltsikhe, payments are collected according to collective meters that measure housing blocks or villages. Sometimes over a hundred customers are served by such communal electricity-meters. The cost of electricity is then evenly divided among them. The public feels that this method has many disadvantages: An employed, woman, 50, Akhaltsikhe: “We have to pay 60 GEL each month according to the meter’s readings. Even if we pay in full but our neighbors do not, then the power is cut for everyone. And with each cut in the supplied energy, our debt continues to grow as the cost of the energy is evenly divided among all customers. This system of payment has pitted neighbors against one another.” Above mentioned payment procedure caused disagreements among population and consequently number of disputes among customers became more frequent. Unemployed man, 24, Akhaltsikhe: “Relations among neighbors has deteriorated because of communal meters. A group of suspicious neighbors broke into my house looking for signs of electricity theft. Their search was in vain.” To ensure fair billing, the public made an agreement with the distribution company to pay for old individual meter installation outside [in the streets]. This has worked in some cases, but has failed in others where the difference between communal and individual meters numbers was substantial Woman from Akhaltsikhe 52, NGO member: “On November 24 we installed old individual meters outside. Previously collective meters we’ve been asked to pay 60 lari per month, but as people were unable to pay we lived for half a year 64 without electricity. Now we pay just 8-12 lari, but we fear someone will steal energy from our line and we’ll have to pay for the theft.” Unemployed woman from Kutaisi, 51: “We did everything we could, even installed old meters at our own expense, but the difference between old and collective meters was sometimes 250 lari. We demanded a line check, but no one was sent. Unlike specialists, simple folks cannot find the perpetrator. Our business is to pay.” However, even the public identified energy thieves, no punitive measures followed, as the distribution company thought it easier to simply collect money from regular paying customers. Unemployed man from Kutaisi, 31: “I found out who was stealing power and informed our substation. They told me to cut the line myself as they had no personnel for the job.” In Akhaltsihe customers used some additional protective measures. Some families united to buy a new collective meter, that would measure electricity consumed by the families and installed it at local substation. Unemployed woman from Akhaltsikhe, 60: “We had been asking our distribution company for a permit to install own collective meters. Finally we were granted the permit on December 11. We had the meter checked by the substation specialists and installed. Since then we have had no problems with payment.” When power is cut customers phone their local substations, but obtaining information is next to impossible. Woman from Kutaisi, student, 21:”Service centers do not provide adequate information, customers are not satisfy with the quality of received information. There is an information deficit.” Woman from Kutaisi, 31: “We are given only 2 reasons for cuts: poor payment, & if we object, then breakage. It’s no use asking for explanations.” There are no public relations services at the substations, and the public is unaware of the obligation of those who answer. Customers complain that the information they receive is inadequate. Man from Akhaltsikhe, unemployed, 26: “The distribution company substations have no public relations services. Different people answer us. Information is mostly unreliable. Rules once set are often changed.” Public access to the main office is most difficult in Kutaisi, where customers managed to meet with the top manager only after staging a rally and with the help of the Liberty Institute. However, neither in Kutaisi nor in Akhaltsikhe did such meetings bring any success. Unemployed woman from Kutaisi, 31: “The clerks at the distribution company could not provide any information. We went to the main office but the guards did not allow us to enter until an NGO representative took us into the top manager’s 65 office. However, he [the manager] could not explain why the power had been cut.” Workingwoman from Akhaltsikhe, 52: “We go to the distribution company almost every day to get a permit to buy a commercial meter and install it at the local substation. Although the local manager agreed to give us a permit a week ago, now he listens patiently and tells us that he is not authorized to grant us a permit.” The distribution company avoids fulfilling its duties and customers have to buy new cables themselves and cover the expenses of transformer repair. The company often violates its own rules. Unemployed woman from Kutaisi, 24: “When the communal meter was installed the electricity distribution company and two representatives from our street signed a contract that detailing which duties and obligations each party assumed, but it never worked. The company representatives were obliged to replace broken feeders, cut illegal lines, etc. They never fulfilled their obligations, but its no use to complain as they dictated the conditions of the contract, which was unfair from the start. Customers are also unable to fulfill their duties.” Retired woman from Akhaltsikhe, 60. “Until we paid our debts we were not allowed to have individual meters outside, even if we paid for them. We were told that unless there was a 90% payment rate we would have no light. Now even if one of us fails to pay the energy is cut to all 11 families.” Participants feel that the distribution company treats customers harshly. The customers have no idea of their rights. Working woman from Akhaltsikhe: “UEDC has tight policy, if they can they would implement their own decision, furthermore are compromising only in rare cases. People are unaware about what is possible for them to do and what their rights are. Commercial meters were installed without warning and soon after the customers were asked to pay 3 or 4 times more [for their electricity]. Everyone refused and the electricity was cut for 4 months. A transitional period was necessary for the people to realize the situation they were in.” Several focus group participants in Kutaisi stated that they had read information about their rights on the back of their electric bill. Only two participants in Akhaltsikhe group knew about “The Guide for Customers,” and only three in Kutaisi. All of them had some connections with the NGO Liberty Institute, worked at the same building, or had acquaintances at NGOs. It should be noted that even they [the acquaintances] had not read “The Guide.” They also had no knowledge about special radio programs or the customer hotline. In Kutaisi the Liberty Institute showed an educational documentary film about the energy service sector in the summer in spite of the fact that they were suffering from acute energy cuts and were irritated. Retired woman from Kutaisi, 51: “In the summer an NGO visited our city and showed us a documentary film on energy systems. People became enraged – 66 we cannot solve our problems, so what will knowledge of the workings of the energy system bring us?” She continued, “The city municipality has no influence with the distribution company. When we went to the municipality, the former mayor told us the company was a state within the state.” People have also tried to protect their rights in courts, but most such attempts have been unsuccessful. The Akhaltsikhe community chairman filed a suit against the distribution company demanding that they absorb the expenses for replacing old individual meters. Although he won the case, the court decision was not enforced. With the help of the Young Lawyers’ Association NGO, two citizens in Kutaisi also won a case against the company, but again the ruling was not enforced. Focus group participants say that only NGOs have influence on the distribution companies. Kutaisi customers were able to meet with the top manager only when the Liberty Institute interceded. The Akhaltsikhe population rally demanding energy was joined by the Liberty Institute and by representatives of the DEMOCRAT MESKH UNION, who acted as mediators between the public and the distribution company. The energy supply was restored on the condition of a step-by-step payment plan for outstanding debts. Most demonstrators attended town hall meetings organized by the Liberty Institute at their local municipalities. News about these meetings spread informally in both towns, as participants stated that they went to the rally to learn when electric service would improve and at when individual meters would be installed. At both the Kutaisi and Akhaltsikhe meetings, both municipal officials and distribution company representatives attended. Focus group participant said that they were satisfied with the meetings because the company promised them that improved payment would guarantee electricity and that individual meters would soon be installed. However, these promises were not kept: Working woman from Kutaisi, 31: “The energy company representatives told us to take the old meters outside, and to organize payment in order to have power. People fulfilled their part of obligations: while previously only 20% of customers paid, now it is 80%, but there were still no results. People learned of the Liberty Institute slogan: ‘FORGET THE DARK PAST, WORK FOR A BRIGHTER FUTURE.’ People do what they can to save money by economizing on everything, and if they pay their bills, then the second party in such an arrangement must live up to its obligations as well.” Unemployed man from Akhaltsikhe, 29: “It turned out those meetings were superficial. We were promised that individual meters would be installed when arrears were covered but two months have already passed and installation has not begun yet.” Although installation started in Kutaisi, new meters were hooked up only to old posts which were not connected to the power grid – and even this process has now been discontinued. Customers believe that individual meters are uncomfortable for the distribution company. Retired woman from Kutaisi, 51: “Individual meters were installed on some streets, but the way the process was handled was unsystematic. They used old posts that could fall any at moment. Now even that process has stopped and 67 people are under the impression that the distribution company prefers using communal meters. That’s why they ceased installing individual meters.” Focus groups members think that their involvement in the energy sector reform process is poor, as they have no information from the mass media. Moreover, meetings between company representatives and clients are infrequent and unproductive. Woman from Kutaisi, student, 21: “There is nothing interesting about the electricity sector published in the mass media for those how want to learn more about energy problems. The press writes only about Tbilisi’s problems and writes nothing about problems in the regions.” The group in Akhaltsikhe cited the installation of individual meters as being most urgent. Kutaisi group participants feel that privatization may help solve the energy problem and also improve public relations for the distribution company. Unemployed woman from Kutaisi, 31: “The way out is privatization, provided the process is carried out transparently and by the rules. The government is unable to resolve the problem. We think that the financial resources allotted for individual meters have ended up in private pockets.” Woman from Kutaisi, student, 21: “Relations between the public and the company must improve. A neutral area must be created for accomplishing this objective, as people are aggressive toward the company. This mediating ring could be made up of NGOs or journalists.” TBILISI FOCUS GROUP There is little difference in energy problems between Tbilisi districts Avlabari and Varketili, although their relations with Telasi and the Energy State Regulation Company (ESRC) are similar. Varketili is a relatively new district with modern apartment blocks and individual meters were installed as early as 2000. Varketili focus group participants complained mostly about the debt accumulated in the previous period when payment was fixed based on the physical floor area of flats. The practice of fixing energy cost based on accommodation led to the buildup of enormous debt. Public indignation was further exacerbated by the poor power supply in that period – each day customers would receive only 1 or 2 hours of electricity but they were still billed for the time that they had no electricity. The same type of debt has been a problem for Avlabari residents as well. Unlike Varketili, in Avlabari the installation of individual household meters has not started yet. Woman from Avlabari, 57: “Before new individual meters were installed four years ago, I paid my bills regularly according to my old meter reading. Suddenly Telasi declared those readings as inaccurate and instead based the cost on the total area of each flat. Consequently I received a bill for 900 lari.” Avlabari is an old district with many dilapidated and vacant houses. However, the absent owners learned by chance that they had accumulated energy debt. 68 Woman from Avlabari, 57: “We moved home 11 years ago, yet recently we received a bill for 800 lari. I was told at Telasi that all those years I had been hiding from the tax collector, even though I had papers to prove that their statement was ridiculous.” Woman from Avlabari, 43: “We had a flat in a ruined house. Since 1995 we’ve been living at another place, as our old house had neither ceiling nor floors, and not even a roof over our heads. I learned from our neighbors that we had been sent a bill for 1,150 lari. Even if the space of the flat were calculated, this amount is still improbable. We were advised at Telasi to request a commission to investigate how the debt had been calculated.” Focus group participants were unaware of the starting period of debt accumulation, some of them thought about 1996 while others about 1999. In efforts to resolve the arrears problem, members of focus group visited various organizations. Avlabari residents went to the Telais Business Center and to the main office. Varketili residents visited ESRC. With the support of their constituency, desperate Varketili residents filed a lawsuit against Telasi to have their pre-2000 debts annulled. Unemployed woman from Varketili, 39: “We filed the suit, and as it was election time, our deputy was eager to help, but our efforts were of no avail. While power is no longer cut, bills arrive regularly with the old debt included.” Unemployed woman from Avlabari, 56: “I could not accomplish anything at Telasi. In the best case they would share a few words, but they showed no desire to talk with people. There is total lack of discipline at Telasi.” Participants agreed that restructuring of old debt would be a great accomplishment. Unemployed woman from Varketili, 49: “We sent an appeal to ESRC officials for the annulment of pre-2000 debt and the Business Center manager agreed to restructure the debt. However, at both organizations we were told that we still had to pay the debt in all instances. This was at a time when we had been without electricity in Varketili for a week.” While the public doubts the legality of the debt and questions its origins, customers have been unable obtain any relevant information as to its justification. Unemployed woman from Varketili, 39: “I asked at the Telasi Business Center about the method used for calculating the relationship between space and fixed payment for consumed energy. They replied that the calculating mechanism was incorporated within the law. However, they refused to give me the text of the law, even though my debt was enormous.” Unemployed woman from Varketili, 49: “I asked at ESRC to be shown where it was recorded that we’d used so much energy, but instead they only insulted me. I didn’t learn anything.” 69 Focus group participants confirm that it has been impossible to get information due to constant changes in management, and current officials refuse to comment on decisions made by their predecessors. Self-employed man from Varketili, 60: “Telasi has changed ownership so frequently that we cannot ask them to be responsible for the actions of previous managers.” Unemployed woman from Varketili, 49: “Our rights have been violated regularly under TELASI ownership. The only answer one gets at Telasi is, ‘it says so in the computer and nothing can be changed.’” Electricity customers chiefly address the Business Center but the attitude here is perceived is as condescendingly patronizing. Unemployed woman from Avlabari, 55: ““When I entered the room, the manager had his feet propped up on the table and talked with me without any respect. Even if I made a mistake in asking him about something that was not his concern, he should have explained something to me, or given a bit of advice.” Customers are often unable to find competent help at Telasi’s Business Centers and main offices. They say that very few clerks are employed and the customer queues are enormous, and this leads to commotion. Customers cannot decide which counterperson to approach, as all of them are constantly busy. As the operators have no time for individual cases, they usually send customers to the manager, who in turn tries to dismiss customers with standard answers such as, ’The data is not saved in the computer’ or ‘Write an appeal and we’ll establish a commission.’” Unemployed woman from Varketili, 39: “Everyone has rights and responsibilities, whereas Telasi only accentuates the responsibilities of the customer. Their desk-managers are overworked. Wouldn’t it be better to have more people employed to answer customers?” Unemployed woman from Varketili, 39: “You can never figure out which queue to stand in. Even after reaching the counter you never get the answers. The problem is finding the right person to ask.” Unemployed woman from Avlabari, 43: “I could not find out at what level decisions were made. Neither an accountant nor the manager could give me a competent answer. As it turns out, one should address Telasi officials in written form to receive an answer from them. Finally a boy in the guard told me how to write an appeal, but I never found out what documents to prepare, so I lost heart and stopped worrying about the entire affair.” Participants confirm that in order to have your problem solved, you should ask for a commission. However, they say, Telasi is reluctant to satisfy customer requests. There have been cases in which the commission’s ruling was not entered in the computer database and customers continued to receive old bills. As Telasi never bothers to end such violations, it provokes the justified indignation of the public. 70 Self-employed woman from Avlabari, 48: “Collectors from Telasi came to my kiosk to assess the amount of the energy spent and to fix the payment, but in spite of this, the next month I was billed in the old manner. When we asked for an explanation at the business center, we were told they had not received any documents confirming the new payment system, even though we had a copy of that document with us. Now I don’t know what to do. If I don’t pay the bill, my debt will grow and my electricity will be cut.” Most focus group participants learned about ESCAs two years ago. Varketili focus group participants said they had obtained information about ESCA through NGO representatives who visited their homes to talk with them; Avlabari focus group participants learned of their ESCA and its leader through personal acquaintances. Varketili participants praised ESCA’s leader as being an active person who takes the problems of his neighbors to heart. Unemployed woman from Avlabari, 43: “I learned about ESCA from my friends and I became interested. People at the association explained to us in detail what kind of documents we would need. As soon as the documents were collected I left them at ESCA and my problem was solved very quickly.” Unemployed woman from Avlabari, 57: “We went to the house exploitation office to get a document that we had been evicted from the house we lived in. The clerk advised us to go to ESCA, whose headquarters were in the same building, for assistance. Consultation at ESCA was so good that I have no doubt that my problem will soon be resolved.” Unemployed woman from Varketili, 32 “I had no electricity for 5 years because of arrears and I could not pay the enormous debt I was asked to cover. I asked for debt restructure but the only answer I got was, ‘We do not care about your problems.’ When I learned about ESCA, I called their office and was told to leave my data at the end of December. They settled everything and never asked to be paid for the service. Now I have already paid part of my debt.” At present target group participants know ESCA’s location and contact telephone numbers. The number of people asking ESCA for help is constantly growing. Most are retirees without relatives, who are unable to cope with the problems. Retired woman from Varketili, 71: “I live near the Association’s office, so when payment problems arose I went there for advice. People at the Association explained everything to me and helped me with my problem.” Woman from Avlabari, NGO member, 47: “As a neutral organization this association is more trusted than other organizations. Moreover, it has never refused to help people. The Association helps people who are unable on their own to protect their rights in dealing with Telasi.” Target group participants highly regarded ESCA’s activities, calling ESCA a trustworthy organization that voiced customer’s wishes. One participant said that Telasi officials never bothered to explain in clear language to their customers how they should resolve their difficulties. ESCA has knowledge of the energy sector and it is also in a position to state terms and conditions to Telasi. 71 Unemployed woman from Avlabari, 55: “Telasi has had service centers since the beginning, yet their work was ineffective. ESCA is trusted because it does not represent Telasi, which just makes customers pay. I trust ESCA people so much that I left the title to my house with them. I would never have done such a thing with Telasi.” Unemployed woman from Avlabari, 55: “Larisa is a great specialist who showed Telasi clerks their mistakes. These clerks never pay any attention to ordinary people.” Participants see ESCA activities as mutually profitable for Telasi and for its customers, who have started paying their Telasi debts after receiving ESCA’s help. Customers are stating that ESCA clearly explain origin of debts and their responsibilities and only after terms for payment of debts became acceptable. Unemployed woman from Varketili, 38: “ESCA has established relations with the public that Telasi never could have. Telasi listens to ESCA’s advice and criticism.” Unemployed woman from Varketili, 49: “I trust Mzia, the Association leader, who contacted me to explain why I couldn’t avoid paying my debt and promised I would make arrangements.” Unemployed woman from Varketili, 32: “ESCA members Darejan and Larisa spend extra time for particular customers. Sometimes they manage to cut their debts in half, or arrange step-by-step payments. It’s nice when you know you are not being cheated.” Focus group participants admit they prefer to have ESCAs solve problems or carry out research for them. Although ESCA workers gave them the “Guide for Customers” and “Electricity Tariffs,” they say they never read these books. Participants knew nothing of the program for schools to increase awareness of energy sector specifics among pupils. Unemployed woman from Varketili, 39: Hand books were distributed during meetings. Could not read the book, however read some parts. Woman from Avlabari, NGO member, 47: “I have no time for reading. I entrust that to ESCA.” Focus group participants say they know the rules for joining ESCA, and seven participants from Avlabari and four from Varketili have joined the Association. Focus group participants from Varketili said that costomers will be wiling to pay 3 lari membership fee in order to join ESCA, while one lari for membership fee was preferred by Avlabari focus group participants. Also, participants believed that ESCA’s activities should be more visible among general public. Only few are aware of ESCA and population’s knowledge is based on personal discussions about ESCA. They add that if ESCA expands its activity to include other services, then more people will become members, and more people will turn to 72 them for assistance. For some participants, payments for delivered services were more acceptable than payment of monthly membership fee. Unemployed woman from Varketili, 49: It is more realistic if customer pays per delivered service, because public would not pay for not compulsory fees. Until problems appear no one thinks about it. Unemployed woman from Avlabari, 43: “ESCA needs advertising to increase public awareness of its existence. If there were more information in plain language about its activities, more people would be able to benefit from ESCA assistance, and still more would join the Association – and ESCA profits would grow as a result.” Unemployed woman from Varketili, 28. If ESCA’s scope of activities are widened they will became more important for public, thus public will be willing to pay 3 lari as a membership fee and this is due to many unresolved problems existed in Georgia. 73 APPENDIX F. Abbreviations AED: Academy for Educational Development APS: Annual Program Statement CENN: Caucus Environmental NGOs Network ESCA: Energy Services Consumers’ Association FEF: Fair Election Foundation FOI: Freedom of Information GEL: Georgian Lari GNERC: Georgian National Energy Regulatory Commission GoG: Government of Georgia GSE: Georgian Transmission and Dispatch Company GWEM: Georgia Wholesale Electricity Market IRs: Intermediate Results ISFED: International Society for Fair Elections and Democracy M&E: Monitoring and Evaluation MEF: Ministry of Energy and Fuels MIS: Management Information Systems MOU: Memorandum of Understanding NGO: Non-governmental Organization PSA: Public Service Announcements RAO UES: United Energy Systems of Russia RDA: Research and Development Association USAID: United States Agency for International Development UEDC: United Energy Distribution Company VS: Vote Smart 74