MID-TERM PERFORMANCE EVALUATION OF THE USAID/GEORGIA ENHANCING CAPACITY FOR LOW EMISSIONS DEVELOPMENT STRATEGIES (EC-LEDS) CLEAN ENERGY PROGRAM January 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Mendez, England & Associates MID-TERM PERFORMANCE EVALUATION OF THE USAID/GEORGIA ENHANCING CAPACITY FOR LOW EMISSIONS DEVELOPMENT STRATEGIES (EC-LEDS) CLEAN ENERGY PROGRAM FINAL REPORT Prepared under AID-114-1-13-00001/AID-114-T0-15-00005 Submitted to: USAID/Georgia Submitted by: Arvid Kruze, Team Leader Nils Junge, Evaluation Specialist Frank Pool, Green Building Certification and Rating Expert Giorgi Abulashvili, Local Energy Expert Contractor: Mendez England & Associates 4300 Montgomery Avenue, Suite 103 Bethesda, MD 20814 Tel: 301- 652 -4334 www.mendezengland.com DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government CONTENTS Executive Summary ........................................................................................................................................................... 1 Evaluation Purpose and Evaluation Questions............................................................................................. 1 Project Description............................................................................................................................................ 1 Evaluation Tools, Design, Methods and Limitations................................................................................... 1 Findings.................................................................................................................................................................. 2 Recommendations.............................................................................................................................................. 4 Overarching Issues and Recommendations.................................................................................................. 5 1.0 Evaluation Purpose andQuestions.................................................................................................................... 7 1.1 Evaluation purpose.................................................................................................................................. 7 2.0 Project Background ............................................................................................................................................ 8 2.1 Greenhouse Gases in Georgia ............................................................................................................. 8 3.0 Evaluation Methods & Limitations................................................................................................................... 9 3.1 Methodology............................................................................................................................................. 9 3.2 Limitations............................................................................................................................................... 10 4.0 Findings and Conclusions................................................................................................................................ 11 4.1 Evaluation Question 1. What are the major strengths/accomplishments of the EC-LEDS program?.................................................................................................................................................. 11 4.1.1 Findings – Component 1........................................................................................................ 11 4.1.2 Findings – Component 2........................................................................................................ 12 4.1.3 Findings – Component 3........................................................................................................ 14 4.1.4 Conclusions ............................................................................................................................... 16 4.2 Evaluation Question 2. What are the constrants and challenges that inhibit the EC-LEDS’ progress toward achieving the program objectives during the remaining term of the program? What are the outstanding needs?............................................................. 16 4.2.1 Findings – Component 1........................................................................................................ 16 4.2.2 Findings – Component 2........................................................................................................ 18 4.2.3 Findings – Component 3........................................................................................................ 20 4.2.4 Conclusions............................................................................................................................... 22 4.3 Evaluation Question 3. How is the program perceived by the GOG and local municipalities that lead the work for more effective development and implementation of the LEDS strategy, SEAPS and green building certification and rating system? And how is the program perceived by other non-public stakeholders and direct beneficiaries?........ 22 4.3.1 Findings – Component 1........................................................................................................ 22 4.3.2 Findings – Component 2........................................................................................................ 23 4.3.3 Findings – Component 3........................................................................................................ 23 4.3.4 Conclusions............................................................................................................................... 24 5.0 Recommendations................................................................................................................................. 24 5.1 What course correction or further works is needed for meeting major objectives of the program by Winrock and local implementers?....................................................................... 24 5.1.1 Component 1 ........................................................................................................................... 24 5.1.2 Component 2 ........................................................................................................................... 25 5.1.3 Component 3 ........................................................................................................................... 25 5.2 OVERARCHING ISSUES AND RECOMMENDATIONS ........................................................... 27 ANNEXES ANNEX 1: SOW ANNEX 2: Work Plan ANNEX 3: Interviewee organizations and people met ANNEX 4: Documents reviewed ANNEX 5: Telephone Questions to GB Beneficiaries and Stakeholders ANNEX 6: Winrock Statement of Differences ACRONYMS BAU Business As Usual BP British Petroleum BREEAM Green Rating Tool CA Cooperative Agreement DCA Developmental Credit Authority DWG Decision Ware Group E5P Eastern European Energy Efficiency and Environmental Partnership EBRD European Bank for Reconstruction and Development EC-LEDS Enhancing Capacity for Low Emissions Development Strategies EE Energy Efficiency EPB Energy Performance of Buildings EU European Union GB Green Building GBCG Green Building Council of Georgia GEL Georgian Lari (currency) GHG Green House Gases GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit GOG Government of Georgia GWh Gigawatt-hour INDC Intended Nationally Determined Contributions JRC Joint Research Center KFW German Development Bank KII Key Informant Interviews LEDS Low Emission Development Strategy LEED Green Rating Tool MOU Memorandum of Understanding Muni-EIPMP Municipal Inventory, Projection and Mitigation Planning MW Megawatt NAMA Nationally Appropriate Mitigation Actions NGO Non-Government Organization PPP Public-Private Partnership SEAPS Sustainable Energy Action Plans USG United States Government USAID U.S. Agency for International Development UNFCC United Nations Framework Convention on Climate Change 1 EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The main purpose of this mid-term performance evaluation was to provide an objective analysis of the Enhancing Capacity for Low Emissions Development Strategies (EC-LEDS) Clean Energy Program funded by the United States Agency for International Development (USAID) Mission in Georgia. EC-LEDS is being implemented by Winrock International over the period 2013-2017 for a total cost of $6,1 million. The evaluation of EC-LEDS was conducted during October – November 2015 by a team assembled by Mendez England & Associates (ME&A), which included one local and three international experts. Key evaluation questions focused on program accomplishments, constraints, stakeholder perceptions, and possible course corrections. The findings of this evaluation will be used to make informed decisions on the future directions of USAID/Georgia’s assistance in the development of low emission development strategies. PROJECT DESCRIPTION The EC-LEDS Clean Energy Program comprises three components: Component 1 - Georgian Municipal Energy Efficiency, which supports 10 municipalities in quantifying and reducing greenhouse gas (GHG) emissions, and institutionalizing climate change mitigation; Component 2 - Green Building (GB) Rating and Certifying System, which is intended to introduce a voluntary system for rating and certifying GBs in Georgia and build market demand for certified buildings; and Component 3 - National EC-LEDS Working Group and Advisory Assistance, which provides advisory assistance to the Government of Georgia (GOG) to articulate concrete actions, policies, programs and implementation plans under the bilateral EC-LEDS initiative. Components 1 and 2 were planned to be implemented throughout the life of EC-LEDS. In the last two years of the program they were to be continued by a local organization. Component 3 was expected to be completed by the end of the third year. EVALUATION TOOLS, DESIGN, METHODS AND LIMITATIONS The evaluation used a mixed-methods approach, combining qualitative and quantitative research methods and analysis. The information sources comprised document review, meetings with implementing partners, key informant interviews (KIIs), site visits, a stakeholder and beneficiary survey, and analysis of available program monitoring. A total of 34 people representing 21 stakeholder organizations were interviewed. Of these organizations, 12 were beneficiaries of the USAID assistance. In addition, 27 people representing 21 organizations were surveyed in connection with Green Buildings. The results of this survey are summarized in Annex V. Limitations of the evaluations included: (i) focus groups that were originally planned as part of the methodology proved not possible; the solution was to hold a series of in-depth interviews instead; (ii) strategic bias (e.g., overly positive answers) may have been present in answering questions given that the stakeholders were project beneficiaries and interested in continued support; the evaluation sought to minimize this bias to the extent possible by comparing responses from non-beneficiaries; and (iii) projects had not yet been implemented, and funds had not yet been disbursed, which rendered evaluation of sub-projects impossible; however, such a circumstance is normal for mid-term evaluations. 2 FINDINGS Evaluation Question 1: What are the major strengths / accomplishments of the EC-LEDS program? Component 1: Genuine and useful assistance has been provided to seven Georgian municipalities1 that are signatories to the European Union’s Covenant of Mayors, which was launched to endorse and support local governments in implementing sustainable energy policies. Cities and local authorities that want to join or become signatories to the Covenant of Mayors must follow certain steps and take certain actions. These include, among other commitments, creating an inventory to quantify GHG emissions, developing a Sustainable Energy Action Plan, and establishing a Sustainable Energy Office. In particular, useful assistance has been provided in developing or revising Sustainable Energy Action Plans, as well as developing energy efficiency projects and identifying funding sources. As a result, municipalities’ understanding and awareness of energy efficiency issues has increased. Significant efforts have been made to accommodate low municipality capacity for developing Sustainable Energy Action Plans, which has resulted in the development and application of alternative approaches. There is strong program logic in targeting Covenant of Mayors signatories. The approach is a good example of donor coordination/collaboration - in this case between USAID and the European Union (EU). Component 2: The support of the EC-LEDS program has clearly added to the capacity of local organizations specializing in Green Buildings to increase their potential range of membership support, activities, and products. With EC-LEDS’ support, the framework for a local and low cost Green Building rating tool has been developed, as compared to more comprehensive and more expensive alternatives used internationally. A number of strategies and campaigns to increase public awareness of Green Buildings and their benefits have been undertaken, including, among others, development of marketing/action plans for Green Building certification, a “Green Building of the Year” award, a youth-focused promotional television program, and an architectural Green Building “best course-work” contest. These have contributed to increasing public awareness of not only Green Buildings but also of building energy efficiency. A number of reports have been produced to develop a monitoring/reporting/verification plan to measure the energy savings and GHG emission reductions associated with the introduction of Green Building standards. However, the introduction of the preferred energy efficiency rating tool “Display” into Georgia, in accordance with the EU Energy Performance of Buildings Directive, is arguably the biggest accomplishment under Component 2. Component 3: Component 3 accomplishments have included: (i) the development of a “Business as Usual” scenario for the purpose of ultimately developing a low emission development strategy for the country; and (ii) capacity-building to a cross-section of ministries within the GoG on the concept of a low emission 1 This is as measured by the number of Sustainable Energy Action Plans (SEAPs) completed by the project for the municipalities. The number is likely to grow as the project progresses. Also, it is noted that in at least two instances (Telavi and Zugdidi in particular), the administrations split into two municipalities, so, in effect, EC-LEDS has helped at least nine COM-signatory cities: Tbilisi, Batumi, Kutaisi, Zugdidi Municipality, Zugdidi City, Gori, Telavi Municipality, Telavi City, and Akhaltsikhe. EC￾LEDS has also provided significant assistance to Rustavi, which completed a SEAP without input from EC-LEDS. 3 development strategy (LEDS) and an awareness of potential GHG mitigation measures that can be adopted within the country. Evaluation Question 2: What are the constraints and challenges that inhibit the EC-LEDS’ progress toward achieving the program objectives during the remaining term of the program? What are the outstanding needs? Component 1: Although the project implementer has done an exemplary job of supporting municipalities, more could be done to build capacity and reduce dependence of municipal staff on external assistance. Accessing financing beyond municipal budgets has proven to be difficult, mainly because of Ministry of Finance approvals, which are reportedly difficult to obtain. A closer analysis of the partial grants program offered through the project has also found that, although EC-LEDS has provided significant support to municipalities in developing sustainable energy action plans (SEAPs) and staff capacity, and thereby identifying priority investment areas, the impact of the financing on leveraging other investments is not always clear. For example, in most cases, the additional financing comes from the municipal budgets, and far outweighs the grant amounts or any other possible sources of debt financing. It is quite likely that municipal funds would have been allocated even without EC-LEDS grants in order to comply with the COM conditions. Component 2: It is the evaluation team’s position that, because energy efficiency comprises only around 10% to 30% of the much wider Green Building concept, using this concept to achieve building energy efficiency is an intrinsically indirect way to achieve energy savings or GHG reductions in buildings. The Green Buildings concept is not very closely aligned with the overall EC-LEDS GHG reduction objectives. However, USAID funding has now effectively ended for this Component. Component 3: Although the language in the Cooperative Agreement (CA) between USAID and the implementer is not that straightforward, a main feature of EC-LEDS should be to develop an LEDS. This is not being accomplished and an LEDS acceptable to the GOG will not be in place before the implementer finishes the work in Georgia in September 2016. Possible constraints behind this lack of progress in developing an LEDS include: (i) a desire by the GOG to control the LEDS development process with limited capacity; (ii) a lack of initiative by the implementer; (iii) a lack of direction provided in the program description included in the CA; and (iv) the complexity of the model used to help determine the effect of the various GHG mitigation measures. The relative contribution of each of these constraints is not entirely clear. Evaluation Question 3: How is the program perceived by the GOG and local municipalities that lead the work for more effective development and implementation of the LEDS strategy, sustainable Energy Action Plans and Green Building Certification and Rating System? How is the program perceived by other non-public stakeholders and direct beneficiaries? Component 1: The following perceptions have been observed: (i) municipality stakeholders characterized the project implementers as being good communicators, responsive, actively involved, and providing assistance of high technical quality; (ii) assistance received from EC-LEDS in implementing activities under the Covenant of Mayors commitments was beneficial; and (iii) EC-LEDS trainings were described as “very useful” and “well organized, informative.” 4 Component 2: The wider concept of Green Buildings and Green Buildings ratings was generally regarded as a useful and relevant activity by those beneficiaries and stakeholders interviewed. However, there is minimal public willingness to pay for “green” buildings or “green” features in a building. There was, however, a perception of a strong link between energy efficiency and tangible energy savings. Component 3: Perceptions of EC-LEDS within the GOG are generally positive. Support for the computer model used to help determine the effect of various GHG mitigation measures, MARKAL, is perceived as the most important ongoing activity; however, GOG Ministries are expecting an LEDS from the project. RECOMMENDATIONS What course-correction or further work is needed for meeting major objectives of the program by Winrock and local implementers? Component 1: In its remaining years, EC-LEDS should consider providing support to institutional strengthening and sustainability of the process, rather than fostering a situation under which municipalities must continue to rely on ongoing external support. Future project work could be explicitly refocused to follow a new, overarching principle of providing capacity and support (but on a transitional basis) for municipalities to ensure they are able to maintain Covenant of Mayors’ commitments without reliance on donors. In the same vein, simplifying changes in the methodology used to calculate energy savings and GHGs should be considered; e.g., by linking these values directly to growth in Gross Domestic Product (GDP) and perhaps one or two other explanatory variables. Although this might have implications on a national planning level where MARKAL is used to make these estimates, simplifications such as this might produce results that could be judged as being “close enough” for national planning purposes. Regarding the grants program, it might be more effective, if possible, to refocus on a more global municipal street lighting program, rather than relying on project funding to replace a small number of street lights in each municipality (which is currently being done). A technical assistance program would conduct analysis, piloting, and consultation on behalf of all municipalities and would have significant economies of scale by reducing duplication of effort. Although this may be beyond the scope of project guidelines, its feasibility should be assessed. Finally, find a way to help municipalities secure other more significant financing from sources such as the European Bank for Reconstruction and Development (EBRD). To date, most of the financing has been through municipal budget funds. The often-cited major constraint to borrowing is the Ministry of Finance. Also, Georgian legislation is not flexible in allowing municipalities to obtain commercial financing. Component 2: Component 2 has now finished its current operations. USAID has declined to provide any transitional funding to the recommended local organization (GBCG) for this Component for future Green Buildings support work. From the findings, the evaluation team agrees that this decision is sound. However, the $1.38 million that Winrock asked USAID to apply to GBCG for continued Green Building development and support is still available. Given the rapid progress achieved with the “Display” building energy rating tool since April 2015, the close alignment of the use of “Display” with the municipalities’ responsibilities, the potential for “Display” to drive actual energy efficiency (EE) improvements in municipality, government, and private sector buildings in the municipality’s geographical area of responsibility, then some further support for “Display” by the EC-LEDS project in the next three years seems to be highly desirable. 5 Component 3: The most important course-correction for Component 3 is to push for the development of LEDS scenarios as quickly as possible, as these are long overdue in view of the project objectives and remaining time requirements. Based on discussions with Ministry of Energy staff and with Remissia, the computer model used to examine effects of various GHG reduction measures, MARKAL, is clearly not sustainable in Georgia without ongoing outside support. This is evidenced by the continuing need to rely on assistance from MARKAL’s developer, DecisionWare Group (DWG), for tasks that should be relatively simple (e.g., the updating of emissions scenarios to reflect new data) - this, after having had MARKAL in-house for over five years. It is acknowledged that the Ministry of Energy feels that further assistance toward MARKAL support will not be necessary beyond 2016. However, this is not supported by the views of other stakeholders. If it is desired to continue using MARKAL, then provisions should be made to maintain support beyond 2016 (if the Ministry of Energy will have it), as well as to provide sufficient training so that continued outside help is eventually limited to general software updates. Even so, high staff turnover at the Ministry of Energy may be problematic in achieving sustainability. It may, therefore, be desirable to examine the feasibility and acceptance of changing to a simpler energy/GHG modeling software system. Given that it will be less than nine months to the end of the project at the time this report is issued, it is quite possible that support to the GOG may be required beyond September 2016 for developing an acceptable LEDS. The terms of reference for such assistance should be more prescriptive on the work to be accomplished than the description provided in the current CA. Finally, once accepted, an LEDS will need to be implemented. An obvious implementation activity would be the development of laws and regulations in line with the overall LEDS. First, however, an action plan identifying all the required activities and their timing will need to be developed. More areas of required assistance would probably need to be identified within such a plan. OVERARCHING ISSUES AND RECOMMENDATIONS In conducting the evaluation, a number of overarching issues came to light, which affect the project as a whole. These are not technically a part of the evaluation, but they must be highlighted, as they relate heavily to achieving the overall project goals and therefore cannot be ignored. They include: • With reference to the project indicators, it should be mentioned that progress appears to be much better on outputs - e.g., preparing SEAPs, providing training, and raising awareness of GBs among developers - than outcomes, such as GHG reductions and energy saved. As of now, no grant money has yet been disbursed. Also, no LEDS has yet been prepared, even for discussion purposes. The most significant and challenging aspects of the project are yet to be realized, with less than nine months remaining in Winrock’s full-time engagement in the project. • As mentioned earlier, as per the CA, Components 1 and 2 of EC-LEDS were intended to be implemented throughout the five years, to be continued by a local organization over the last two years of the program. For a number of reasons, the original plan to transition to a local firm is at risk and probably will not happen without a change in direction initiated by USAID. A possibility for intervention would be the engagement of a local partner familiar with energy efficiency issues that meets USAID’s eligibility criteria to act as an umbrella organization over institutions that are currently providing technical assistance under Winrock, and who would probably not be eligible for USAID funding for the purpose of transitioning. Support might also be provided under the recently launched Human and Institutional Capacity Development (HICD) 2020 project, which is meant to “achieve tangible improvements in the human and institutional capacity of USAID’s strategic partner organizations in Georgia, including governmental, non-governmental, and for-profit entities”. 6 • Significant interventions by USAID and Winrock appear to be required in order to achieve any course corrections between now and September 2016. To start, USAID should develop a firm schedule of deliverables with Winrock to cover the following: - Finding a qualified local firm for the Years 4-5 transition phase; - Setting milestones for the development of a draft LEDS working paper for discussion with the GOG that will lead to the eventual development of an actual LEDS. According to USAID, there is already a schedule for providing a draft LEDS in the March/April 2016 timeframe; however, this may be affected by the GOG’s current desire to develop another Business As Usual (BAU) scenario; - Working on expanding the DISPLAY labelling program. To accomplish the above in the limited time available, a new approach to implementing the Components on the part of Winrock will be required. To this end, staff changes at Winrock may be necessary. 7 1.0 EVALUATION PURPOSE AND QUESTIONS 1.1 EVALUATION PURPOSE This is a report on the Mid-Term Performance Evaluation of the Enhancing Capacity for Low Emissions Development Strategies (EC-LEDS) Clean Energy Program funded by the United States Agency for International Development (USAID) Mission in Georgia. EC-LEDS is being implemented by Winrock International over the period 2013-2017. The project funding is $6.1million. The evaluation of EC-LEDS was conducted during the period October – November 2015 by a team assembled by Mendez England & Associates (ME&A), located in Bethesda, MD. The team comprised four key experts: Mr. Arvid Kruze (Team Leader), Mr. Nils Junge (Evaluation Specialist), Mr. Frank Pool (Green Building Certification and Rating Expert), and Mr. Giorgi Abulashvili (Local Energy Expert). The findings, conclusions, and recommendations in this report emanate from the collective efforts of the above-mentioned team. The purpose of the evaluation was to provide USAID/Georgia with an objective analysis of EC-LEDS. The primary audience for this evaluation is the USAID Mission in Tbilisi and USAID/Washington. A secondary audience may include local government officials with whom USAID collaborates and other donors. The intended use of this evaluation report is to provide information and lessons learned to enable USAID/Georgia to make strategic decisions on possible course corrections in the project and future programming. EVALUATION QUESTIONS The evaluation questions addressed in this report are: 1. What are the major strengths / accomplishments of the EC-LEDS program? 2. What are the constraints and challenges that inhibit the EC-LEDS’ progress toward achieving the program objectives during the remaining term of the program? What are the outstanding needs? 3. How is the program perceived by the Government of Georgia (GOG) and local municipalities that lead the work for more effective development and implementation of the Low Emission Development Strategy (LEDS) strategy, Sustainable Energy Action Plans (SEAPS), and Green Building (GB) Certification and Rating System? How is the program perceived by other non￾public stakeholders and direct beneficiaries; and 4. What course-correction or further work is needed for meeting major objectives of the program by Winrock and local implementer(s)? A complete Statement of Work (SOW) for the evaluation is provided in Annex I, while the Work Plan is contained in Annex II. Answers to each evaluation question are provided in Section 4, Findings and Conclusions. Since the Evaluation Question 4 requires the team to provide recommendations for future work, the answer to this question is included under Section 5: Recommendations. 8 2.0 PROJECT BACKGROUND 2.1 GREENHOUSE GASES IN GEORGIA As can be seen in Figure 1, recent projections by the EC-LEDS project forecast that GHG emissions will increase by about 100% between 2012 and 2030 (from 10,900 kilotons to 21,800 kilotons) in order to meet the growing energy demands of the expanding industry, transport, and residential sectors. This projected growth of greenhouse gas (GHG) emissions in parallel with Georgia’s economic growth, the lack of an energy conservation culture, and the absence of institutional capacity and policies that promote energy efficiency (EE) and conservation, are all factors contributing to the expected increase in emissions. Energy consumption and the resulting emissions are exacerbated by outdated, inefficient energy systems currently used in Georgia. Vast amounts of electricity are wasted by outdated, inefficient lighting devices and heating systems, and large amounts of heat are released into the atmosphere because of uninsulated buildings - all waste that could be avoided through insulation and energy efficient technologies. Figure 1 – Present and Forecast Sources of GHGs in Georgia SOURCE: “Markal-Georgia LEDS Reference/Business-As-Usual Scenario Report”, Winrock International, September 2014. 2.2 EC-LEDS PROJECT EC-LEDS is a global United States Government (USG) initiative to support developing countries’ efforts to pursue long-term, transformative development, and accelerate sustainable economic growth while slowing the growth of GHG emissions. The initiative does this by building capacity in partner countries and providing targeted technical assistance on LEDS. On December 17, 2012, USAID and the Ministry of Environment Protection of Georgia signed a memorandum of understanding (MOU) that supports LEDS and provides the framework for bilateral cooperation in Georgia. EC-LEDS in Georgia comprises three components: Component 1 - Georgian Municipal Energy Efficiency, which supports 10 municipalities in quantifying and reducing GHG emissions, and institutionalizing climate change mitigation; Component 2 - GB Rating and Certifying System, which is intended to introduce a voluntary system for rating and certifying GBs in Georgia and build market demand for certified buildings; and Component 3 - National EC-LEDS Working Group and Advisory Assistance, which provides advisory assistance to the GOG to articulate concrete actions, policies, programs, and 9 implementation plans under the bilateral EC-LEDS initiative. Components 1 and 2 were to be implemented throughout the five years of EC-LEDS and to be continued by a local organization in the last two years of the program. Component 3 was expected to be completed by the end of the third year. 3.0 EVALUATION METHODS & LIMITATIONS 3.1 EVALUATION METHODOLOGY Information/Data sources The evaluation used a mixed-methods approach, combining qualitative and quantitative research methods and analysis. The information sources comprised document review, meetings with implementing partners, key informant interviews (KIIs), site visits, a survey, and analysis of available program monitoring. Because of key informant time constraints, in-depth interviews were undertaken instead of originally-intended focus groups for Component 2. Research instruments The research instruments comprised a series of questionnaires developed for the main stakeholder types. Semi-structured questionnaires were used for municipal representatives and key informants, and for in-depth interviews of Component 2 beneficiaries and stakeholders. A closed-ended questionnaire was used for the telephone interviews with beneficiaries and stakeholders. The questionnaires were piloted in the field and then revised to reflect an improved understanding of the project and to better capture the relevant information. The questionnaires are contained in Annex 2 of the Work Plan. Document review The evaluation team conducted a desk review of relevant EC-LEDS project documents and key background material and websites concerning GHG emissions in Georgia in order to gain a thorough understanding of: (i) the context of GHG emissions in the country and its commitments going forward; (ii) the contractual terms of reference for the EC-LEDS project, the Work Plans, and activities undertaken; and (iii) project outcomes, outputs, and indicators. A full list of documents reviewed is provided in Annex III. 3.1.1 Qualitative research and analysis Qualitative research consisted of semi-structured interviews with key informants. The evaluation team interviewed over 30 key informants, most of whom were identified during the data collection process. Selection of interviewees was based on perceptions of who would have the best knowledge of the project and/or sector. In most cases, more than one organizational representative was involved in the interview, allowing the evaluation team to capture a more diverse range of opinions and perspectives. Based on the document review, the team identified five target groups of interest that have either participated in the project or are intimately familiar with it. These stakeholder groups are: i) National level institutions – interviews with seven GOG ministry and agency staff and officials, representing three Ministries, with a focus on Component 3 (LEDS). ii) Municipalities - The evaluation team traveled to 8 out of the 10 municipalities that have been receiving project support under Component 1 (GeMunee) and met with municipality officials and staff in Tbilisi, Akhaltsikhe, Batumi, Gori, Kutaisi, Rustavi, Telavi, and Zugdidi. 10 iii) Non-public stakeholders and beneficiaries, including 21 organizations and 27 people who have participated in the project’s GB activities, including construction companies, developers, architectural companies, and educational institutions involved in GB rating and certification institutionalization activities of the program. iv) Project implementers including seven staff from Winrock International (both current and former), GB Council of Georgia (GBCG), and Remissia. v) International donors – European Bank for Reconstruction and Development (EBRD), German Development Bank (KfW), and Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) - working in the energy and energy efficiency sector. vi) USAID Mission staff in Tbilisi. A list of individuals and institutions met and interviewed is provided in Annex IV. 3.1.2 Quantitative research and analysis The quantitative analysis comprised a survey of the 50 non-public stakeholders and beneficiaries participating in the project’s GB component, based on a list provided by the implementer. The method used was telephone interviews, complemented by in-depth interviews. The focus of the questionnaire and interviews was the GB concept in Georgia. The survey was conducted by the local firm, IRMS. Originally 2-3 focus groups were envisioned with this stakeholder group but for various reasons in￾depth interviews were held instead. Further quantitative analysis of project indicators was conducted on secondary data provided by Winrock. All four team members took notes at each interview, and one of the team members reviewed and synthesized interview notes, which were entered into an Excel spreadsheet according to stakeholder type and question topic. The meeting notes were then circulated and checked by other evaluation team members against their own notes for accuracy. A separate sheet in the Excel workbook was maintained for each type of stakeholder. This spreadsheet database was then used as a basis for the analysis. Individual responses were compared against each other and were reviewed to determine themes using expert judgment. 3.2 LIMITATIONS The limitations of the evaluation included: i) Focus groups were originally planned as part of the methodology in order to help the evaluators develop a better understanding of the GB concept in Georgia through open discussions with the most knowledgeable persons. However, this proved not possible due to scheduling constraints, and some lack of interest and awareness on the part of the stakeholders contacted. The solution was to hold a series of in-depth interviews instead. ii) At both the GOG and municipal level, only staff was interviewed and, as a result, strategic bias (e.g. overly positive answers) may have been present in answering questions concerning the implementers, given that the stakeholders were project beneficiaries and interested in continued support. The evaluation sought to minimize this bias to the extent possible by comparing responses from non-beneficiaries. iii) Municipal projects partially funded by the USAID grant scheme had not yet been implemented, and funds had not yet been disbursed, which rendered evaluation of sub-projects impossible. Such a circumstance is normal for mid-term evaluations. 11 iv) Similarly, because no EC LEDS projects had been completed yet, direct observation of municipality projects outcomes was not part of the evaluation. 4.0 FINDINGS AND CONCLUSIONS 4.1 EVALUATION QUESTION 1. WHAT ARE THE MAJOR STRENGTHS / ACCOMPLISHMENTS OF THE EC-LEDS PROGRAM? 4.1.1 Findings – Component 1 In 2008, the European Union (EU) launched the Covenant of Mayors (COM) program to endorse and support local governments in implementing sustainable energy policies. Cities and local authorities that want to join or become signatories to the COM must follow certain steps and take certain actions. Among other commitments, these include creating an inventory to quantify GHG emissions, developing a SEAP, and establishing a Sustainable Energy Office. As of November 2015, 14 Georgian cities had signed on to the COM. The Cooperative Agreement (CA) between USAID and Winrock provides six areas of recommended technical assistance to the municipalities: (i) development and implementation of SEAPs; (ii) establishment of Sustainable Energy Offices or Regional Sustainable Energy Resource Centers; (iii) development of monitoring/reporting/verification plans; (iv) development of sustainable energy public awareness plans; (v) identification and implementation of demonstration projects via “partial grants”; and (vi) Development Credit Authority (DCA) guarantees and financial institution assistance. The last item, the DCA facility, was dropped early on in the project. Based on visits to the municipal offices of eight municipalities and interviews with the implementers, the evidence suggests that EC-LEDS is making a significant contribution. The program has provided direct assistance to seven municipalities in preparing and/or revising their SEAPs, while three to four others have received assistance or training, all of which is designed to help them meet their obligations under the COM program. EC-LEDS provides technical assistance in the form of awareness raising, training, advisory services, identifying and facilitating the financing of potential projects, monitoring and verification, and development and revisions of the municipality SEAPs. EC LEDS, through the consultant firm Remissia, has helped municipality staff prepare or revise their SEAPs, a key obligation for COM signatories. Support was provided to collect input data (through funding short-term consultants) and Remissia experts developed the SEAP itself. Since Georgia (and other COM eastern countries) faced challenges when using the standard EU baseline approach in arriving at emission reduction requirements, an alternative “business as usual” (BAU) approach was proposed. This would have avoided unfairly penalizing municipalities for their “unnaturally” low consumption levels in 1990, the recommended base year. However, the BAU approach proved too complex for municipalities to calculate on their own, because it required data that often was not possible to obtain, obliging cities to rely on external support. Given this, the Joint Research Centre (JRC)2 developed an optional approach; however, this too came with some drawbacks, as noted in the Year 2 Progress Report. The project then developed a simpler tool called the Municipal Inventory, Projection and Mitigation Planning (Muni-EIPMP) tool, which may be described as a modified (and simplified) MARKAL model designed specifically for municipal LEDS purposes (as opposed to national level LEDS planning). The partial grant program is designed to support implementation of demonstration projects in 2 JRC is the scientific and technical arm of the European Commission which provides technical support to COM signatories. Source: http://www.covenantofmayors.eu/+-JRC-+.html 12 municipalities, with up to $50,000 of grant funding available, which must cover a maximum of 20% of total project implementation costs. Projects have been selected in four municipalities, with $213,910 in total earmarked for five projects. The partial grants program has, based on Winrock’s calculations, leveraged $3,358,112 of funding - primarily from municipal budgets and, in two cases, from British Petroleum (BP).3 EC-LEDS has also assisted municipalities in identifying or applying for financing, by advising them on opportunities and application procedures, a form of assistance that the EBRD, as a partner donor, highlighted as a particular strength of the EC-LEDS project. In the case of Rustavi municipality, this led to the municipality winning a bid for a $910,000 project to implement EE projects in several kindergartens. Municipality staff reported that their awareness and understanding of EE issues was significantly strengthened through the EC-LEDS program, increasing their capacity to address EE issues in the target sectors, including: buildings, transport, street lighting, green zones, and waste management. One municipality summarized the key project benefits as “the introduction of modern technology, awareness of concepts, and a strategy for achieving our goals.” Other examples of project support include: • Tbilisi bus fleet replacement. EC-LEDS worked with the municipality to develop plans to replace its fleet of 1,000 buses with fuel efficient buses that meet Euro VI emissions standards. EBRD is planning to finance the project, which is expected to take 2-3 years. However, 100% attribution to EC-LEDS for this activity might be somewhat generous, as the activity had started under a previous USAID project implemented by Winrock. • Street lighting. All eight municipalities visited by the evaluation team had plans to change their street lights to energy efficient (usually LED) bulbs and associated fittings, using either partial grants funds or own municipal budget. In some cases they had already begun. This is a popular measure in large part because the accompanying investment returns (which translate directly into budget savings) can be: 1) quickly realized; 2) substantial; and 3) easily verified. For example, Zugdidi municipality estimated it could cut its street lighting expenditures by GEL 570,000 ($230,000) per year, and Batumi municipality estimated savings of GEL 2 million ($833,333) per year. • Potential brownfield developments in Tbilisi and Zugdidi. The project has helped facilitate a public private partnership (PPP) initiative involving the donation of "brownfield" sites4 by municipalities for private development in Zugdidi and Tbilisi. A study on the legal structure for these initiatives has been undertaken. In Tbilisi, a 40 hectare site would be taken over by a developer with plans to construct GB luxury homes, with potential investments of $15 million to $35 million. The initial proposal was that the city would donate the land, while the developer would take responsibility for its rehabilitation. The project is currently on hold. In Zugdidi, the brownfield development is still in the planning stage. 4.1.2 Findings – Component 2 USAID’s CA with Winrock recommended the following program interventions: (i) development of a voluntary system for rating and certifying GBs; (ii) development of a promotional strategy and campaign; and (iii) development of a monitoring/reporting/verification plan to measure the energy savings and 3 USAID. EC LEDS Annual Report (October 2014 – September 2015) 4 Previously developed industrial/commercial areas which, after cleaning and decontamination, can be used for new developments. 13 GHG emission reductions associated with the introduction of GB standards. In its first 18 months of implementation, Component 2 was focused on promoting wider GB issues along with capacity development support for GBCG. With this EC-LEDS support, GBCG has now completed the framework for a local and lower cost alternative to the more comprehensive and more expensive main international GB rating tools, in particular LEED5 and BREEAM6, which are the two most widely recognized energy/environmental assessment methodologies used globally in the construction industry today. EC-LEDS’ support has clearly helped GBCG develop its capacities and activities to a point where a competition has been held and where the winner was to receive a free LEED or BREEAM GB rating (although this is not going ahead now with the end of USAID funding for this component). This would have been the first LEED or BREEAM rating undertaken in Georgia. Such a rating would have been a useful first pilot demonstration of an internationally recognized GB rating in Georgia. The GBCG has been in existence since 2010 as what appears to have initially been an informal interest￾group oriented organization aiming at supporting and promoting the wider concept of GBs in Georgia. The support of the EC-LEDS project has clearly added to GBCG’s capacity, potential range of membership support, activities and products, and its exposure to potential members for GBCG membership. Accordingly, with the assistance of EC-LEDS over the past two years, GBCG has a core group of around 10 members and is now ready to become a formal membership-based national GB advocacy and support organization. Amongst the 30 people interviewed as part of this mid-term evaluation, around 80% thought that if the annual GBCG membership fee was $50-$100 for individuals and $200-$1,000 for corporates, their organization would join GBCG7. Thus, with the support of EC￾LEDS, GBCG now has a potential membership base of probably around 25-35 individuals and companies. The potential membership base of GBCG would appear to be sufficient to pay for a part-time GBCG Chief Executive Officer (considering that the Turkish GBC has around 3 full time staff and 140 paid members for a country with 16 times the population of Georgia and a higher per capita income as well; therefore, a GBCG seems likely to be a very minimally staffed and low membership base organization for the foreseeable future). From March 2015, a new focus on building EE is apparent in EC-LEDS project documentation, including critically investigating the link of EE rating tools to the EU Energy Performance of Buildings (EPB) Directive. In April 2015, a report was finalized on using “Display” or “Energy Passport” as the preferred building EE assessment tools. This was when it first became apparent in EC-LEDS literature that there was a need to have an EE tool that would be used widely and that would also be compatible with the EU EPB Directive that Georgia will need to start addressing as part of its EU Association Agreement. Display was chosen as the EE rating tool for existing buildings. By July 22, 2015, the first people had been trained on the use of Display. Eighteen municipality and 14 other (mostly EU Monitoring Mission) buildings were then rated with Display and certificates to this effect have now been signed by USAID and GBCG, ready for display on the relevant buildings. The Display rating is visually similar to the EU appliance energy rating label, so the Display building energy rating brand awareness and credibility can piggyback on that of the EU appliance energy label, which should already be familiar to many people in Georgia, and which will be widely used as the common label for energy performance in the country. Thus, Display should have no problem being seen as a 5 Building Research Establishment Environmental Assessment Method 6 Leadership in Energy and Environmental Design 7 The membership fee levels are hypothetical, created for the survey as a way of gauging potential interest. They are partly based on the consultant’s knowledge of comparable membership fees for the Turkish Green Building Council. 14 credible energy performance label for buildings in Georgia. The Display building energy rating label is based on actual building energy performance, so its use would be a strong and direct motivator for those buildings with Display labels to improve the EE of their buildings and, hence, to subsequently improve the rating level displayed on the label. The use of Display by municipalities for their own buildings, as well as for use in wider government and privately owned buildings in the municipality, is a tangible way for COM signatory municipalities to demonstrate progress towards their COM commitments. As a widely used EU municipality-focused building EE tool, Display is fully compatible with Georgia Municipality EU COM and EPB requirements and commitments. Winrock has undertaken a number of strategies and campaigns to increase public awareness of GBs and their benefits, including, among others, development of marketing/action plans for GB certification, the previously-mentioned GB award, a youth-focused promotional television program, and an architectural GB “best course-work” contest. These have contributed to increasing public awareness of not only GBs, but also building EE. However, it is difficult to measure tangible results stemming from these efforts. In response to the Component 2 requirement to develop a monitoring/reporting/verification plan to measure the energy savings and GHG emission reductions associated with the introduction of GB standards, a number of reports have been produced. However, while these reports undoubtedly have technical merit, the outputs should be tempered with the findings on the constraints and challenges inhibiting progress on EC-LEDS in Georgia, as discussed under the next Evaluation Question. 4.1.3 Findings – Component 3 On December 17, 2012, USAID and the Ministry of Environment Protection of Georgia signed a MOU that supports LEDS and provides the framework for bilateral cooperation in Georgia. Areas mentioned for cooperation include activities that: increase and encourage the use of clean and energy efficient resources; support the development of a national GHG inventory system; improve the policy environment in low emission economic growth; expand economy-wide and technical modeling efforts; and improve governance of Georgia's natural resources. In the CA, the opening line to the description of Component 3 states that this initiative provides “a strategic framework for the GOG to articulate concrete actions, policies, and programs that slow the growth of emissions, while advancing economic growth and meeting Georgia’s development objectives”. Thus, it can be assumed that the articulation of such concrete actions, policies, and programs is a major goal of this Component. Initially, at the project appraisal stage, Component 3 was called National-Level LEDS Planning. Subsequently, in the CA, it became National EC-LEDS Working Group and Advisory Assistance (which is not a subtle difference). The CA envisaged the formation of a working group to achieve the goals and actions agreed by USAID and the GOG in the MOU, as well as assistance in carrying out the activities mentioned in the MOU. Winrock began working on the project in the fall of 2013 and issued a final Year 1 Work Plan in March 2014. The Work Plan broke down Component 3 into four sub-activities, which are apparently taken from excerpts in the Component 3 description contained in the CA: (i) ensure SEAP activities are consistent with national policies and priorities; (ii) ensure that municipal-level data, findings and results inform national policies, programs and actions; (iii) analytical capacity-building; and (iv) provision of advisory assistance to GOG. Working within this framework of sub-activities is a project Steering Committee, Working Groups and Sub Working Groups, with representation from all relevant Ministries and Winrock. In September 2014, in support of the above activities, Winrock published a report “MARKAL-Georgia LEDS Reference/Business-as-Usual Scenario Report.” In April 2015, it published another report, “MARKAL-Georgia EC-LEDS Reference Scenario Report.” Each of these reports provides a BAU 15 scenario, which essentially forms the reference scenario from which GHG emission reductions by mitigation scenario may be derived, along with associated costs. By running the MARKAL model with combinations of mitigation measures affecting the BAU scenario, the effects of these measures may be derived and analyzed. Aside from these two reports on the BAU scenario, Winrock has, as described in its latest annual report, undertaken a number of trainings, arranged and coordinated a number of high-level meetings amongst stakeholders to discuss LEDS, developed and refined both the MARKAL model and the BAU scenario, and provided “valuable advisory assistance” to the GOG. Given all of the above, it appears that the major accomplishment of Component 3 to-date is the development(s) of a BAU scenario, through reports published in September 2014 and April 2015. It should also be mentioned that, at the request of the GOG, the BAU scenario is currently being re￾visited once again - to be updated with more current and presumably better data. The capacity-building and advisory assistance is, no doubt, of value, but without any tangible results in terms of, in particular, actually developing an LEDS; it is difficult to ascribe much value to it at this point in time. However, it is worth noting that several stakeholders mentioned that very little was known about LEDS within the GOG at the beginning of the project and that the project’s capacity-building efforts at least led to much better awareness of this very important topic. Regarding the BAU scenario, the usefulness of this scenario by itself is questionable without carrying through the analysis to its ultimate objective - i.e., the development of an LEDS. This is not to trivialize the BAU scenario, as a great deal of effort appears to have been expended on this important step in the development of the LEDS. However, the process needs to be completed. It is actually difficult to comprehend how a draft LEDS, even in rough form or in the form of several possible scenarios, has not yet been developed. After all, a reference scenario, however approximate and seemingly in a continuous state of being refined, has been in place since September 2014. As gathered from interviews and Winrock’s Year 2 progress report, the Ministry of Energy, which has taken ownership of MARKAL, has actually been running the model to examine the effects of various GHG mitigation measures. This begs the question of why next steps have not been taken on a more aggregate basis to develop an LEDS. This would first involve examining the effects of all possible mitigation measures (within reason) through MARKAL. Next, a list of the measures and their effects would be compiled. Finally, various LEDSs can be developed by selecting combinations of different mitigation measures. Till now, the working groups have been learning about and discussing LEDS on a very conceptual level, with little in the way of results, which is puzzling because an LEDS is only a combination of mitigation measures, related costs and effects, with an eye on the bottom line of GHG emissions and associated costs - but which must be agreed to by all GOG stakeholders. In light of the above, a better approach to developing the LEDS would have been to continue the analysis, as described above, after the first BAU scenario was developed in September 2014. In this way, perhaps several draft LEDSs could have been developed. These draft strategies would then have formed the basis for more substantive discussion by the GOG. The strategies could then have been modified appropriately, if necessary, with the BAU scenario as it was refined. It is actually quite likely that the basic strategies would not have changed very much as the BAU changed and was updated. Presently, the GOG is expecting an LEDS report to emerge from the project. Winrock plans to have this report ready in September 2016, which is at the end of its full-time involvement in the project. According to USAID, a first draft for discussion with the GOG is to be delivered sometime over April￾May 2016. 16 4.1.4 Conclusions For Component 1, it can be concluded that: (i) Winrock has clearly provided genuine and useful assistance to municipalities in developing or revising their SEAPs, in developing EE projects and identifying funding sources. As a result, municipalities’ understanding and awareness of energy efficiency issues has increased; (ii) Significant efforts have been made between project implementers and the JRC to accommodate low municipality capacity for developing SEAPs, which has resulted in the application of alternative approaches, such as the EIPMP; and (iii) There is a strong program logic in targeting COM municipalities and the approach is a good example of donor coordination/collaboration - in this case between USAID and the EU. The introduction of the Display building energy rating tool into Georgia is arguably the biggest accomplishment under Component 2. Other accomplishments in line with the recommended areas laid out in the CA include: (i) the development and implementation of an extensive promotional strategy and campaign for GBs; and (ii) the development of a monitoring/reporting/verification plan to measure the energy savings and GHG emission reductions associated with the introduction of GB standards. However, these outputs should be tempered with the findings on the constraints and challenges inhibiting progress on EC-LEDS in Georgia, as discussed under the next Evaluation Question. Component 3 accomplishments have included: (i) the development of a BAU scenario for the LEDS and (ii) GOG capacity-building on LEDS. However, these accomplishments are, for the time being, heavily qualified. Their ultimate indicator of success will be the development and acceptance by the GOG of an actual LEDS. 4.2 EVALUATION QUESTION 2. WHAT ARE THE CONSTRAINTS AND CHALLENGES THAT INHIBIT THE EC-LEDS’ PROGRESS TOWARD ACHIEVING THE PROGRAM OBJECTIVES DURING THE REMAINING TERM OF THE PROGRAM? WHAT ARE THE OUTSTANDING NEEDS? 4.2.1 Findings – Component 1 Obtaining complete and reliable data for SEAP development has proven to be a particular challenge for the municipalities. This is due to a number of factors including, among others, minimal data being collected by municipalities to monitor economic activities, private sector unwillingness to share its data, and data not being in the required format. The reasons are elaborated more fully in the project document “Municipal Emission Inventory, Projection and Mitigation Planning Tool.” The implications of the data inadequacy are that it has proven difficult for municipalities to develop a baseline inventory, and their consequent ongoing reliance on external help. Also, municipal capacity for updating SEAPs is limited. SEAPs can be developed using either a baseline year (e.g., 1990) or a BAU analytical approach. BAU is generally the preferred choice (also at the national level) because it makes GHG reduction commitments easier to meet over the long-term, given Georgia’s very low energy consumption levels relative to similar countries and its potential for emissions growth. However, Winrock’s Annual Report notes that the “BAU approach, is complex and beyond the capabilities of most municipalities.” As noted above, this means that municipalities must rely on external support, at present provided by Remissia through the project, to conduct the necessary calculations, and to develop, monitor and revise their SEAPs. This raises the issue of sustainability - municipalities will likely remain dependent on outside support as long as they use the relatively more sophisticated BAU approach as utilized under EC-LEDS. The project’s focus on providing technical assistance in developing and updating SEAPs is clearly welcomed by municipalities, which lack the expertise and resources to undertake this type of analytical work, and to even collect the data, themselves. While a solution was developed (as described above), the new approach seems likely to result in indefinite municipality dependency on external support, despite the argument that it is “simple enough to be successfully used at the municipal level, but 17 comprehensive enough to provide [municipal] planners…the insights they require.”8 Based on discussions with municipal staff, it was far from clear that they were capable of using the tool on their own. While the project conducted a total of four trainings on Energy Inventory, Projection and Mitigation Planning (EIPMP) during Year 2, it is unclear how much capacity building is planned, and whether it would be successful, i.e. enable municipalities to wean themselves from external support. Turnover of staff who have been trained by the project will always be a risk. For example, Batumi’s energy efficiency specialists who worked on the SEAP have all subsequently left their positions. Identifying and obtaining financing is a constraint. The initial plan of obtaining funding and implementing projects through the USAID DCA program, which provides a partial credit guarantee, was not feasible. With the DCA program not being feasible, alternative financing mechanisms came to light, such as the Eastern Europe Energy Efficiency and Environmental Partnership (E5P), an EU financing program that focuses on municipal energy efficiency projects, specifically rehabilitation of water and wastewater systems, solid waste management, and insulation of public buildings. Georgia signed an E5P contribution agreement in March 2015. The Municipalities and Winrock have faced procurement challenges. Disbursement procedures have proven to be complex and administratively burdensome for municipalities, slowing down the process of grants disbursement. This is largely related to local procurement legislation, specifically, the requirement for municipalities to use a "price-based" rule (the lowest price bid must be accepted, regardless of quality). In order to work around this issue, an arrangement was agreed upon in several cases under which Winrock would conduct procurement on behalf of municipalities through USAID. In these instances, Winrock held individual meetings with municipalities, introducing them to the grants manual and describing the grants selection, disbursement, and management process in detail. No potential problems were identified during this process. Still, procurement difficulties arose. USAID noted that delays could have been avoided had Winrock conducted due diligence of USAID and municipality procurement rules in advance. EC-LEDS financing activities under the grants program thus far are summarized in Table 1. It can be seen that Winrock claims credit for about $3.6 million in terms of total project cost. However, without the Rustavi project, the total project amount would be $2.7 million rather than $3.6 million (it is unclear how the Rustavi kindergartens project of $910,000 is considered ‘leveraged’, considering that no grant money is being allocated in this case). It should be noted that, while EC-LEDS assistance was clearly critical for Rustavi in winning the grant, the effect of leveraging funding through the partial grants program was nonexistent in this case. It may also be noted that, while the projects and likely financing sources shown on Table 1 are positive developments, no grants had yet been disbursed as of November 2015, which means that the amounts shown are all provisional. Table 1: EC-LEDS Financing Activities (in USD) Municipality Project Recipient Grant amount Other amount Total project cost Other source(s) of funding Batumi LED in parks Municipality 18,250 994,788 1,013,038 Municipality Batumi LED in streets Municipality 52,950 826,947 879,897 Municipality Kutaisi EE at Torpedo base EEC 42,720 171,278 213,998 BP, Municipality Zugdidi LED in Streets Municipality 49,990 255,280 305,270 Municipal 8 Winrock. Year 2 Progress Report 18 Municipality Project Recipient Grant amount Other amount Total project cost Other source(s) of funding budget Tbilisi EE at Elders house EEC 50,000 199,819 249,819 BP, Municipality, State Anti￾Trafficking Agency Rustavi SUDeP kindergartens SDAP - 910,000 910,000 EU, Municipality Total 213,910 3,358,112 3,572,022 Source: Winrock International It should also be noted that that the USAID grant of $50,000 maximum per project is, on average, only 8% of the financing, which is quite small and unlikely sufficient to make a difference in a "go - no go" decision. In several cases, the remaining amount comes entirely from the municipal budget, which means that resources have been reallocated from other municipal needs, rather than outside financing. The CA notes that the grants “are to be used strategically, either to test new technology/project types that have no precedent in the country or to leverage commercial financing so that the proposed investments can reach greater scale and serve more intended beneficiaries. When designed with commercial investment in mind, grant funding can cover critical gaps between what a lender is willing to fund and the total project costs.” However, given the legislative environment in Georgia, commercial borrowing at the municipal level is an unlikely source of funding. The most viable forms of additional financing, aside from municipal budgets, are therefore other donors or foundations. The above problem is most apparent in the approach used for financing street lighting programs under the partial grant program. Replacing street lights is a generally relatively straightforward project. Street lighting is therefore a good investment candidate. However, a street light replacement program is not inexpensive. Each municipality has thousands of street lights and the cost for replacing each street light fitting can be several hundred dollars (depending on type and quality) and could take a decade to complete. Because of onerous requirements and apparent reluctance on the part of the Ministry of Finance to allow municipalities to borrow, there is a potential missed opportunity for both significant energy efficiency gains and significant budget savings, as the municipalities are generally funding the balance themselves, and not borrowing. The partial grants program is supporting three street lighting programs (two in Batumi and one in Zugdidi), but the number of street light fittings to be replaced are a tiny fraction of the total (less than 5% in Zugdidi and 1% in Batumi). The number of street light fittings replaced in other municipalities is equally small. These are considered to be pilot projects, which will be studied before the replacement programs are scaled up. Regarding the Sustainable Energy Offices, municipalities face several challenges relating to staffing and funding these offices. Thus far, in the municipalities visited, none have been established. Expected revenue streams from these offices (from energy audits, for example) are not yet sufficient, there are financial constraints in the municipalities' budgets, and regulations limit the level of staffing allowed in municipality budgets, based on a ratio to population. 4.2.2 Findings – Component 2 In Component 2, it is the evaluation team’s position that the concepts of “Green Building” and “Building Energy Efficiency” were conflated right from the beginning in the project design and in the initial operation and reporting regarding the promotion and uptake of wider GB concepts, and the launch and support of a local GB rating scheme in Georgia. Actually, EE comprises only around 10% to 30% of the much wider GB concept (depending on the weighting in any particular GB rating scheme). So using GB 19 concepts to achieve building EE is an intrinsically indirect way to achieve GHG reductions in buildings. In addition, the operative assumption in the project design and in its first two years of operation was that there is a latent demand for GBs and their GB ratings in Georgia. However, in GBCG’s five years of existence to date, including two years of strong support under EC-LEDS, not one building in Georgia has yet been rated under the two main international GB rating schemes of LEED and BREEAM. The CEO of GBCG is a qualified LEED and BREEAM GB scheme assessor, as is the founder of the Turkish GB Council who also worked closely in the EC-LEDS support of GBCG and the wider development of GB schemes in Georgia under the project. Thus, it very much appears that there is no actual market interest by building developers or owners in paying for international GB ratings for their buildings in Georgia. This is evidenced by the fact that no one has been prepared to actually pay for even one international GB rating in the 5 years since GBCG was formed. As previously mentioned, Component 2 was primarily focused in its first 18 months on delivering GHG reductions through building energy efficiency gains that were expected to arise from the development and anticipated widespread uptake and use of a local simplified GB rating scheme developed for Georgia. The EC-LEDS project therefore supported the development of a simplified GB rating scheme for Georgian buildings and conditions. The rationale given was that such a localized and simplified GB rating scheme would be less costly to apply, and hence would be likely to be applied in large numbers of buildings. And local GB rating scheme users would have to pay licensing fees to GBCG and to use only GBCG endorsed assessors, thereby creating a wider and higher fee GBCG membership base. However, such a simplified scheme necessarily uses more judgement rather than quantitative assessment criteria, and would also have to develop its own market credibility, which cannot be achieved cheaply or quickly. It is very instructive that a national GB rating scheme in Turkey was tried and did not get market traction, leading to its abandonment. GB ratings in Turkey are now primarily performed to international GB rating schemes’ criteria, in particular LEED and BREEAM. But if GB ratings are performed to international rating scheme criteria, then there is no need for the rating assessors or the rating recipients to join GBCG, further eroding its paid membership base (as is the case in Turkey, where the Turkey GB Council membership base has reduced from 170 to 140 paid members). Thus, there appears to be no market demand for the use of established international GB ratings in Georgia, and there is no evidence why a local GB rating scheme in Georgia should succeed when a localized national GB scheme was tried in Turkey and failed. Component 2 was implemented by Winrock on the basis of an accelerated two years’ support program, with Winrock expecting to lead an early transition to a local organization (namely GBCG), which was to receive direct support from USAID (this was originally envisaged to happen at the end of Year 3 in the project design). The request was for USAID to fund GBCG to the tune of $1.38 million over two years, which would fund nine full time equivalent staff and three full time equivalent contractors. Winrock expected to provide staff on secondment to GBCG in the two year accelerated development phase, although this would call into question the whole objective of transitioning to a local organization. In view of the above, the following should be noted: (i) the proposed GBCG budget was 22 times larger than GBCG had ever previously handled; (ii) GBCG was assessed as lacking the necessary capacity, and does not seem to even be eligible to apply to receive such funds under USAID direct funding criteria; (iii) GBCG lacks any paid membership base; and (iv) there are no plans elaborated in the project literature how GBCG is supposed to be sustainable after the proposed two-year major funding support by USAID. Ideally, GBCG would have around three full time equivalent staff to undertake its activities in its proposed post-major intervention funding phase. However, the funding membership base for a post support GBCG would be 25-35 members, which would at the very best fund one full time equivalent staff position. Thus, GBCG would probably have a significant funding shortfall and be dependent, for the foreseeable future, on ongoing donor funding, or it would have to shrink back to being a small membership funded advocacy group for GBs in Georgia, comprising perhaps of dedicated academics and 20 some GB supporters in the construction industry. In any event, funding support by USAID to GBCG at the end of Year 2 did not take place, so Component 2 is no longer being financially supported by USAID. 4.2.3 Findings – Component 3 A disconcerting feature of Component 3 is that it has not moved very much beyond the development of a BAU scenario, which is only a step (albeit an important one) in developing an LEDS. In April 2015, EC￾LEDS prepared a presentation for the GOG to provide examples of GHG mitigation policies that are best practice worldwide in each sector that can be adapted and adopted for Georgia. It also presented measures from a variety of organizations ranging from the intergovernmental panel on climate change working group to nationally and locally prepared climate action plans and presented best practice mitigation policies. However, the presentation did not contain specific mitigation options for Georgia, as it was more of a notional outline of possible directions for mitigating GHG emissions. Although these initiatives were probably useful, a working document with estimates of likely GHG reductions by mitigation measure adding up to Georgia’s total reduction commitments (which would form the basis of an LEDS) has not been produced. It should be mentioned that the CA never actually says that an LEDS is expected to be developed. On the other hand, it does say that the “strategic framework” laid out by the “bilateral” EC-LEDS initiative should “allow the GOG to articulate concrete actions, policies, and programs that slow the growth of emissions…” This has not happened either, except for: (i) a construction code law incorporating language on energy efficiency and GBs; and (ii) Georgia’s Intended Nationally Determined Contributions (INDC) filed with the United Nations Framework Convention on Climate Change (UNFCC). Oddly, while Winrock has considered the first item in its project indicators, it has not considered the INDC. In any case, the GOG is expecting a report on LEDS, the adoption of which (or something similar) would provide a very robust indicator of the success of the Component. Regarding Georgia’s INDC, Georgia’s submission deals extensively with commitments arising from the Forestry sector, which, through better forestry practices, will effectively act as a carbon sink on emissions. However, the vast majority of emissions reductions are, in fact, going to arise from the Energy, Industrial Processes, Agriculture, and Waste sectors. Mitigation measures applied to the Forestry sector are described in detail in the INDC because an LEDS for the Forestry sector has been developed, while an LEDS for the other sectors has not. This is somewhat disappointing, given that three of these sectors have been the focus of the LEDS effort and could have been better presented in Georgia’s immediate emission reduction plans had an LEDS been developed for each of them. Possible constraints on the lack of moving forward with the development of a LEDS are: • The GOG controls the LEDS development process which, by design, is host-country-led throughout the world and it is very important to the USG that it remains this way. The inherent problem with this arrangement is that less is likely to get done, compared to a USAID consultant led effort, simply because GOG staff have other duties and cannot be 100% devoted to LEDS. As it is, the project Steering Committee and Expert Working Group are chaired by the Ministry of Environment Protection. As a result, Winrock is relegated to more of a participant role, as opposed to being a leader. • A lack of initiative by Winrock to package various mitigation measures into LEDS alternatives that could be presented to the working groups and Steering Committee. Winrock has already produced two BAU scenario reports; thus, there appear to be no or little constraints on running MARKAL for undertaking the necessary analyses. While it is recognized that the mitigation measures ultimately adopted by the GOG will require extensive consultation, there has been nothing stopping Winrock from developing initial LEDS scenarios – just to form bases for preliminary discussions. LEDS scenarios could then be modified accordingly and presented again to the GOG in an ongoing, 21 iterative process. A number of such iterations should eventually lead to agreement by all parties within the GOG. One would think that the parties at this stage of the project would have had at least something on paper to discuss. As previously described, beyond the BAU scenarios, EC-LEDS has provided only a presentation to the GOG of examples of GHG mitigation policies, as well as potential best practice measures that could be applied from a variety of sources. However, these were not proposed as specific mitigation options for Georgia. • Th EC-LEDS agreement and the CA seems to have vague directions as to how the work should be carried out. This vagueness seems to have been an evolutionary process, as “National-Level LEDS Planning” in the project appraisal document became “National EC-LEDS Working Group and Advisory Assistance” in the CA, which seems to relieve Winrock somewhat of the responsibility for helping to develop a plan. On the other hand, there are the “concrete actions, policies and programs” that Winrock is obliged to support, which one would think implicitly include the development of a LEDS. While it is recognized that EC-LEDS is a new initiative for which no identical template exists that could have been emulated in developing the CA, there must be scores of similar projects that have dealt with policy issues and strategic directions to be employed by governments and that have faced similar problems of limited counterpart staff capacity coupled with government leadership of the process. If such a template was actually used in developing the terms of reference, then it could very well be that many such higher-level technical assistance projects unavoidably suffer from the same constraint. In any case, the lack of clarity in the Component 3 description on tasks, roles and deliverables has probably contributed to Component 3 stalling at the development of BAU scenarios and not leading to an actual LEDS. • The MARKAL model. This model was introduced to Georgia during the late 2000’s, but apparently fell into disuse by the GOG before being resurrected around 2012, with MARKAL training provided by USAID under another project. It is a rather complex model, as Ministry of Energy users admit to requiring ongoing support from MARKAL’s developer, DecisionWare Group (DWG). This finding is supported by Remissia, which acknowledges the model’s complexity as well as the model’s need for data that are not necessarily available for Georgia, and confirms that simpler alternatives to MARKAL exist. Other than Remissia and Ministry of Energy users, a former employee of the project mentioned the complexities of MARKAL as posing a problem to the GOG, even mentioning that a GOG official - who was actually interviewed by the evaluation team and did not cite MARKAL as being a problem when given the opportunity to do so - had at one time complained openly about MARKAL’s complexity. This official’s apparent endorsement of MARKAL to the evaluation team is supported by the position of the Ministry of Energy, which feels that MARKAL support will not be necessary beyond 2016. However, this must be tempered by the Ministry’s admission that the use of MARKAL for developing an LEDS was at least partly justified by the resources that had already been invested into it. Needless to say, this “sunk cost” argument only leads to the possibility of ongoing wasted resources. If this is true, it might be that a disproportionate share of Component 3 resources have been directed to MARKAL support, thus taking away required assistance in other areas (e.g., actually developing an LEDS from modelling results), especially when other simpler and lower cost alternatives may have been available. In any case, it can be seen above that the information provided to the evaluation team regarding MARKAL has been somewhat contradictory. However, when the evidence is taken together, it is clear, at least to the evaluation team, that the continued use of MARKAL is not sustainable at the Ministry of Energy without ongoing support from Remissia and DWG. Whether or not MARKAL should actually be replaced by a simpler model is an open question that requires further examination. To help in any possible further analysis of the MARKAL model, it might be beneficial to consult with USAID’s Governing for Growth (G4G) project in Georgia, which 22 contains an energy component that might be conducive for using MARKAL. It is understood that this project has at least examined the MARKAL model, its application by the Ministry of Energy and its outputs9. 4.2.4 Conclusions For Component 1, it can be concluded that: (i) although the project implementer has done an exemplary job of supporting municipalities, more could be done to build capacity and reduce dependence of municipal staff on external assistance; (ii) accessing financing beyond municipal budgets has proven to be difficult; and (iii) although Winrock claims to have “leveraged” $3.36 million for five projects through the partial grants program, closer analysis finds that the real amount is considerably less than that, and furthermore, it is not quite clear just how much credit the EC-LEDS grants program can really take for this financing. For Component 2, it is the evaluation team’s position that, because EE comprises only around 10% to 30% of the much wider GB concept, using this concept to achieve building EE is an intrinsically indirect way to achieve energy savings or GHG reductions in buildings. This is not completely in line with EC￾LEDS objectives. For this, and other reasons regarding the sustainability of GBCG, it is questionable whether further funding for Component 2 activities focused on GBs is desirable. For Component 3, a number of constraints may be behind the lack of progress in developing an LEDS. They include: (i) GOG leadership over the LEDS development process with limited capacity; (ii) a lack of initiative by Winrock; (iii) insufficient direction provided in the project description; and (iv) the complexity of the MARKAL model. The relative contribution of each of these constraints is not entirely clear. 4.3 EVALUATION QUESTION 3. HOW IS THE PROGRAM PERCEIVED BY THE GOG AND LOCAL MUNICIPALITIES THAT LEAD THE WORK FOR MORE EFFECTIVE DEVELOPMENT AND IMPLEMENTATION OF THE LEDS STRATEGY, SEAPS AND GREEN BUILDING CERTIFICATION AND RATING SYSTEM? AND HOW IS THE PROGRAM PERCEIVED BY OTHER NON-PUBLIC STAKEHOLDERS AND DIRECT BENEFICIARIES? 4.3.1 Findings – Component 1 The following perceptions on Component 1 have been observed through direct interviews with the beneficiary municipalities: • Municipality stakeholders characterized the project implementer as being good communicators, responsive, actively involved, and providing assistance of high technical quality. • All municipal key informants interviewed provided positive feedback on the assistance they were receiving from EC-LEDS in implementing their COM commitments. When asked, almost all reported that EC-LEDS made a major contribution to their EE work.10 • EC-LEDS trainings were described as “very useful” and “well organized, informative.” In one municipality, a respondent felt that the training sometimes came at the expense of other activities, and felt more project resources could be devoted to implementation. 9 “MARKAL Georgia Assumptions and Data Sources, Report on MARKAL Georgia Input for Business as Usual Scenario”, USAID report produced by Deloitte Consulting LLP for the Governing for Growth in Georgia project, November 2014. 10 When asked what the share of EC LEDS contribution was toward their EE efforts, most municipalities responded “80%”. Whether or not one regards the answer as oddly identical, it was clear that the EC LEDS support was appreciated. 23 • Technical assistance (including through Remissia) is highly valued in terms of the responsiveness of staff, good communication, and good facilitation (connecting with other programs, funding sources). 4.3.2 Findings – Component 2 The wider concept of GBs and GB ratings is regarded as a useful and relevant activity by the majority of those interviewed by the evaluation team. 26 of 27 of stakeholders (96%) that participated in a telephone survey said they would support the use of a well-established, existing, and internationally recognized GB certification scheme such as LEED or BREEAM. The COM municipalities, the government officials, and other donors that the evaluation team met also had a generally positive perception of GB and GB ratings in high level terms (in fact, nobody had a negative impression). However, when one probed a bit, it seemed that many people were more focused on the EE aspect of GBs, as 15 of the 27 respondents thought that an energy efficiency rating is more relevant than a wider GB rating for Georgia. The explanation is that the EE aspects are directly related to operational energy costs, whereas non-energy GB aspects are really a “nice to have” aspect of buildings. There was only a minimal sense that the public at large, developers, owners or tenants were prepared to pay any more for a “green” building. Only 11 of the 27 respondents thought that there would be demand for a rating covering all aspects of the GB concept. Based on the response rate to the survey (54%), the interest in GBs seems to be confined to a small core group of people and organizations in Georgia. This response rate seems somewhat low when it is considered that the names were provided by Winrock in the guise of GB stakeholders who had in one way or another participated in project-sponsored activities. In contrast, the perception gained is that the EE of buildings is perceived to be more directly related to lower energy costs and and/or warmer buildings in winter. This GB high level support in principle versus EE support in practice is shown by the fact that no GB ratings have been undertaken in five years of GBCG existence and two years of EC-LEDS support, whereas in around six months of Display building energy rating support, 18 municipality buildings and around 14 other buildings were labelled. 4.3.3 Findings – Component 3 There are different perceptions of Component 3, depending on the perspective of the stakeholder. As far as the GOG beneficiaries are concerned, perceptions are mixed, and depended largely on respective interests. Perceptions gathered from the interviews were: • Within the Ministry of Economy and Sustainable Development, only one stakeholder was available for an interview. This was within the Spatial Planning and Construction Policy Department. Not surprisingly, the discussion centered around building codes and Winrock’s support in comparing American and European codes, which was found to be useful. The Ministry’s participation in working groups and a working group focus on “strategic ideas and thoughts” were noted, as well as the need for a mid- to long-term plan, as opposed to individual projects with “no framework”. Finally, it was mentioned that the EC-LEDS project is helping the municipalities more than the GOG. • Ministry of Energy staff were enthusiastic about MARKAL and its applications. However, they noted that working group meetings had recently stopped and that the EC-LEDS process seemed to be missing a “driving force.” They stated that the EC-LEDS project is much needed in order to continue training on MARKAL. • A Deputy Minister of Energy was happy that the MARKAL model was finally being used by Ministry of Energy personnel to actually test the effects of various GHG policy measures. However, an overall strategy is needed. It is hoped that the project will produce a “solid” document for three or four sectors (e.g., Energy, Transport, Industry and Agriculture) that will cover 80% of emissions. Also, the Ministry of Economy and Sustainable Development should be leading the EC-LEDS effort, not Environment Protection, as this Ministry covers all sectors in the economy. • The Climate Change Service within the Ministry of Environment Protection, which is essentially 24 leading Component 3, found the project most useful in terms of MARKAL support and in the determination of mitigation measures. It, too, is expecting the development of an LEDS from Winrock and Remissia. Regarding other stakeholders, perceptions were: • An implementer (not Winrock) was effusive in underscoring the importance of Component 1 to the municipalities, but saw a problem with Component 3 in that the GOG expects tangible “materials” to be produced from the project that can be used as the basis for making decisions. Governments generally do not undertake analytical work. Others typically undertake this work, write reports and make recommendations. The government then makes decisions based on this and other information. • Among the donors interviewed - EBRD, kfW, and GIZ - only one, GIZ, knew about EC-LEDS’ Component 3 work. This was because GIZ is the GOG’s INDC advisor. GIZ is essentially “waiting” for an LEDS and is coordinating its efforts in providing support to preparing the INDC with the working groups - which requires LEDS input. Despite the lack of an LEDS, cooperation with the EC￾LEDS project has been “successful.” 4.3.4 Conclusions For Component 1, the program is seen positively by municipal staff, who expressed satisfaction regarding the quality of support, responsiveness, raising awareness, and facilitation. As they lack the necessary technical expertise, having project implementers manage SEAP development and revisions is highly valued. The challenges faced by the program, outlined above, are not attributed to the project implementer. For Component 2, the wider concept of GBs and GB ratings is generally regarded as a useful and relevant activity by those beneficiaries and stakeholders interviewed. However, there is minimal public willingness to pay for “green” buildings or “green” features in a building. There was, however, a perception of a strong link between EE and tangible energy savings, which is borne out by the relative success of the Display building energy rating support provided by Winrock. For Component 3, perceptions of EC-LEDS were generally positive. MARKAL support is perceived as the most important ongoing activity; however, GOG Ministries are expecting some sort of strategy document from the project. One implementer noted a possible disconnect between the GOG’s expectations and Winrock’s outputs. 5.0 RECOMMENDATIONS 5.1 WHAT COURSE-CORRECTION OR FURTHER WORK IS NEEDED FOR MEETING MAJOR OBJECTIVES OF THE PROGRAM BY WINROCK AND LOCAL IMPLEMENTER(S)? 5.1.1 Component 1 The following course-corrections are recommended: • Support institutional strengthening and sustainability of the process, rather than fostering a situation under which municipalities must rely on external support. In its remaining years, EC-LEDS should consider devoting more attention to the establishment of the municipal structure dedicated to the COM (such as establishing sustainable energy offices), instead of drafting SEAPs for municipalities. Future project work could be explicitly re-focused to follow a new, overarching principle of providing capacity and support (but on a transitional basis) for municipalities to ensure they are able to maintain COM commitments without reliance on donors. • To reduce dependence on a project that is slated to end, consider helping municipalities to shift to a 25 more simple approach to calculating emission reductions for the SEAPs, as opposed to using the currently complex BAU approach. This would give the municipalities a greater ability to prepare the SEAPs on their own, and generate greater local ownership as well. This does not necessarily mean adopting a “base year” approach (and adopting this approach might result in bigger problems over the longer-term), but developing a simplified BAU approach that municipalities can easily understand, One such possible approach may be to link BAU emissions to only GDP and population projections. This would be simple and transparent and should be compatible with the EU JRC approach. As there may be implications on a national planning level where MARKAL is used to make these estimates, the effect of such simplifications should be compared against national level results for consistency before being adopted. • Regarding the various municipal street lighting programs, rather than relying on project funding to replace a small number of street light fittings, the project should consider implementing a nationwide street light replacement program during the final years of the project. A technical assistance program would conduct analysis, piloting, and consultation, on behalf of all municipalities. Knowledge sharing would be at the national level via the “pilots” currently being implemented. Financing may be sought through borrowing, with the Ministry of Finance dealing with only a single request for borrowing approval. This approach would have significant economies of scale by reducing duplication of efforts. The total EE effect would be large, and budget savings would accrue to municipalities far sooner than if the current piecemeal approach is used. If possible, part of the transition award could be used to fund a national EE street lighting program and/or EE building program. • Find a way to help municipalities secure other more significant financing from sources such as EBRD. To date, most of the financing has been through municipalities’ own budget funds. EC-LEDS has already provided some assistance in accessing other financing sources, specifically to Tblisi and Zugdidi. 5.1.2 Component 2 Component 2 has now finished its current operations. The transition award to a local subcontractor has been extensively discussed and various options considered, with USAID declining to provide any transitional funding to GBCG. From the findings, the evaluation team agrees that this decision is sound, given the demonstrated lack of interest in clients to pay for GB ratings, the questionable capacity of GBCG to even be able to apply for direct funding under USAID eligibility criteria, the lack of any GBCG paid membership base after five years since its founding, and the lack of any long-term financial self￾sustainability plan for GBCG. However, the $1.38 million that Winrock sought to apply to GBCG for continued GB development and support is still available. Given the rapid progress achieved with the Display building energy rating tool since April 2015, the close alignment of the use of Display with the COM signatory municipalities’ responsibilities, the potential for Display to drive actual EE improvements in municipality, government, and private sector buildings in the municipality’s geographical area of responsibility, then some further support for Display by the EC-LEDS project in the next three years seems to be highly desirable. 5.1.3 Component 3 LEDS The most important course-correction for Component 3 is to push for the development of LEDS scenarios as quickly as possible, as these are long overdue in view of the project objectives and remaining time requirements. As previously mentioned, these scenarios need not be “approved” beforehand by the GOG but should be regarded as bases for discussion to help develop an LEDS. Given the lack of progress since issuance of the September 2014 BAU scenario report, it is unlikely that an acceptable-to-all strategy can be issued at the last moment by Winrock before it takes leave of its 26 full-time involvement in the project after September 2016, as seems to be the current plan. It is likely that considerable discussions will be required and, for that, a “working copy” of a draft strategy is required much before Winrock’s departure date. It is possible that Remissia can continue in helping to develop alternative strategies for the GOG. However, the shorter-term effort to September 2016 may be hampered by the fact that the GOG has recently requested an update of the BAU scenario using more recent and more reliable (2013) data. Ultimately, this data should be used to develop the LEDS, but conducting the update might take a while, thus further pushing back the ability to produce and conduct consultations on any possible draft LEDS. Decisions will therefore need to be made on whether to use dated information to develop initial strategies or to wait. The fact that the GOG made the request and is also supposedly leading the LEDS effort suggests that it will choose to take the latter course - unless it is feasible and acceptable for USAID to intervene (if it chooses to do so). Also, the mere fact that stale data will now be used to develop an LEDS may be completely unacceptable to some - even though the resulting mitigation strategies under the more recent dataset are not likely to be very different than those arising from the older data. For example, the possible measure of requiring all new buildings and/or existing publicly owned buildings to have a minimum standard level of insulation is very unlikely to change just because the BAU scenario changes. Energy savings, costs and resulting GHG reductions might change from one dataset to the next (and probably by not very much), but the measure will not. This, after all, is a “common sense” measure for Georgia, for which GHG reductions can be approximated using a back-of￾the-envelope calculation. Thus, the GOG’s inclination to keep re-visiting the BAU scenario is a likely barrier that USAID will need to address in pushing for the development of an LEDS as soon as possible. MARKAL It is evident that using MARKAL in Georgia is not sustainable without ongoing outside help, despite claims from the GOG to the contrary. Just to make updates requires support from DWG, although it could very well be that performing this update is not that obvious, as basic data may be provided in a different format than that used previously, which may then require new adjustments to be made to the data to “fit” into MARKAL or, perhaps parts of MARKAL may need to be modified to accept the new data. In any case, this expertise does not exist in Georgia. Even local MARKAL “experts” such as Remissia will clearly need this DWG support into the foreseeable future, beginning with the current planned update of the BAU scenario. If it is desired to continue using MARKAL, then provisions should be made to keep providing this support beyond 2016 (if the GOG will have it), as well as sufficient training so that continued support from DWG is eventually minimal (i.e., at the level that most software companies provide; e.g., software updates and not assistance in developing model inputs). It may also be desirable to examine the feasibility and acceptance of changing to more simple software. To help in the analysis, it might be beneficial to consult with USAID’s Governing for Growth (G4G) project in Georgia, which contains an energy component conducive to using MARKAL for certain applications. It is understood that this project has at least examined the MARKAL model, its application by the Ministry of Energy, and its outputs. Other EC-LEDS support beyond 2016 Given that it will be less than nine months to the end of the project at the time this report is issued, it is quite possible that support to the GOG may be required beyond September 2016 for developing an acceptable LEDS. The terms of reference for such assistance should be more prescriptive on the work to be accomplished than the description provided in the current CA. Implementation assistance Once accepted, an LEDS will need to be implemented. An obvious implementation activity would be 27 the development of laws and regulations in line with the overall LEDS. First, however, an action plan identifying all the required activities and their timing will need to be developed. More areas of required assistance might be identified within such a plan. Certain donor coordination will likely be required, with other agencies getting involved in areas that would be affected by the LEDS, as evidenced by the current GIZ involvement in developing Georgia’s INDC. Also, EBRD will be financing a National Energy Efficiency Action Plan that will need to be coordinated with an LEDS, as will an upcoming Danish financed program “Support to Energy Efficiency and Sustainable Energy in Georgia” At a lower level, several Nationally Appropriate Mitigation Actions (NAMAs) are being funded by various donors and being implemented by different organizations. Although not directly affected by an LEDS, the implementation of such projects will need to be consistent with the provisions and priorities laid out in future laws and regulations. 5.2 OVERARCHING ISSUES AND RECOMMENDATIONS In conducting the evaluation, a number of overarching issues have come to light, which affect the project as a whole. These are not technically a part of the evaluation, but they must be highlighted, as they relate heavily to achieving the overall project goals and therefore cannot be ignored: • With reference to the project indicators, it should be mentioned that progress appears to be much better on outputs - e.g., preparing SEAPs, providing training, raising awareness of GBs among developers - than outcomes such as GHG reductions and energy saved. The truth is that no grant money has yet been disbursed. Also, no LEDS has been prepared, even for discussion purposes. The most significant and challenging aspects of the project are yet to be realized, with less than nine months remaining in Winrock’s full-time engagement in the project • As mentioned earlier and as per the CA, Components 1 and 2 of EC-LEDS were intended be implemented throughout the five years of EC-LEDS, to be continued by a local organization over the last two years of the program. For a number of reasons, the original plan to transition to a local firm is at risk and probably will not happen without strong corrective actions by USAID. To discontinue the program would be unfortunate, first because its goals are critical and also because discontinuance would be against the spirit of the recently launched USAID Forward reform program, under which local participation in such programs is being promoted. Corrective actions may include, among other possibilities: • Engagement of a local partner familiar with energy efficiency issues that meets USAID eligibility criteria to act as an umbrella organization over institutions such as GBCG and Remissia, who appear to not be eligible for USAID funding for the purpose of transitioning; • Possible provision of support from the recently launched HICD 2020 project, which is meant to “achieve tangible improvements in the human and institutional capacity of USAID’s strategic partner organizations in Georgia including governmental, non-governmental, and for-profit entities”. • Significant interventions by USAID and Winrock appear to be required in order to achieve any course corrections between now and September 2016. To start, USAID should develop a firm schedule of deliverables with Winrock to cover the following: - Finding a qualified local firm for the Years 4-5 transition phase; - Setting milestones for the development of a draft LEDS working paper for discussion with the GOG that will lead to the eventual development of an actual LEDS. According to USAID, there is already a schedule for providing a draft LEDS in the March/April 2016 timeframe; however, this may be affected by the GOG’s current desire to develop another BAU scenario; - Working on expanding the Display labelling program. 28 To accomplish the above in the limited time available, a new approach to implementing the Components on the part of Winrock will be required. To this end, staff changes at Winrock may be necessary. 29 ANNEXES 30 ANNEX 1: SCOPE OF WORK 31 1. Scope Name of the Project to be evaluated: Enhancing Capacity for Low Emissions Development Strategies Clean Energy Program Project Number: AID-114-A-13-00008 Project Dates: September 27, 2013 – September 26, 2017. Project Funding: $6,076,168 Implementors: Winrock International AOR: Nicholas Okreshidze Non-personal services for a mid-term performance evaluation 1 of the Enhancing Capacity for Low Emissions Development Strategies Clean Energy Program (EC-LEDS). 2. Purpose This evaluation must assess strengths and weaknesses of the project and provide recommendations to USAID for course corrections or further work and inform future planning. The results of the evaluation will be used by USAID/Caucasus for improving ongoing interventions in the areas of (1) institutionalization and implementation of climate change mitigation measures in Georgian target municipalities, (2) promotion and facilitation of private sector investments in energy efficiency and green buildings, and (3) building capacity of the GOG to develop and implement national Low Emissions Development Strategy in support of the USG’s EC-LEDS initiative. The primary audience of the evaluation will be USAID and in particular its Economic Growth (EG) office, and the implementing partner (Winrock). The results of the study will be shared with local stakeholders (Ministries of environment, energy economy, regional development, partner NGOs, municipalities, etc.,) and other donors working in this area. Finally, evaluation results will also be used for reporting purposes to Washington- based stakeholders. 3. Summary of Specific Technical Requirements The Contractor must: - Teleconference with USAID/Georgia to discuss the upcoming work. - Submit detailed evaluation design and workplan to the Task Order COR (TOCOR) prior to the team’s visit to Georgia. - Provide incoming briefing for USAID management to present the detailed evaluation design. - Conduct field work in accordance with the USAID-approved evaluation design and workplan. - Conduct outgoing briefing for USAID management to present the preliminary findings of the evaluation. - Provide evaluation report to USAID TOCOR in accordance with reporting guidelines. - Submit USAID-approved evaluation report to Development Experience Clearinghouse 32 (DEC) within 30 calendar days following the acceptance of the report by the TOCOR. - Submit quantitative dataset in a machine-readable format to the Development Data Library (DDL) as part of the Open Data initiative. 4. Key evaluation questions to be addressed Question 1: What are the major strengths / accomplishments of the EC LEDS program? • The evaluation team must review actual progress toward achieving key expected results and identify major accomplishments, as well as the strengths of the program’s implementation approaches. Question 2: What are the constraints and challenges that inhibit the EC-LEDS’ progress toward achieving the program objectives during the remaining term of the program? What are the outstanding needs? • The evaluation team must determine major constraints and challenges that hinder the achievements of the program’s major objectives and identify outstanding needs. Major program objectives include: a reduction of GHG emissions in Georgia by at least 236,372.9 metric tons of CO2 equivalent; facilitation of up to $14 million in private sector investments in clean energy; and energy savings of up to 315 GWh (the equivalent of approximately $22 million). In answering this question, the evaluation team must examine the following four priority support areas under EC/LEDS program: 1) progress of Georgian municipalities in institutionalizing and implementing climate change measures; 2) development and adoption of a voluntary system for rating and certifying green buildings in Georgia; 3) capacity￾building of the GOG EC/LEDS committee to develop the national LEDS strategy; and 4) engagement of representatives from various GOG ministries and various stakeholders in the preparation and/or adoption of SEAPs, LEDS strategy and Green Building Certification and Rating system. • GOG stakeholders include the Ministry of Environment, Ministry of Energy and Ministry of Economy and Sustainable Development which are the lead ministries of the GOG’s LEDS process, as well as participating municipalities and other GOG ministries and agencies regarding the SEAPs and Green Building Certification and Rating System. The GOG engagement can be determined as the level of support that the ministries and other GOG players have been providing to the program, such as delivering LEDS-related data and information, leading the LEDS strategy development, and advancing the development of a voluntary system for rating and certifying green buildings. Engagement of municipalities can be defined through determining the progress of municipalities in facilitating the development of SEAPs. Question 3: How is the program perceived by the GOG and local municipalities that lead the work for more effective development and implementation of the LEDS strategy, SEAPS and Green Building Certification and Rating System? How is the program perceived by other non- public stakeholders 33 and direct beneficiaries (e.g. construction companies, developers, architectural companies, educational institutions involved in green building rating and certification institutionalization activities of the program)? • Program is defined as assistance provided to date and planned, including results achieved/implementation progress to date, outstanding needs moving forward, nature of collaboration, and participating actor contributions, roles, and responsibilities. Question 4: What course-correction or further work is needed for meeting major objectives of the program by Winrock and local implementer(s)? • Based on the findings and conclusions related to questions#1 and question#2, the evaluation team must make specific recommendations with regard to EC/LEDS implementation approaches. For example, the team may identify opportunities to improve the collaboration with GOG stakeholders, or suggest additional interventions to reduce GHG emissions or leverage public/private sector clean energy funds. • The answer to this question will be used to inform the programmatic approaches under the first phase of the award (currently under implementation by Winrock International) as well as to guide activities under the second phase of the award (under one or more local implementers). • Course corrections may be defined by any size, e.g. small ideas for more effective communication, or large course corrections to achieve programmatic results. 5. Methodology The Contractor must propose the best methods that minimize bias and provide strong evidence. The Contractor must suggest the use of various data collection and analysis methods, both quantitative and qualitative, such as document review, key informant interviews, focus group discussions, survey instruments, and others. The Contractor must justify their inclusion of any data collection methodology as well as their selection process for all methodologies. For example, for a survey or mini-survey (if proposed), the number of respondents and their selection process should be explained and justified, and methodology for survey administration proposed. The same is true for key informants, focus group discussions, and other methods as well. Selected respondents should be representative of women, youth, and vulnerable groups, where appropriate. The Contractor must develop a detailed evaluation design, including a data collection plan and data collection tools. The evaluation design must explain how the evaluation Contractor intends to conduct the study in detail, including a detailed description of one or more proposed methodologies as well as limitations of proposed methodologies. The proposed research design must explain in detail what methods will be used to obtain answers for each evaluation question. The evaluation Contractor must explain in detail how the proposed methodology (mix of methods) to conduct the study generate evidence to ensure rigor and reliability of results; and how and why the proposed methodology will minimize bias. The evaluation design must include a detailed evaluation matrix (the illustrative matrix is given below). The design must also include the data analysis plan for each question, draft questionnaires (to be included as an attachment), and other data collection instruments or their main features, criteria 34 for assessing responses to evaluation questions, known limitations to the evaluation design, and a dissemination plan. The evaluation design must also include specific sub-questions for each evaluation question, where needed. The evaluation matrix, including its sub-questions, provided below is only illustrative and USAID expects that the Contractor will suggest the best methods that would generate the most reliable and evidence-based answers to the key evaluation questions. Illustrative evaluation matrix: Evaluation Question Definition Data Source Methodolo gy Question 1: What are the major strengths / accomplishments of the EC LEDS program? • The evaluation team must review actual progress toward achieving key expected results and identify major accomplishments, as well as the strengths of the program’s implementation approaches. Project documentation: weekly and quarterly reports, M&E plan, results framework, other reports. Municipal governments, GOG ministries, project staff, partner NGO staff, USAID/EG Office, training/TA/grant recipients, international financial institutions, other donors, other stakeholders, Green Building Council of Georgia, construction companies, developers, architectural companies, educational institutions, and other entities involved in green building rating and certification Key Informant Interviews Focus Group Discussions Direct Observation Program documentat ion review Question 2: What are the constraints and challenges that inhibit the EC-LEDS’ progress toward achieving the program objectives during the remaining term of the program? What are the outstanding needs? • The evaluation team must determine major constraints and challenges that hinder the achievements of the program’s major objectives and identify outstanding needs. Major program objectives include: a reduction of GHG emissions in Georgia by at least 236,372.9 metric tons of CO2 equivalent; facilitation of up to $14 million in private sector investments in clean energy; and energy savings of up to 315 GWh (the equivalent of approximately $22 million). In answering this question, the evaluation team must examine the following four priority support areas under EC/LEDS program: 1) progress of Georgian municipalities in institutionalizing and implementing climate change measures; 2) development and adoption of a voluntary system for rating and certifying green buildings in Georgia; 3) capacity-building of the GOG EC/LEDS committe to develop the national LEDS strategy; and 4) engagement of representatives from various GOG ministries and various stakeholders in the preparation and/or adoption of SEAPs, LEDS strategy and Green Building Certification and Rating system. • GOG stakeholders include the Ministry of Environment, Ministry of Energy and Ministry of Economy and Sustainable Development which are the lead ministries of the GOG’s LEDS process, as well as participating municipalities and other GOG ministries and agencies regarding the SEAPs and Green Building Certification and Rating System. The GOG engagement can be determined as the level of support that the ministries and other GOG players have been providing to the program, such as Evaluation Question Definition Data Source Methodolo gy delivering LEDS-related data and information, leading the LEDS strategy development, and advancing the development of a voluntary system for rating and certifying green buildings. Engagement of municipalities can be defined through determining the progress of municipalities in facilitating the development of SEAPs. Question 3: How is the program perceived by the GOG and local municipalities that lead the work for more effective development and implementation of the LEDS strategy, SEAPS and Green Building Certification and Rating System? How is the program perceived by other non-public stakeholders and direct beneficiaries? • Program is defined as assistance provided to date and planned, including results achieved / implementation progress to date, outstanding needs moving forward, nature of collaboration, and participating actor contributions, roles, and responsibilities. • Other Non-public stakeholders and beneficiaries include construction companies, developers, architectural companies, educational institutions involved in green building rating and certification institutionalization activities of the program. Municipal governments, GOG ministries, project staff, partner NGO staff, USAID/EG Office, training/TA/grant recipients, international financial institutions, other donors. Green Building Council of Georgia, construction companies, developers, architectural companies, educational institutions, and other entities involved in green building rating and certification Key Informant Interviews Focus Group Discussions Mini￾Survey (if required) Evaluation Question Definition Data Source Methodolo gy Question 4: What course￾correction or further work is needed for meeting major objectives of the program by Winrock and future local implementer(s)? • Based on the findings and conclusions related to questions#1 and question#2, the evaluation team must make specific recommendations with regard to EC/LEDS program implementation approaches. For example, the team may identify opportunities to improve the collaboration with GOG stakeholders, or suggest additional interventions to reduce GHG emissions or leverage public/private sector clean energy funds. • The answer to this question will be used to inform the programmatic approaches under the first phase of the award (currently under implementation by Winrock International) as well as to guide activities under the second phase of the award (under one or more local implementers). • Course corrections may be defined by any size, e.g. small ideas for more effective communication, or large course corrections to achieve programmatic results. Project documentation: weekly and quarterly reports, M&E plan, results framework, other reports. Municipal governments, GOG ministries, project staff, partner NGO staff, USAID/EG Office, training/TA/grant recipients, international financial institutions, other donors, other stakeholders. Green Building Council of Georgia, construction companies, developers, architectural companies, educational institutions, and other entities involved in green building rating and certification. Key Informant Interviews Focus Group Discussions 6. Work location Tbilisi and selected Georgian regions, and the U.S. The teams will travel outside the capital as needed to visit Georgian municipalities (e.g. Rustavi, Poti Zugdidi, Gori, Batumi, Kutaisi, Akhaltsikhe, Telavi ) in order to meet with key players in diverse parts of the country and to get a better sense of the overall context within Georgia. 7. Summary of skills and qualifications of the evaluation team and suggested level of effort The Team Leader (international) must have demonstrable experience in conducting evaluations and/or assessments in energy policy, energy efficiency and/or low emission. He/she must have extensive knowledge and experience in USG’s efforts for the development of LEDS strategies in other countries. Experience in Georgia and/or in the Europe and Eurasia region will be an advantage but is not required. The team leader will be responsible for the day to day management of the team, data collection and synthesis, presentations and final reports. Fluency in English language is required. Evaluation Expert must have justifiable experience in planning and conducting evaluations using various data collection and analysis methodologies, preferable (not required) in the energy sector. Evaluation Expert will also travel to the field. He/she will review documents, develop evaluation design and instrument, and assist in report writing. Green Building Certification and Rating Expert will have experience and/or knowledge in either of the existing rating systems for certifying green (environmentally sustainable and energy efficient) buildings, including but are not limited to: EU Energy Performance in Buildings Directive; Energy Star; LEED Green Building; Building Codes Assistance Project (BCAP); and Collaborative Labeling & Appliance Standards Program (CLASP). He/she will have a knowledge and experience in introduction of market-driven rating and certification system for green buildings in developing countries as Georgia is. Locally-hired experts/consultants must have justifiable experience in energy efficiency, energy strategy, green building, low emissions, and/or other related field. Experience of participating as a team member in conducting a USAID or other donor-funded project assessments/evaluations will be an advantage. English language knowledge is a requirement for locally hired staff. The Contractor must provide information about the selected evaluation team members including their CVs, and explain how they meet the requirements set forth in the SOW. USAID may request an interview with any of the proposed evaluation team member/s via conference call/Skype or any other means available. All Team members will be required to provide a signed statement attesting to a lack of conflict of interest, or describing an existing conflict of interest. The Evaluation team shall demonstrate familiarity with USAID’s Evaluation Policy (http:/www.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf) The following levels of effort are illustrative and should serve only as an example of the staff that may be mobilized under this Task Order. These levels may not reflect the actual level of effort contracted, and the contractor will be expected to submit its own estimate of the level of effort needed to fulfill the objectives. No of Work Days in Country /Consultant No of Days for preparation and Report Writing Total No of Work Days /Consultant International Technical Expert – Team Leader 19 14 33 (plus travel days) Evaluation Expert 19 8 27 (plus travel days) Local Consultant 19 4 23 Green Building Certification and Rating Expert 19 8 27 (plus 2 travel days) A six-day work week will be authorized in Georgia. 8. Deliverables The contractor will be required to provide USAID with the following deliverables: a. Final Work Plan and Evaluation Design Final Work Plan and Evaluation Design document for the evaluation shall be completed by Contractor and presented to the TOCOR three days prior to the team’s arrival in country. The evaluation design will include a detailed evaluation design matrix (including the key questions, methods and data sources used to address each question and the data analysis plan for each question), draft questionnaires and other data collection instruments or their main features, known limitations to the evaluation design, and a dissemination plan. The final design requires TOCOR approval. The work plan will include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation team. b. In-brief with the mission: will be held within five days of the team’s arrival in country. This will be a maximum of 30 minute presentation of the plan, namely, how the questions asked in SOW will be answered. Prior to in brief, the evaluation team may have working meeting/s with TOCOR and Winrock AOR to agree all the details of the design. c. Conduct fieldwork: The in-country evaluation must expand upon the analysis in the desk review and in the facilitated discussion through methods proposed by the evaluation team that might include interviews with focus groups of sub-contractors, beneficiaries or end- users, Georgian government, engineering companies, other private sector entities, field visits, and mini-survey, if proposed. The evaluation team should spend 19 work days in- country. d. Mission out-brief: After finishing the fieldwork, the evaluation team must present an outline (in bullets, possibly in power point or as a handout) of the evaluation report with general findings, conclusions, and anticipated recommendations. The evaluation report must follow the “Criteria to Ensure the Quality of the Evaluation Report” included in Appendix 1 of the attached USAID Evaluation Policy. This presentation of preliminary findings will take place two-four days prior to the evaluation team leader’s departure from Georgia. The team will present their findings to USAID during a debriefing for all interested USAID staff at the end of their visit in Georgia. e. Draft reports: The Contractor must submit to TOCOR a draft report within 20 working days of completing the out-briefing with USAID. This document must explicitly respond to the requirements of the SOW, answer the evaluation questions, be logically structured, and adhere to the standards of the USAID Evaluation Policy of January 2011, and the criteria to ensure the quality of the evaluation report. The reports must not exceed 25 pages, excluding executive summary and annexes. f. Final Evaluation Report The Contractor must incorporate USAID’s comments and submit the final report to TOCOR within five (5) working days following receipt of the final batch of USAID’s comments on the draft report. The Contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse at http://dec.usaid.gov within 30 calendar days of final approval of the formatted report with USAID consent. In case it is determined that the full report includes sensitive information, the Contractor must produce sanitized version for submission to DEC; the latter also requires TOCOR’s clearance. The evaluation final report should include an executive summary, introduction, background of the local context and the projects being evaluated, the main evaluation questions, the methodology or methodologies, the limitations to the evaluation, findings, conclusions, and recommendations and lessons learned (if applicable). The executive summary should be 3-5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include: • The Evaluation Scope of Work • Any “statements of differences” regarding significant unresolved difference of opinion by funders, implementers, and/or members of the evaluation team • All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides • Sources of information, properly identified and listed • Disclosure of conflicts of interest forms for all evaluation team members, either attesting to a lack of conflict of interest or describing existing conflict of interest. g. All records from the evaluation (e.g. interview transcripts and summaries, focus group transcripts, code books, etc.) must be provided to the evaluation TOCOR as requested. All quantitative data collected by the evaluation team must be provided in an electronic file in easily, machine readable format agreed upon with the TOCOR. The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. In addition, the dataset must be submitted to the Development Data Library (DDL) as part of the Open Data initiative. Reporting Guideline The illustrative format for the final evaluation report is as follows: 1. Executive Summary—summarizes key points, concisely states the purpose, background of the project, main evaluation questions, methods, findings, conclusions, recommendations and any lessons learned; should be sufficiently detailed, yet brief, to serve as a stand-alone product (3-5 pp) 2. Introduction—state the purpose, audience, and outline of the evaluation (1 pp) 3. Background—provide a brief overview of the project and the study implemented (1-2 pp) 4. Methodology— the evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology. Greater detail can be included in the appendices (2-3 pp); 5. Findings/Conclusions/Recommendations—explicitly answer each evaluation question; the report should distinguish between findings (the facts), conclusions (interpretation of the facts), and recommendations (judgments related to possible future programming) (10-15 pp); however it should be clear what is the link between them; 6. Lessons Learned (if not covered in findings, conclusions and recommendations) (2–3 pp); 7. Annexes—annexes must include this statement of work and its modifications (if any); any “statements of differences” regarding significant unresolved difference in opinion by funders, implementers, and/or members of the evaluation team; a glossary of terms; sources of information, properly identified and listed; clear documentation of schedules, meetings, interviews and focus group discussions, and any tools used in conducting the evaluation, such as focus group scripts or questionnaires, checklists and discussion guides used; and signed disclosures of conflict of interest. The evaluation design should also be attached to the report. The report format should be presented in Microsoft Word and use 12-point type font throughout the body of the report, using page margins 1” top/bottom and left/right. The body of the report should ideally be within 20-25 pages, excluding the executive summary, table of contents, references and annexes. The final report must follow USAID branding and marking requirements. Per the USAID evaluation policy, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report.2 • The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the projects, what did not and why. • Evaluation reports shall address all evaluation questions included in the statement of work. • The evaluation report should include the statement of work as an annex. • Evaluation methodology shall be explained in detail, and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. • Sources of information shall be properly identified and listed in an annex. • Recommendations shall be supported by a specific set of findings. • Recommendations shall be action-oriented, practical and specific, with defined responsibility for the action. 9. Logistical Support USAID/Caucasus will provide an initial list of in-country contacts two weeks prior to the team’s arrival but will not assist in the logistics of appointing meetings. Hence, the Mission will not be responsible for arranging logistics for the evaluation team. The Contractor must suggest how they plan to arrange translation, transportation, and logistical support to the evaluation team. 10. Project Documents for Review The TOCOR, through the Mission’s Economic Growth office and the AOR of the EC-LEDS activity, will put the contractor in contact with its implementing partner and may provide help with a small number of meetings (such as meeting with USG agencies). Relevant reports and other project documentation will be provided by the Mission to the Contractor two weeks prior to travel to Georgia. The evaluation contractor shall initiate Washington-based work by reading reports and familiarizing him/herself with the projects. These documents are: 1. Cooperative Agreement 2. Winrock International annual, quarterly, and weekly reports 3. Winrock International work plans 4. PMP indicator tables; 5. M&E plans submitted and approved by USAID; 6. Relevant studies/assessments; 7. Other projects documents; 8. Initial list of in-country contacts 11. Other Requirements The evaluation team must be familiar with USAID’s Human Subject Protection Policy and USAID’s Evaluation Policy (http://www.usaid.gov/evaluation). The evaluation team must provide adequate training for its survey staff on survey methodology, USAID’s survey regulations, other relevant regulations, and the data collection plan. The contractor has the responsibility to safeguard the rights and welfare of human subjects involved in the survey research supported by USAID. USAID has adopted the Common Federal Policy for the Protection of Human Subjects, Part 225 of Title 22 of the Code of Federal Regulations (http://www.usaid.gov/policy/ads/200/200mbe.pdf). Recipient organizations must familiarize themselves with the USAID policy and provide “assurance” that they will follow and abide by the procedures of the Policy. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline, need to be agreed upon in writing by the Contracting officer. ANNEX 2: WORK PLAN 1. INTRODUCTION In the preparation of the Work Plan and Evaluation Design for the mid-term performance evaluation of the USAID Enhancing Capacity for Low Emissions Development Strategies Clean Energy (EC-LEDS) Program, the Evaluation Team has followed the guidelines outlined in USAID’s Request for Task Order Proposal (RFTOP) and accompanying Statement of Work (SOW) contained in Annex 1, together with ME&A’s Technical Proposal According to the SOW, the purpose of this evaluation is to “assess strengths and weaknesses of the project and provide recommendations to USAID for course corrections or further work and inform future planning.” Specifically, the assignment is to address four specific evaluation questions and will cover EC-LED’s activities over the period September 2013 (date of signing with the Contractor) to present. 2. EVALUATION TEAM The evaluation of EC-LEDS will be conducted by a team that consists of three international experts, Mr. Arvid Kruze (Team Leader), Mr. Nils Junge (Evaluation Expert) and Mr. Frank Pool (Green Building Certification and Rating Expert), as well as a Local Expert, Mr. Giorgi Abulashvili. Mr. Kruze will assume overall responsibility for the management of the evaluation in collaboration with USAID/Georgia. In short, this encompasses all activities specified in the Evaluation Schedule (Annex 2), including pre￾mobilization, on-site implementation and end-of-assignment deliverables. Mssrs. Junge, Pool and Abulashvili will contribute to the evaluation mission by assisting in preparing documents, conducting small sample surveys and key informant interviews, supporting the organization of focus group discussions (FGDs), and carrying out observations, site visits and additional research as identified by the Team Leader. They will further participate in the planning and implementation of regional field visits (where appropriate) as well as contributing towards the preparation of the draft and final evaluation reports. The above team will be supported by the local organization IRMS which will be responsible for conducting a survey of project beneficiaries. Finally, oversight of the evaluation mission will fall under the remit of Ms. Mirela McDonald, Evaluation IQC Manager with ME&A. 3. EVALUATION TASKS and SUB-TASKS 3.1 Pre-Mobilization Activities Communication A number of email exchanges have taken place since the Task Order was signed among the Evaluation Team members, ME&A Project Manager and Project Coordinator, and USAID staff in order to coordinate the mobilization activities, in general, and to obtain relevant documents, clarify project and evaluation approaches, identify informants, and begin scheduling meetings, in particular. Desk Study Prior to mobilization, the Evaluation Team received a number of EC-LEDS project-related documents from USAID, including: (i) Work Plans, (ii) Annual Reports, (iii) Quarterly Reports, (iv) Contract No. AID-114-A-13-00008 signed between USAID Caucasus and Winrock International Institute for Agricultural Development (Winrock), (v) a project appraisal document package, and (vi) various technical documents and press releases associated with the services currently being carried out by Winrock. All documents provided to the Evaluation Team have been reviewed and used as a source of reference in preparation of this Work Plan. Draft Work Plan and Evaluation Design A Draft Work Plan and Evaluation Design (this document) has been prepared in collaboration with ME&A staff. Following the in-briefing with USAID in Tbilisi, the Team Leader will revise and/or edit the Work Plan and Evaluation Design to accommodate any corrections or adjustments requested by USAID resulting from that meeting. 3.2 In-Country Activities Initial Meetings The Evaluation Team will meet with USAID on Wednesday, October 21, 2015, to discuss the draft Work Plan. We then hope to meet with the implementing partner, Winrock, immediately following the Team’s in-brief with USAID/Georgia, in order for both parties to confirm their understanding of the purpose of the evaluation together with its expectations. These discussions will provide the opportunity to clarify the proposed activities of the Evaluation Team and, at the same time, contribute towards refining the list of interviewees and planning of the evaluation schedule as outlined in the Work Plan. It is proposed that meetings with high-level stakeholders such as the Ministry of Energy will begin the following day, October 22. Data Collection As highlighted in the Evaluation Schedule (see Annex 2), immediately following the in-briefing, the Evaluation Team will begin carrying out interviews with EC-LEDS’s staff, partners, stakeholders and beneficiaries, as well as a cross-section of other relevant individuals and organizations with an interest in the project’s activities and outcomes. The Team’s investigations will initially focus on Tbilisi and then roll out to regions, municipalities, and districts nationwide. This will continue until Friday, November 6, 2015, following which work will begin on preparing a summary of findings, conclusions and recommendations that will be presented to USAID at an out-briefing on Tuesday, November 10, 2015. A preliminary Draft Report will subsequently be submitted to USAID/Georgia by December 8, 2015. Due to the extent of EC-LEDS’s activities, it has been determined that in addition to face-to-face meetings with identified stakeholders in Tbilisi and in the surrounding regions, the most effective data￾gathering approach will need to encompass a range of diverse data gathering methods if USAID’s proposed questions are to be satisfactorily answered. These methods will include Focus Group Discussions (FGDs) and a mini-survey. FGDs will be conducted in the municipalities (district locations to be discussed and agreed with the EC￾LEDS team and USAID) with the purpose of obtaining feedback from the project’s main beneficiaries. The precise number of FGDs will be determined following discussion at the USAID in-briefing and subsequent meeting with the entire EC-LEDS team shortly thereafter. For both face-to-face interviews and FGDs, specifically designed questionnaires will be prepared (see Annex 3 for examples) to be used as aide-memoires for note-taking and later referenced when preparing the draft Final Report. Specific approaches for both qualitative and quantitative data gathering and analysis, as well as methodology for the evaluation are outlined in Section 5. Mini-survey. The Evaluation Team will also conduct a mini-survey with the support of IRMS. The survey will have 10-15 questions and it will be in the form of a census, i.e. targeting 100% of the 50 non-public stakeholders and beneficiaries identified on the provided list of ‘Green Building Contacts’, and representing some 37 different organizations. Some questions will be close-ended with prepared response categories (e.g. yes/no/don’t know, on Likert-type 5-point scale) and some will be open-ended, qualitative. The interview is expected to last no more than 15-20 minutes. Data Analysis Throughout the in-country period of the evaluation, the Evaluation Team will meet regularly to discuss the outcome of each day’s activities. As members may be attending some meetings individually, it is essential that the Team meet regularly to discuss, digest and analyze data and information gathered during the key informant interviews, FGDs and site visits. Quantitative data collected will be analyzed using established evaluation techniques and industry standard data analysis tools. For qualitative data resulting from stakeholder interviews, where much of the evidence may be anecdotal or inferred, the Team will use triangulation to identify any inconsistencies and ensure reliability. Triangulation will assist the Evaluation Team to reduce the “response bias” in which respondents tend to tell the evaluators what they want to hear. The majority of data analysis will take place in the final days that the Evaluation Team is in Georgia (November 7-10), in preparation for the out-briefing with USAID/Georgia. 3.3 End-of-Evaluation Activities Out-briefing On Tuesday, November 10, 2015, the Evaluation Team will conduct an out-briefing with USAID/Georgia during which initial preliminary findings will be presented in summary format together with key issues arising from the evaluation. This will take the form of a PowerPoint presentation supplemented by briefing notes for reference purposes. At this stage, comments and/or suggestions offered by USAID/Georgia will be acknowledged and addressed in the draft Final Report. Completion and Submission of Draft Report to USAID On completion of the in-country mission and following the Team’s return to home base, a Draft Final Report will be prepared and submitted to USAID/Georgia Tuesday, December 8, 2015. Final Report submitted to USAID with integrated comments By Tuesday, January 5, 2016, it is envisaged that comments of USAID/Georgia for integration into the Final Report will have been received. The Final Report will subsequently be re-submitted to USAID/Georgia no later than Tuesday, January 12, 2016. 4. EVALUATION DESIGN MATRIX In preparing the evaluation design shown in the table immediately below, the Evaluation Team took into consideration the various documents of the EC-LEDS project forwarded by USAID. Following an in￾depth review of this material and considering the nature of the evaluation questions, the Evaluation Team prepared the following design matrix depicting each evaluation question. As seen in the matrix, each evaluation question relates to every one of the three Components under EC-LEDS. Detailed analysis of each of these questions will be presented in the main body of the Draft and Final Reports. Evaluation Questions Indicators / issues Data Source Methodology Question 1: What are the major strengths / accomplishments of the EC LEDS program? Pertinent to: Component 1 - GeMunee Component 2 - Green Building Rating and Certification System Component 3 - Working Group and Advisory Assistance • SEAPs in progress/ completed (Component 1) • Number/type of measures and activities launched • Number/ type of activities for which financing has been obtained • Number/type of activities implemented • Municipal measures introduced (Component 1) • Cooperation levels • Institutional measures developed • GHG emissions reductions • Funding disbursement levels (Component 1) • Progress against plan (PMP) • Additional financing facilitated for municipal pilot projects (Component 1) • General relevance, effectiveness, cost efficiency and sustainability of each component Project documentation: weekly and quarterly reports, PMP, results framework, technical reports, press releases, other reports. Stakeholders: Municipal governments, GOG ministries, project staff, partner NGO staff, USAID/EG Office, training/TA/grant recipients, international financial institutions, other donors, other stakeholders, Green Building Council of Georgia, Survey of non-public stakeholders, other entities involved in green building rating and certification Key Informant Interviews Focus Group Discussions Telephone interview survey Direct Observation Analysis of tools Mini Case Studies Question 2: What are the constraints and challenges that inhibit the EC￾LEDS’ progress toward achieving the program objectives during the remaining term of the program? What are the outstanding needs? Pertinent to: Component 1 - GeMunee Component 2 - Green Building • SEAPs in progress/ completed (Component 1) • Number/type of measures and activities launched • Number/ type of activities for which financing has been obtained • Number/type of activities implemented • Municipal measures introduced (Components 1 and 2) • Cooperation levels • Institutional measures developed Project documentation: weekly and quarterly reports, PMP, results framework, other reports. Stakeholders: Municipal governments, GOG ministries, project staff, partner NGO staff, USAID/EG Office, training/TA/grant recipients, international financial institutions, other donors, other stakeholders, Document Review Key Informant Interviews Focus Group Discussions Telephone interview survey Direct Observation Analysis of tools Mini Case Studies Evaluation Questions Indicators / issues Data Source Methodology Rating and Certification System Component 3 - Working Group and Advisory Assistance • GHG emissions reductions • Funding disbursement levels • Progress against plan (PMP) • Additional financing facilitated for municipal pilot projects • Funding needs versus availability • General relevance, effectiveness, cost efficiency and sustainability of each component Green Building Council of Georgia, Survey of non-public stakeholders, other entities involved in green building rating and certification Question 3: How is the program perceived by the GOG and local municipalities that lead the work for more effective development and implementation of the LEDS strategy, SEAPS and Green Building Certification and Rating System? How is the program perceived by other non-public stakeholders and direct beneficiaries? Pertinent to: Component 1 - GeMunee Component 2 - Green Building Rating and Certification System Component 3 - Working Group and Advisory Assistance • Stakeholder perceptions and ratings relating to: o Program design o Program implementation o Program relevance o Expected impacts Stakeholders: Municipal governments, GOG ministries, project staff, partner NGO staff, USAID/EG Office, training/TA/grant recipients, international financial institutions, other donors, other stakeholders, Green Building Council of Georgia, Survey of non-public stakeholders, other entities involved in green building rating and certification Key Informant Interviews Focus Group Discussions Telephone interview survey Direct Observations Question 4: What course-correction or further work is needed for meeting major objectives of the program by Winrock and future • Communication effectiveness • Collaboration levels with GOG stakeholders • Additional interventions to reduce GHG Project documentation: weekly and quarterly reports, M&E plan, results framework, other reports. Stakeholders: Municipal Document Review Key Informant Interviews Telephone interview survey Focus Group Discussions Evaluation Questions Indicators / issues Data Source Methodology local implementer(s)? Pertinent to: Component 1 - GeMunee Component 2 - Green Building Rating and Certification System Component 3 - Working Group and Advisory Assistance emissions • New or additional tools • Additional interventions to leverage public / private sector clean energy funds  Course corrections may be defined by any size, e.g. small ideas for, or large course corrections to achieve programmatic results. governments, GOG ministries, project staff, partner NGO staff, USAID/EG Office, training/TA/grant recipients, international financial institutions, other donors, other stakeholders, Green Building Council of Georgia, Survey of non-public stakeholders, other entities involved in green building rating and certification 5. METHODOLOGY The following approach further elaborates on how the Evaluation Team envisages tackling the entire evaluation process. 5.1 Quantitative Research and Analysis 5.1.1 General Approach Quantitative data (e.g. number of actions / activities to be accomplished against plan) will be sourced from EC-LEDS’s Annual Work Plans, PMPs and other project-related periodic reports. The collective outcome of this evaluation will be a thorough assessment of the performance of EC-LEDS for the period under scrutiny (in this case September 2013 to present) relative to the planned objectives as envisaged in its original contract and elaborated and/or revised upon in subsequent modifications / amendments. In this instance, as the project was tasked with accomplishing certain target indicators, verification of performance from a statistical point of view can essentially only focus on a review of EC-LEDS’s records which the Evaluation Team will assume accurately reflect whether a numeric indicator was achieved or not. The findings from document review will be cross-referenced with findings accumulated from our qualitative research approach to determine the extent to which evidence gathered contributes towards the Evaluation Team’s understanding of EC-LEDS’s impact on its intended beneficiaries. This, in turn, will enhance the Team’s prospect of being able to provide satisfactory and meaningful answers to USAID’s evaluation questions outlined in the Evaluation Design Matrix above. Where appropriate, the Evaluation Team will prepare relevant charts summarizing the outcome(s) of collective responses to questionnaires, aimed at shedding light on whether action in the field reflects the quantifiable data reported by the project and represents real progress or not in terms of EC-LEDS achieving its intended results, mainly on a year-on-year basis since inception. This exercise will function in tandem with the team’s qualitative approach (outlined below). While analysis of existing quantitative data on EC-LEDS’s activities will undoubtedly be invaluable to the Evaluation Team in terms of contributing to its understanding of the project’s performance to date, the Team proposes to add significantly to this analysis by conducting a mini-survey as described above. Local Georgian company IRMS will be tasked with carrying out the mini-survey from the design stage through the presentation of preliminary findings and completed analysis for inclusion in the Evaluation Team’s final report. It is important to note here that the Evaluation Team is aware that each of the project’s three components has specific indicator targets that should be met in accordance with EC-LEDS’s mandate and whose performance is reflected in Annual Reports and PMPs. Analysis of this data has two purposes: 1) to determine the extent to which EC-LEDS is achieving objectives against plan from a quantitative point of view; and 2) to provide the background against which interview questionnaires can be posed to best elicit responses aimed at answering USAID’s specific evaluation questions. Below is a brief synopsys of how the proposed mini-survey will be designed and implemented. Further elaboration can take place at the planned USAID in-briefing at the start of the evaluation mission if clarification is needed. 5.1.2 Survey and Sampling Methodology The mini-survey will have 10-15 questions and it will be in the form of a census, i.e. targeting 100% of the 50 non-public stakeholders and beneficiaries identified on the provided list of ‘Green Building Contacts’, and representing some 37 different organizations. Some questions will be close-ended with prepared response categories (e.g. yes/no/don’t know, on Likert-type 5-point scale) and some will be open-ended, qualitative. The interview is expected to last no more than 15-20 minutes. The data will be entered, cleaned, coded, checked for quality, and then analyzed using SPSS software. Data analysis will be primarily descriptive in nature, given the limits imposed by the small number of observations. The data will be assessed to determine if it is normally distributed, and if not, reasons for non-normality will be investigated. The output will be in the form of frequency tables and some cross￾tabulations. It is proposed to conduct three FGDs with the non-public stakeholders and beneficiaries identified above, following the telephone survey of the same group. The composition of the groups will be determined based on the results of the survey, e.g. based on type of engagement with the program, or private sector / non-private sector divisions. The FGDs will allow the Evaluation Team to probe themes that emerge from the telephone survey, and help with interpretation of the results. A signed Informed Consent will be solicited before starting any FGD, which will ensure participants of their anonymity. 5.2 Qualitative Research and Analysis The required approach under this activity assumes particular importance given the geographic spread of EC-LEDS’s beneficiaries and the time available to the Evaluation Team to conduct its enquiries. Here, the Team’s approach will be to identify, locate, and meet with as representative a body as possible with direct or indirect knowledge and/or experience of the project for the period September 2013 to present. To start, these would be municipal authorities in each of the proposed municipalities to be visited and which can be seen in the schedule in Annex 2. In effect, there is less concern here as to whether a particular action has been implemented or not (verifiable or not from REAP records) as opposed to whether the result of that action met the needs and aspirations of the intended recipients. In so doing, the Evaluation Team will pay particular attention to avoiding subjective opinion and hearsay as these effectively add little or no value to understanding the facts. On the other hand, perceptions are valid and will be included in the Team’s findings as they may enhance the quality of recommendations for the remainder of the project or any future planned interventions by USAID/Georgia. In order to effectively carry out the qualitative research needed, the following approach is proposed for different interviewee groups. This is not an exhaustive list - others may be identified at the start of the evaluation mission in Georgia: 1. EC-LEDS In addition to project-related documentation already received prior to the start of the evaluation, the Evaluation Team will review EC-LEDS’s reporting procedures to include data collection and analysis methods using information technology – software programs / excel sheets – where available for scrutiny. In particular, attention will be paid to Work Plans, PMPs and Annual and Quarterly Reports. Data Gathering Approach: Meetings with REAP management and staff by the Evaluation Team, initially and throughout the period of the in-country evaluation mission. 2. Municipal Authorities Structured interviews will take place with representatives to determine attitudes and satisfaction levels. Specifically, they will be asked questions on how the program has benefitted them, how well does the assistance tie in with the support and commitments made with respect to other agreements (such as the EU-initiated Covenant of Mayors (COM)) and, the constraints and challenges that may be inhibiting the objectives of the EC-LEDS’ program from being realized. Also, they will be probed on what might be missing in terms of additional support to realize these objectives. Data Gathering Approach: Direct interviews with key personnel. 3. Financial Institutions: Due to their (potential) involvement in EC-LEDS, feedback from these bodies may provide the Evaluation Team with information on their knowledge of EC-LEDS, how the project may have helped them so far in their lending activities (if at all) and, their inclination to provide credit under the terms of the USAID loan guarantee. Data Gathering Approach: Direct interviews with key personnel familiar with EC-LEDS (or a similar project using the same credit facility) from relevant institutions to be identifed in collaboration with the EC￾LEDS COP and program team members as appropriate. 4. International Community: World Bank, EU, other development agencies: The effect and impact of EC-LEDS’s activities will be known to various members of the international donor community such as those mentioned above. Therefore, their views will enrich the Evaluation Team’s understanding of how EC-LEDS is currently perceived. Data Gathering Approach: Direct interviews using structured questionnaires with key personnel familiar with EC-LEDS. 5. Official Bodies: Government Ministries, Local Authorities: Those associated with the project will have a vested interested in its activities and its progress / impact on intended beneficiaries, especially those who are direct beneficiaries under Component 3 of the services. Data Gathering Approach: Direct interviews using structured questionnaires with key individuals involved with or familiar with EC-LEDS. 5.3 Limitations As pointed out in the technical proposal, there are several limitations inherent to the design of this evaluation. The most serious anticipated limitations are the following although others may become apparent at the start of the evaluation mission following discussions among team members: 1. Selection Bias: As some key informants may decline to be interviewed, there is a possibility of selection bias, i.e. those respondents who choose to be interviewed might differ from those who do not in terms of their attitudes and perceptions, affiliation with government/non￾government structures, and socio-demographic characteristics and experience. 2. Recall Bias: Since a number of questions raised during the interviews will deal with issues that took place in the past, recall bias cannot be excluded. As EC-LEDS project activities were launched in September 2013, some respondents may find it difficult to accurately compare situations before and after the project. 3. Halo Bias: There is a known tendency among respondents to under-report socially undesirable answers and alter their responses to approximate what they perceive as the social norm (halo bias). The extent to which respondents will be prepared to reveal their true opinions may also vary for some questions that call upon the respondents to assess the performance of their colleagues or people on whom they depend upon for the provision of services. To mitigate this limitation, ME&A will provide the respondents with confidentiality and anonymity guarantees, where possible; conduct the interviews in the settings where respondents feel comfortable; and establish rapport between the interviewer and the respondent. FGDs will be conducted among peer groups to encourage the expression and development of ideas that may not be accepted outside of subgroups. 6. PROPOSED SITE VISITS AND MEETINGS The Evaluation Team has already requested a list of stakeholder/ interviewees from the COP of EC-LEDS. These will be provided on meeting with the EC-LEDS team on October 21. The EC￾LEDS COP has also offered to provide assistance in planning trips outside Tblisi. ANNEX 3: LIST OF ORGANIZATIONS AND PEOPLE INTERVIEWED Meetings: Number of people interviewed Names and positions Winrock 3 Inga Pkhaladze, COP Giorgi Giorgobiani, Deputy COP Irina Sulava, PR person Tbilisi Municipality 2 Giorgi Chachanidze Head of Economic Policy Khatia Arabidze, Officer - Dept of Economics Ministry of Economy 1 David Gigineishvili, Head of Spatial Planning and Construction Policy Department Ministry of Energy - staff 3 Avtandil Todua, Analytical Department Marita Arabidze, Head if Energy Efficiency and Renewable Energy Division Natalia Jamburia, Chief Specialist, Division for Energy Efficiency and Alternative Energy Resources Ilia State University 1 Ia Kupatadze, Graduate Student in Architecture GBCG 1 Levan Natadze, CEO, Founding Member Remissia 1 Marina Shvangiradze, Director kfW 1 Nino Shanidze, Senior Project Coordinator EBRD 2 Sophiko Chikhraze, Senior Analyst Tea Melikadze, Associate Banker, Municipal and Environmental Infrastructure Team Ministry of Energy - Dpty Minister 1 Mariam Valishvili, Deputy Minister Gori Municipality 1 Givi Khuroshvili, Contractor under COM/EC LEDS Batumi Municipality 1 Tite Aroshidze, Department Head of Economy/Finance Department Zugdidi Municipality 3 Merab Kvaraia, Chairman of City Council Gia Gasashvili, Econ. Policy and Invest. Strategy Dept Giorgi Todua, Dept Chair of City Council Kutaisi Municipality 3 Paata Kldiashvili,Head of the Economic Development and Local Self￾Government Property Management Department Bachuki Gogonadze, Chief Specialist, Econ. Dept. Irakli Koglechidze, Head of Econ Dev. Section Akhaltsikhse Municipality 2 Giorgi Kopadze, Mayor Naira Samsonidze, Head of Municipal Supervisory Service Consultant 1 Dana Kenney, former Winrock COP Climate Change Office 2 Grigol Lazrievi, Head of Climate Change Service Kakha Karchkhadze, Winrock LEDS Advisor Telavi Municipality (region) 1 Zurab Enukidze, Main Specialist - Municipal Economic Service Telavi City Hall (urban) 1 Giorgi Akhvlediani, Senior Specialist Municipal Service for relations with International Foundations and NGOs Turkish GBC 1 Duygu Erten, Winrock GB consultant Former Winrock employee 1 Mariam Bakhtadze, Water Sector Environmentalist, GWG Project Rustavi 2 Zurab Tabaghua, Deputy Head of infrastructure Dept Theona Galogre, Head of Economic Development Division GIZ 1 Irakli Samkharadze, National Key Advisor, Global INDC Support Programme USAID 2 Nicholas Okreshidze, Agreement Operating Representative Veronica Lee, Environmental Officer Total people 38 ANNEX 4: LIST OF DOCUMENTS REVIEWED BIBLIOGRAPHY OF DOCUMENTS AND SOURCES REVIEWED Amex International. (2012). Mid-Term Performance Evaluation of the New Applied Technology Efficiency and Lighting Initiative (NATELI). Covenant of Mayors website: http://www.covenantofmayors.eu/+-JRC-+.html Ministry of Energy and Georgia’s Low Emission Development Strategy. (nd) Presentation USAID (2013). Project Appraisal Document EC-LEDS Clean Energy Program. USAID. (2012). Non-U.S. Organization Pre-Award Survey Guidelines and Support. USAID. (2013). Cooperative Agreement Clean Energy AID-114-A-13-00008. USAID/U.S. Department of State. (2014). U.S. Government Supports Low Emission Economic Growth. Fact Sheet. Winrock International (2014). Assessment of Available Financing Sources for Mitigation Projects. Winrock International (2014). EC-LEDS Baseline Survey Report. Winrock International (2014). EC-LEDS Work Plan Year 2. Winrock International (2014). EC-LEDS Work Plans Year 1. Winrock International (2014). National Communication Strategy. Winrock International (2015). MRV Framework and Methodology. Winrock International. (2014). Georgia Green Building Assessment Report. Winrock International. (2014). Green Building Training Plan. Winrock International. (2014). Markal Georgia LEDS Reference-Business as Usual Scenario Report. Winrock International. (2014). Memo on Capacity Building Needs of MOE to Manage the LEDS ProcessWinrock International. (2014). Report On the Second Meeting of the Green Building Rating and Certification Working Group. Winrock International. (2014). Memo Regarding GB Principles. Winrock International. (2014). Memo Regarding GB Accreditation. Winrock International. (2014). Report on Analysis of Green Building Rating and Certification Systems. Winrock International. (2015). Competition Evaluation Report for Green Building of the Year Award, Including MRV Plan. Winrock International. (2015). Cross Cutting Action Plan. Winrock International. (2015). GB Rating and Certification Training Report. Winrock International. (2015). Green Building Marketing Action Plan. Winrock International. (2015). Inventory of Greenhouse Gas Emissions, BAU Scenario Development and Identification of Mitigation Measures in Waste and Greening Sectors. Winrock International. (2015). MARKAL-Georgia EC-LEDS Reference Scenario Report. Winrock International. (2015). MEMO summarizing targeted marketing for GB certifications, and commitments for certifications obtained from target audience. Submitted by: Alliance to Save Energy Winrock International. (2015). Organizational Development Assessment of Five Georgian Sustainable Development NGOS Winrock International. (2015). Report on Adopting Display or Energy Passport Tool, and Developing Guidelines. Winrock International. (2015). Report on Comparison of EPBD and IECC Requirements for Building Energy Performance. Winrock International. (2015). Report on the Energy Performance Methodology for Georgia Winrock International. (2015). Review of International Experience in Building Energy Performance Certification and Labeling. Winrock International. (2015). Sustainable Energy Action Plans - for Batumi, Gori, Kutaisi, Zugdidi. Winrock International. (2015). Training Report on Building Energy Labeling by Display. Winrock International. (draft). Green Building Certification Marketing Action Plan. Winrock International. Annual and Quarterly Progress Reports. Winrock International. Media coverage reports. ANNEX 5: TELEPHONE QUESTIONS TO GB BENEFICIARIES AND STAKEHOLDERS Survey statistics: Population: 50, representing 36 organizations Survey response rate: 27 out of 50 = 54% Organization response rate: the 27 respondents came from 21 different organizations Survey responses Q1 To your knowledge, is there a Green Building Council already established in Georgia? Responses Frequency Percent Yes 25 92.6 No 2 7.4 Total 27 100.0 Q2 When do you think the GBCG was established? Responses Frequency Percent In 2014 10 37.0 Not yet formed 1 3.7 Don't Know 16 59.3 Total 27 100.0 Q3 Do you think there is a demand for a combined rating covering all aspects (planning, site preparation, energy use, water use, materials, demolition etc.) of the Green Building concept? Responses Frequency Percent Yes 11 40.7 No 16 59.3 Total 27 100.0 Q4 Do you think that an energy efficiency rating is more relevant than a wider green building rating for Georgia? Responses Frequency Percent Yes 15 55.6 No 12 44.4 Total 27 100.0 Q5 Many countries have their own Green Building Council, which are membership organizations. Would Georgia’s GBC be viable as a membership organization? Responses Frequency Percent Yes 16 59.3 Don’t know 10 37.0 Depends on the fee 1 3.7 Total 27 100.0 Q6 If the annual fee were $50-$100 for individuals and $200-$1,000 for corporates, do you think your organization would join? Responses Frequency Percent Yes 22 81.5 No 3 11.1 Don’t know 2 7.4 Total 27 100.0 Q8 Do you think Georgia is ready for such a national GBC organization at present? Responses Frequency Percent Yes 23 85.2 No 3 11.1 Don’t know 1 3.7 Total 27 100.0 Q9 Why/why not? (Multiple answers possible) Responses N Percent of cases 1. Regulation 8 29.6 2. Active promotion, public awareness campaign 18 66.7 3. Education and training required, developing materials 5 18.5 4. Depends on country economy and stability 1 3.7 5. EU directives 2 7.4 6. GOGs involvement 1 3.7 7. General interest, low demand 5 18.5 8. Requires investment; PPP component 3 11.1 9. Demo Projects 1 3.7 Total 44 163.0 Q10 Would you support the use in Georgia of a well-established existing and internationally recognized Green Building certification scheme such as LEED and BREEAM? Q11 Do you think that a local (Georgian) rating and certification scheme for green buildings is needed instead of, or as well as, LEED, BREEAM etc? Frequency Percent Yes 25 92.6 No 1 3.7 Don’t know 1 3.7 Total 27 100.0 Frequency Percent Yes 26 96.3 Don’t know 1 3.7 Total 27 100.0 ANNEX 6: WINROCK STATEMENT OF DIFFERENCES