0 September 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Arvid Kruze, Bonnie Walter, and Fuad Strik through Social Impact, Inc. PERFORMANCE EVALUATION OF THE REGULATORY AND ENERGY ASSISTANCE PROGRAM (REAP) FINAL REPORT USAID/BOSNIA AND HERZEGOVINA PERFORMANCE EVALUATION OF THE REGULATORY AND ENERGY ASSISTANCE PROGRAM (REAP) FINAL REPORT USAID/BOSNIA AND HERZEGOVINA September 2013 Task Order # AID-168-M-13-00001 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii CONTENTS Contents ............................................................................................................................................................iii Acronyms ..........................................................................................................................................................iv Executive Summary ..........................................................................................................................................v Evaluation Purpose and Evaluation Questions ...........................................................................................1 Project Background ..........................................................................................................................................2 Evaluation Methods and Limitations .............................................................................................................8 Findings, Conclusions, and Recommendations........................................................................................ 11 Annex I: Evaluation Scope of Work .......................................................................................................... 34 Annex 2: Interviewee Organizations and People Met ........................................................................... 39 Annex 3: Documents Reviewed ................................................................................................................. 41 Annex 4: Evaluation Methodology and Limitations ................................................................................ 49 Annex 5: REAP Performance Based Management Plan (PBMP) .......................................................... 53 iv ACRONYMS AEAI BiH CDCS COR Advanced Engineering Associates International Bosnia and Herzegovina Country Development Cooperation Strategy Contracting Officer’s Representative DPA Dayton Peace Accord EBRD European Bank for Reconstruction and Development EP Elektroprivreda EU European Union FBiH Federation of Bosnia and Herzegovina FERK Regulatory Commission for Electricity in Federation of Bosnia and Herzegovina GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit GWh HZHB IPA Gigawatt-hour Hrvatske Zajednice Herceg-Bosne Instrument for Pre-Accession Assistance IR Intermediate Result kV Kilovolt ISO Independent System Operator OHR Office of High Representative MOFTER Ministry of Foreign Trade and Economic Relations MOU Memorandum of Understanding MW Megawatt NARUC National Association of Regulatory Utility Commissioners NGO PBMP Non-Government Organization Performance Based Management Plan REAP Regulatory and Energy Assistance Program RS Republika Srpska RSERC Regulatory Commission for Electricity in Republic of Srpska SEE South Eastern Europe SERC State Electricity Regulatory Commission SOW Scope of Work Transco Transmission Company USG United States Government USAID U.S. Agency for International Development v EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The main purpose of this final performance evaluation is to provide USAID/Bosnia-Herzegovina (BiH) with an objective analysis of USAID assistance to BiH’s energy sector under the $7.4 million Regulatory and Energy Assistance Program (REAP) project. The REAP project, undertaken over the period 2007–2013, is one of the largest projects in the USAID/BiH Economic Growth Portfolio and, as such, must undergo a performance evaluation according to USAID Evaluation Policy. The findings will be used to make informed decisions on the future directions of USAID/BiH’s assistance in the energy sector. Key evaluation questions focus on whether there is evidence that the project contributed to the enhancement of practices and understanding in the BiH energy sector of the USAID intervention, which was aimed at assisting relevant BiH institutions in meeting the provisions of the South Eastern Europe (SEE) Energy Community Treaty (the Treaty). PROJECT BACKGROUND The REAP project is part of USAID’s committed assistance, begun in 2000, to the BiH energy sector. As part of the USAID mission’s strategy to harmonize the sector with European Union (EU) energy directives and help establish a regional energy market, USAID has been supporting BiH’s efforts to meet the provisions of the Treaty signed in Athens in 2005 and further bolstered by subsequent EU Directives, including the Second and Third Packages. EVALUATION TOOLS, DESIGN, METHODS, AND LIMITATIONS The Evaluation Design and methodology included a review of pertinent documents and interviews with beneficiaries of USAID assistance and stakeholders within the BiH energy sector who had direct knowledge of and experience with the project. Based on the document review, the team identified four target groups of interest: (i) REAP project beneficiary organizations at both State and Entity levels including Ministries, regulators, and operating companies; (ii) activity implementers; (iii) international donors (including USAID); and (iv) other stakeholders, including BiH NGOs and BiH media. For each stakeholder group, specific questions were designed that focused on its interest in the REAP project and that contributed to answering the five evaluation questions. Data collection was aided by the use of an Excel workbook to organize and store data culled from interviews. Data were organized so as to permit the team to identify patterns in the responses and elicit the common themes and findings that could be supported by evidence in the data. To ensure that the data were internally valid and reliable, interview data were triangulated between respondents and between sources, with areas of agreement or disagreement identified. Ten themes emerged from the interview data, each of which has been given a theme number and description (see Annex 4). vi ANSWERS TO EVALUATION QUESTIONS The first evaluation question is To what extent has REAP contributed in a measureable way to BiH’s integration into the regional and EU electricity market and, moreover, which of REAP’s interventions in the electricity sector have helped in BiH’s integration into the regional and EU electricity market? Answer: It was found that REAP contributed to significant progress on BiH’s integration into the EU electricity market. Significant milestones include (i) implementing EU energy market directives and regulations on cross-border network access (achieved); (ii) liberalization of the market for all non-household customers (almost fully achieved); and (iii) liberalization of the market for all customers (to be achieved by January 2015). The real measure of the success of BiH’s integration into the regional and EU electricity markets, and REAP’s contribution, will be the point at which full retail competition takes place, scheduled for January 2015. This point has not yet been reached. Nevertheless, most of the enabling legislation, the regulatory processes required, the regulations themselves, and knowledge of precise requirements by BiH stakeholders appear to have been addressed. The main text of this report presents a breakdown of measurable progress in meeting integration and Treaty requirements and those that are as yet incomplete. Ten specific interventions by REAP, out of many1 , were identified as helping in BiH’s integration into the regional and EU electricity market. The second evaluation question is What have been the main obstacles in the implementation of REAP? Answer: Obstacles have been political in nature (or politically caused) and have included (i) the politicization of energy in BiH and the use of energy laws and energy institutions as a political tactic by certain factions in the RS as part of the ongoing fight over the nature of the State and the three-way structure of its institutions; (ii) a general lack of interest or sense of urgency at the state and entity levels concerning energy as an issue, due to a lack of understanding of the long-term potential benefits that could accrue to the country; and (iii) the absence of progress towards a national and integrated strategy for energy. The third evaluation question is whether there are any important elements of the Treaty that REAP has not addressed, in particular those pertaining to improved economic aspects of governance relevant to business activity. After discussion with USAID, the phrase “pertaining to improved economic aspects of governance relevant to business activity” was interpreted to mean “to assist commercial development.” 1 Over its six years in providing continuous assistance to BiH power sector stakeholders, it can be said that REAP undertook hundreds of interventions in conducting its day-to-day activities. Some of these interventions are shown in Annex 3 under “REAP Major Deliverables”. The ten major interventions were identified by the implementer and subsequently verified by the evaluators. vii Answer: REAP has addressed all the technical considerations for regulation relevant to the opening of the market and market integration. However, despite the assistance activities already provided on commercial development in line with USAID strategy, more work with the four electricity companies may have helped them to adapt to the future situation, according to various respondents, including the companies themselves. The fourth evaluation question is What is the relevant stakeholders’ estimate of the effects associated with USAID/BiH disengagement from BiH energy policy assistance after REAP ends in September 2013? Answer: According to stakeholders interviewed, REAP will be missed and progress will slow down once the project ends. However, most of the beneficiary organizations express confidence in their ability to use the information provided by REAP and apply it to resolving issues related to market opening. The fifth evaluation question is How well is REAP’s implementation coordinated/synchronized with other donors’ activities that are aimed at integrating BiH into the regional and EU electricity market? Answer: In the area of assistance provided by REAP, no issues exist in coordinating efforts because no other donors, except the EU, are addressing electricity market integration. The efforts undertaken by the EU and USAID are not conflicting and appear well coordinated in terms of the type of assistance provided. RECOMMENDATIONS Recommendations for possible further assistance on BiH’s electricity market integration should be based on a number of considerations, including (i) whether six years of technical assistance is enough and whether the beneficiaries should be left alone from the time REAP ends in September 2013 to the intended market opening date of January 1, 2015; (ii) the political situation in BiH, which has been the primary cause of project delays and obstructions to market integration; (iii) USG policy on a unified BiH and implications for its electricity supply industry; and (iv) timing of possible assistance relative to Treaty timetables on market integration. Upon a detailed analysis of the situation, it is recommended that:  Some continued assistance beyond REAP is desirable, but it should be limited in scope and focused on facilitating, mediating, and providing neutral technical expertise where disagreements might arise between the various stakeholders, leading up to 2015 and carried over into assistance implementation on market opening. Continuing with the theme of restructuring and commercializing the electric power companies, as contained in the original SOW for REAP, should at least be explored, as, according to stakeholders, there is an ongoing need for the companies to better understand how to function in the future environment.  A fuller assistance package, which might include support in the development of a national electricity development strategy, should be considered only in light of subsequent political developments in BiH, including the internal political will to deal viii with the outstanding policy issues and the political will of the EU, since, according to informed respondents, they hold the power to break the deadlock of the current political struggles by withholding funds and applying sanctions. 1 EVALUATION PURPOSE AND EVALUATION QUESTIONS EVALUATION PURPOSE The main purpose of this end-of-project performance evaluation is to provide USAID/Bosnia￾Herzegovina (BiH) with an objective analysis of USAID assistance to BiH’s energy sector under the $7.4 million Regulatory and Energy Assistance Program (REAP) project. The REAP project, undertaken over the period 2007–2013, is one of the largest projects in the USAID/BiH Economic Growth Portfolio and, as such, must undergo a performance evaluation according to USAID Evaluation Policy. The primary audience for this evaluation is the USAID Mission in BiH and USAID/Washington. Secondary users are local government officials with whom USAID collaborates and other donors. The purpose of this evaluation report is to provide information and lessons learned to inform USAID/BiH’s strategic decisions for energy-sector programming—whether to remain engaged in the sector and, if so, in what ways. The evaluation report will provide information to USAID/BiH so that it can assess the extent to which the Mission’s involvement in the energy sector contributes to the Mission’s Country Development and Development Strategy goal of helping BiH become more stable and closer to Euro-Atlantic integration. EVALUATION QUESTIONS The evaluation questions to be addressed in this report include the following: 1. To what extent has REAP contributed in a measureable way to BiH’s integration into the regional and European Union (EU) electricity market? Which of REAP’s interventions in the electricity sector have helped in BiH’s integration into the regional and EU electricity market? 2. What have been the main obstacles in the implementation of REAP, if any? 3. Are there any important elements of the South East Europe (SEE) Energy Community Treaty (“the Treaty”) that REAP has not addressed, in particular those pertaining to improved economic aspects of governance relevant to business activity? 4. What is the relevant stakeholders’ estimate of the effects associated with USAID/BiH disengagement from BiH energy policy assistance after REAP ends in September 2013? 5. How well is REAP’s implementation coordinated/synchronized with other donors’ activities that are aimed at integrating BiH into the regional and EU electricity market? The evaluation questions are broadly worded and aim at identifying the extent to which the activity has led to compliance with the Energy Treaty requirements and promoted integration into a regional energy market. However, the implementer’s Scope of Work and the evaluation Scope of Work are far more narrowly defined (as the Scope of Work was reduced over time and the evaluation Scope of Work was limited). Consequently, in providing complete answers 2 to the evaluation questions the evaluators found that, in order to fully answer the evaluation questions, it was necessary to consider a broader context than that defined in the relevant Scopes of Work. A complete Statement of Work for the evaluation is provided in Annex 1. PROJECT BACKGROUND INTRODUCTION The REAP project is part of a long history of practically continuous USAID assistance to the BiH energy sector since the year 2000. As part of the USAID mission’s strategy to help BiH harmonize the sector with EU energy directives and help establish a regional energy market, USAID has been supporting BiH’s efforts to meet the provisions of the Treaty signed in Athens in 2005 and further bolstered by subsequent EU Directives, including the Second and Third Packages. The evaluation team—Arvid Kruze (Senior Energy Expert and Team Leader), Bonnie Walter (Senior Evaluation Specialist), and Fuad Strik (Local Energy Expert)—spent three weeks in BiH between June 1 and June 21, 2013, interviewing key beneficiaries of the USAID assistance, reviewing available documentation, and discussing/analyzing findings. A preliminary presentation of findings was made to USAID Mission staff in Sarajevo on June 21. POWER SYSTEM OVERVIEW The BiH electricity system comprises three, largely government-owned, 2 electricity generation and distribution companies; a transmission company (Transco); and an Independent System Operator (ISO). The three ethnic-based companies that produce and distribute electricity are as follows:  Elektroprivreda Republika Srpska (EP RS), based in the southern RS city of Trebinje  Elektroprivreda Bosnia and Herzegovina (EP BiH), based in Sarajevo  Elektroprivreda Hrvatske Zajednice Herceg-Bosne (EP HZHB), based in the southern federation city of Mostar Each Elektroprivreda (EP) has its own territory and is responsible for the generation of electricity and the distribution and supply to customers. EP HZHB has 792 megawatts (MW) installed capacity from six hydropower plants. During the last three years (2010–2012), average production was 1,742 gigawatt-hours (GWh), with total 2 A small percentage of the issued capital is owned by public funds and small shareholders. 3 customer demand from its system of 3,505 GWh. Consequently, EP HZHB needs to purchase electricity from other sources in the country and region to meet demand. EP BiH has 494 MW of hydropower generation and 1,165 MW of coal-fired power generation. With an average production of 7,070 GWh in the last three years and total consumption averaging 4,697 GWh, EPBiH has excess capacity and produces electricity for export. EP RS, with an installed capacity of 1,512 MW, has a balanced generation mix of approximately 50% each for three hydropower and two coal-fired power plants. With a total average yearly production of 5,529 GWh in the last three years and average yearly consumption of 3,674 GWh, it generates electricity for export as well. The three utilities’ combined generating capacity exceeds the BiH demand for electricity. Therefore, BiH is a net electricity exporter. With regional demand forecasted to grow over the medium and long term, how BiH performs in the years ahead is important for its neighbors and the establishment of the regional electricity market. A BiH power system map is shown in Figure 1. 4 Figure 1: Map of Bosnia-Herzegovina’s power system 5 ORGANIZATION OF THE ELECTRICITY SUPPLY INDUSTRY Power sector reform, which began in the early 2000s, has resulted in an organization of the electricity supply industry composed of the following main players (in addition to the previously mentioned EPs):  BiH’s national electricity transmission company, Transco: responsible for owning, operating, and maintaining the transmission system across the whole of BiH. It is divided into four transmission zones that transcend political boundaries. Transco started operation in February 2006 and is based in Banja Luka, Republika Srpska (RS).  Independent System Operator (ISO): responsible for the management and control of the transmission network; directing, scheduling and coordinating power supply; planning and development of the grid; and development of the indicative generation plan with Transco. The ISO started operation in February 2006 and is based in Sarajevo in the Federation of Bosnia-Herzegovina (FBiH, not to be confused with the country or State of BiH).  State Electricity Regulatory Commission (SERC): responsible for regulating transmission and related activities, including international trade. It was established in 2004 and is based in Tuzla, FBiH.  Federation Electricity Regulatory Commission (FERK): responsible for regulating generation, distribution, and supply within the FBiH. It has offices in Mostar, FBiH.  Republika Srpska Electricity Regulatory Commission (RSERC): responsible for regulating generation, distribution, and supply within the RS. It has offices in Trebinje, RS. At the governmental level, the Ministry of Foreign Trade and Economic Relations (MOFTER) is responsible for tasks and duties falling within the jurisdiction of the State of BiH, including defining policies and basic principles, coordinating activities of the sector with the Entities, and consolidating entity plans with those of international institutions in energy. The governments of BiH and RS have their own Ministries of Energy and have offices in Mostar, BiH, and Banja Luka, RS, respectively. Jurisdiction over energy is determined by the Constitution. Since energy was not specifically assigned to the State under its constitution, it fell to the Entities. However, the two Entity Prime Ministers delegated authority to the State for transmission, the ISO and the SERC, prior to passage of the “Act on Transmission of Electric Power, Regulator and System Operator of Bosnia and Herzegovina” in 2002. Generation, distribution, and supply are fully under the jurisdiction of the Entities. In fact, the Entities have more extensive jurisdiction over the energy sector. Thus, all three Ministries, Governments, and Parliaments—State and Entity—are responsible for energy; however, due to the extent of activities in the Entities, their responsibility over energy matters is actually greater. 6 POLITICAL ENVIRONMENT The present State of BiH was founded by the Dayton Peace Accord (DPA) in 1995, which is the peace agreement that put an end to the three-and-a-half-year Bosnian war, fought along ethnic lines between Serbs, Bosniaks, and Croats. The United States Government (USG) was heavily involved in both the initiation and the negotiation of the DPA. The DPA divided the BiH into two entities: the Republika Srpska (RS) and the Federation of Bosnia and Herzegovina (FBiH). The FBiH covers 51% of BiH’s total area, while the RS covers 49%. The Entities, based largely on the territories held by the two warring sides at the time, were formally established because of the tremendous changes in BiH’s ethnic structure. The FBiH and Brcko are ethnically diverse Entities, comprising a mix of Bosniak, Croat, and Serb populations, while the RS is over 90% Serb. The DPA’s main purpose is to promote peace and stability in BiH and to establish the basis for a new unified country. Today, political elites in BiH’s two entities lack a shared vision for the country and a consensus on the political system established under the DPA. RS nationalist rhetoric and resistance to State-level institutions have become more strident in recent years, and RS Prime Minister Milorad Dodik has led a sustained campaign calling for a secession referendum for the RS. All electricity industry stakeholders in BiH are either government-owned or under the purview of the government. This permits the introduction of political interests that may influence an organization to concentrate on political agendas rather than focus on creating an efficiently functioning energy system. THE REAP PROJECT In late 2007, USAID recruited Advanced Engineering Associates International (AEAI) as the contractor to carry out the REAP project. AEAI’s Statement of Work (SOW) for the project was composed of the following major tasks: Task 1: A fully integrated energy sector into the regional market and the EU Under this task, the contractor was to provide services to BiH through the following activities: A. Unification of the Regulators B. Creation of the Legal and Regulatory Framework for Gas C. Transformation of the ISO into a Full Market Operator for all of BiH D. Broader Public Understanding of Specific Energy-Sector Issues and Challenges Expected results were: ˙ Single BiH energy market established ˙ State Regulatory Commission issues licenses and tariffs for all energy sectors ˙ Rules and procedures governing grid codes and market rules are compatible with SEE energy market ˙ BiH energy sector fully integrated into regional/European framework 7 Task 2: Restructuring and commercialization of energy companies Under this task, the contractor was to undertake the following: A. Implementation of the Action Plans B. Broader Public Understanding of Specific Energy-Sector Issues and Challenges Expected results were: ˙ All unbundled (daughter) companies of the three power companies are established as legal corporations and operating under headquarters of Holding Company ˙ All joint stock companies have separate accounting systems operating and full cost￾based tariffs are developed ˙ New unbundled companies are functionally rationalized ˙ Finalized plans for restructuring and commercialization of BiH electricity generation and distribution Project Extension REAP was originally intended to end in 2011. However, in that year, the project was extended by 30 months because the work had not yet finished. The need for the extension arose out of the delays caused by a series of political events, all of which affected the planned dates for achieving objectives. Tasks 1 and 2 above were unchanged under the extension. Nevertheless, expected results had changed somewhat and reflected the events and progress of the previous four years. Revised expected results under Task 1 were: ˙ Applicable electric power sector rules harmonized between regulators ˙ Commercial contracts for operation of a competitive market prepared and in use for eligible customers ˙ Additional preconditions for successful market functioning met with concomitant removal of impediments for market functioning ˙ Legal framework for the liberalization of the gas sector in place, assuming the political preconditions are also met ˙ Road Maps for implementation of Energy Community Ministerial Council Decisions that implement new EU Directives prepared ˙ Regulatory partnership program executed with a U.S. regulatory commission partner Revised expected results under Task 2 were: ˙ Restructuring plan as part of the developed FBiH Energy Policy drafted as prescribed in the new Electricity Law ˙ Cost-based tariff methodology matches practice and regulated companies receive revenue that more fully reflect their costs ˙ REAP’s recommendations regarding restructuring of EPs 8 In a final modification to the REAP contract in 2013, the regulatory partnership program was dropped due to budget cuts. This evaluation covers only Task 1, with the gas sector component removed. EVALUATION METHODS AND LIMITATIONS The Evaluation Design and methodology included a review of pertinent documents and interviews with beneficiaries of the USAID assistance and stakeholders within the BiH energy sector who had direct knowledge of and experience with the project. DATA COLLECTION Evaluation data was collected using two methods: (i) document review and (ii) key informant semi-structured interviews. Over 50 individuals were interviewed (all but two in person), including 12 beneficiary organizations in five different locations in Bosnia—Sarajevo, Tuzla, Banja Luka, Trebinje, and Mostar. A complete list of interviewee organizations and people is provided in Annex 2. Document Review The evaluation team conducted a desk review of relevant REAP project documents and key BiH energy-sector background material and websites in order to (i) gain a thorough understanding of the relationship between REAP and the broader context of energy policy in BiH and the EU together with the Treaty requirements; (ii) understand the contractual Terms of Reference for the REAP project, the Activity Timetables, and Work Plans; and (iii) understand project outputs and indicators. A full list of documents reviewed is provided in Annex 3. The documents were essential for answering Evaluation Questions 1(a) and 3, verifying key informant interview responses against project activities and inputs, and defining the specific achievements in advancing regional integration and meeting requirements of the Treaty. Key Informant Interviews Key informant interviews served as the main data source for this evaluation. The evaluation team interviewed over 50 key informants identified by the USAID/BiH Mission in addition to other key informants identified through the data-collection process. Based on the document review, the team identified four target groups of interest that have either participated in the project or are intimately familiar with it. These stakeholder groups are as follows: (i) REAP project beneficiary organizations at both State and Entity levels, including Ministries, regulators, and operating companies; (ii) activity implementers; (iii) international donors (including USAID); and (iv) other stakeholders, including BiH NGOs and BiH media. 9 Stakeholder Groups For each such stakeholder group, specific questions were designed that focused on its interest in the REAP project and that contributed to answering the five primary evaluation questions. Individuals and organizations were suggested by either Mission staff or the USAID contractor (or both), based on perceptions of who would have the best knowledge of the assistance provided. The sampling method used for this performance evaluation was purposive rather than representative and probabilistic; interviewees who constitute the sample for this evaluation were persons with detailed knowledge sufficient to address the research questions. In all but three organizations, more than one organizational representative was involved in the interview, allowing for diverse opinions and multiple stakeholders’ input. The potential limitations and how these limitations were dealt with are discussed briefly in the Methodological Strengths and Limitations section, and in detail in Annex 4: Evaluation Methodology and Limitations. Data collection was aided by the use of an Excel workbook to organize and store data culled from interviews. Three team members took notes at each interview (during the first several interviews only two team members were in the field), and one of the team members reviewed and synthesized interview notes and entered them into the data-collection spreadsheet. The notes were then circulated to all team members for verification and amplification. A separate sheet in the Excel workbook was maintained for each of the stakeholder groups identified above. Each of these sheets contains all potential interview questions to be asked to that stakeholder group in a single column, with corresponding answers by stakeholders in subsequent columns. This method of organizing and assembling interview information permitted the team to identify patterns in the responses and elicit the common themes and findings that could be supported by evidence in the data. To increase the internal validity and reliability of the data, interview data were triangulated between respondents and between sources, with areas of agreement or disagreement identified. DATA ANALYSIS Data collected and transcribed into the Excel workbook was analyzed using a method of grouping responses to each of the five major questions into themes. This analytical process involves review of the data in the Excel workbook to identify emergent themes at the data￾analysis stage. Ten themes emerged from the data, and have been numbered from 1 to 10 with a theme description. These themes are presented below and are presented in greater detail in Annex 4. It should be noted that scaling of responses was used in the case of only one question, as the types of broad and open-ended evaluation questions did not readily lend themselves to scaling. In a number of cases, the prevalence of the themes in the data has been quantified across stakeholder groups or within a stakeholder group. The frequency of the themes in the data has allowed the formulation of findings and conclusions for each of the evaluation questions. 10 The ten major themes are: 1. Mixed perceptions by stakeholders of the progress on meeting Treaty obligations. 2. Technical assistance as a major support to writing legislation, by-laws, and secondary legislation, and preparing for market regulations. 3. The importance of the implementers as facilitators (bringing people together from all institutions to find common approaches in a nonjudgmental and impartial atmosphere). 4. Increased understanding of market functioning, the roles of regulators, and defining specific regulations to make markets work. 5. Major political and structural obstacles (that are due to the ethnic/political conflicts over the role of the RS and Federation institutions and the role of state institutions) qualified as “politics.” 6. Low level of public understanding of, and interest in, energy issues, as well as fears about future price raises. 7. The effects of market opening on energy prices. 8. Lack of progress on unified national and entity energy-investment strategies and action planning as a brake on investment in energy generation and the effects on regional integration. 9. January 2015 deadline and the ability to meet remaining obligations by that time. 10. Bottom-up and practical approach of REAP assistance compared to the EU’s static and bureaucratic approach. METHODOLOGICAL LIMITATIONS The evaluation team was limited methodologically in several ways. First, recall bias is a common problem in performance evaluations that often require respondents to report on activities and outputs that have occurred over the life of a program in one single instance (ex-post evaluation). This bias is exacerbated in this evaluation in particular given the length of implementation—six years—which made it difficult to obtain specificity in responses. Another limitation to this evaluation is selection bias. Interviewees were selected non-randomly based on availability and engagement with program activities. Undoubtedly, the biggest source of bias in a performance evaluation is response bias, which stems from respondents’ misreporting for any number of reasons, generally protecting a relationship with an individual or organization (e.g., USAID), telling the evaluators what they want to hear, or competing interests of some kind, be they political or institutional. In addition to these common sources of bias, the evaluation was also limited methodologically by the political and ethnic sensitivities in BiH. The team has taken a number of precautionary measures to address these limitations, which are presented in Annex 4: Evaluation Methodology and Limitations. 11 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 1A. TO WHAT EXTENT HAS REAP CONTRIBUTED IN A MEASUREABLE WAY TO BIH’S INTEGRATION INTO THE REGIONAL AND EU ELECTRICITY MARKET? Findings The documents reviewed provided the team a wealth of data on the integration of BiH into the regional and EU electricity market over time. Findings from this document review are presented below. Key informant interview data presented at the end of this section summarizes interviewee perception of the contribution of REAP to these outcomes. The meaning of integration into the regional and EU electricity market BiH’s integration into the regional and EU electricity market is guided by its signing into the Energy Community of South East Europe, which was established between the EU and a number of third-party countries to extend the EU internal energy market to Southeast Europe and beyond. With their signatures to the Treaty establishing the Energy Community, signed in Athens, Greece, on 25 October 2005, the contracting parties committed to implementing the relevant parts of the EU acquis communautaire3 to develop an adequate regulatory framework and to liberalize their energy markets in line with the acquis under the Treaty. The Energy Community aims at establishing a common regulatory framework for energy markets in countries of the contracting parties by extending the acquis of the EU to the territories of participating countries. It covers the fields of energy, environment, and competition contained in the EU legislation. Since its entry into force, the Treaty acquis has been extended on several occasions, such as introduction of the so-called Third Package. It now also includes legislation in relation to security of supply, energy efficiency, oil, renewable energy, and statistics. In line with these developments at the EU level, the Energy Community is obliged to transpose and implement all such provisions. The most relevant provisions in the implementation timetable (regarding electricity markets only) are the following: 3 The acquis communautaire, also simply known as the acquis, may be defined as the accumulated legislation, legal acts, and court decisions that constitute the body of EU law. 12  By July 1, 2007, implement EU energy market directives and regulations on cross￾border network access.  By January 1, 2008, liberalization of the market for all non-household customers.  By January 1, 2015, liberalization of the market for all customers. The first requirement above regarding international trade was partially met before the start of the REAP project, but became fully compliant with EU requirements with the adoption in 2010 of auctions for the allocation of transmission capacities. The second requirement regarding liberalization of the market for all non-household customers is “in process.” To date, two customers (known as “Eligible Customers”) served at 110 kilovolts (kV) are receiving their electricity from sources other than the local supplier. Regulations currently in force provide for an increase in the number of potential Eligible Customers served at lower voltage levels in the very immediate future. Indeed, the RS already allows customer choice for all higher-voltage customers, while in the FBiH, the timetable allows for progressively more customers served at lower voltage levels (except for low-voltage customers) to choose their own supplier by January 1, 2014. Finally, the implementation date of the third requirement of market liberalization (i.e. supplier choice for all customers) is generally regarded as a key date by which to judge the success of meeting the requirements of market liberalization. This is mainly because the number of low￾voltage customers on the power system is so great, which leads to some complexity in matching customer consumption with particular suppliers and assigning related costs for the purpose of financial settlements. This requires not only that appropriate regulations be in force, but also that the processes and logistics be in place. Therefore, almost all stakeholders are looking towards January 1, 2015, as a significant milestone. Level of integration at the start of the project (2007) In 2007, the structure of the electric power industry of BiH was essentially the same as today— composed of the three major EPs (four, including the relatively small Public Utility “Komunalno Brčko”), Transco, and the ISO as operating, state-run companies. The three regulatory bodies of today had also been formed and each Entity had a Ministry of Energy, while the State had an energy department within MOFTER. Each of the three regulatory bodies had established separate jurisdictions. The Entity level bodies presided over electricity markets in the FBiH and RS, respectively, while the State-level body’s jurisdiction included Transco, the ISO, cross-border trade, and the distribution system of Brcko. However, legislation needed to be modified in order to be consistent with Treaty requirements, and regulations compliant with Treaty requirements were practically nonexistent (which is to be expected, as regulations generally follow from primary legislation). While the legal status of the three commissions followed EU directives regarding the independence of the regulator function, the overall efficiency and cost-effectiveness of having the three separate bodies was viewed as problematic and a constraint to EU integration. 13 Current level of integration The contractor’s Performance Based Management Plan (PBMP), summarized in Annex 5, contains certain performance indicators that might be interpreted as indicating measurable progress towards integration into the regional and EU electricity markets. These indicators are taken from a master list of USAID indicators by the contractor;4 however, they are open to some interpretation and are difficult (if not impossible) to verify. More important, while the indicators show what was done in terms of the number of organizations involved, people trained, etc., they say little in terms of actual results regarding level of integration. Despite their limitations in answering the evaluation question, the evaluators have responded to USAID’s instructions to consider the indicators. Two indicators in the PBMP that are considered somewhat relevant are presented in Table 1, followed by a discussion of how the data inform a response to this evaluation question, and their limitations as an adequate measure of integration. Table 1: PBMP Indicators Indicator #1 Ministries Regulatory bodies Utilities NGOs/ media Number of energy agencies, regulatory bodies, utilities, and civil society organizations 3 3 5 approx. 10 Indicator #2 SERC BiH regulator RS regulator Total Number of policy reforms/regulations/ administrative procedures 19 16 10 45 “Number of energy agencies, regulatory bodies, utilities, and civil society organizations undertaking capacity strengthening as the result of USG assistance” could be considered relevant in light of which specific groups are being trained. The fact that all three relevant Ministries in the country are involved, as well as all three regulators and the five utility organizations (three EPs, Transco, and ISO) is significant because these organizations effectively form the institutional backbone of BiH’s electricity industry. Thus, the coverage of relevant stakeholders over the whole of the electricity supply industry in BiH with respect to engagement and training is indeed comprehensive. From the PBMP, it can also be seen that a large number of stakeholders in the BiH electricity supply industry have been trained on market liberalization and Treaty requirements through formal seminars and/or workshops with 4 The list of agreed-upon indicators was provided by the USAID Mission in June 2013. If other indicators are currently in use they have not been communicated. 14 extensive stakeholder engagement. Trainees have included a large number of personnel drawn from MOFTER, SERC, the ISO, Transco, the Ministries of Energy of the FBiH and RS, the regulatory authorities of the FBiH and RS, the three EPs, the media, and NGOs. The PBMP indicates that over 100 people have received training since 2011. The 10 NGOs/media organizations are not so relevant, as they are not an important component of the project, having participated in a limited number of seminars at the beginning of the project. According to the implementer, most are not involved in the electricity industry today. Although a number of stakeholders participated in training, this in itself does not provide any proof of change or success in achieving the objectives of the assistance. Another indicator in the PBMP that might be considered relevant is the “number of policy reforms/regulations/administrative procedures.” This is not really relevant in terms of the items analyzed by the project, as presented in the PBMP in Annex 5. Many items no doubt were “analyzed,” but they may not have resulted in anything and hence are not adequate indicators of achieving the objectives. A more relevant measure could be obtained through a count of regulations passed since 2007 that deal with electricity markets. This information, obtained from each of the regulators’ websites, yields 19 for SERC, 16 for the BiH regulator, and 10 for the RS regulator, for a total of 45. However, these indicators could be judged as weak, since it is the substance of the regulations that really matter, not the number. Based on an analysis of Treaty requirements, the situation in BiH in 2007, and the current situation in BiH, better measures of BiH’s integration into the regional and EU electricity market since 2007 include:  In its “Decision on Approval of the Rulebook on the Allocation of Rights for the Use of Cross-Border Transmission Capacities,” dated March 24, 2010, 5 SERC adopted auctions for the allocation of cross-border transmission capacities, which follows the previously mentioned EU market directive on cross-border access. This method for congestion management of international trade flows in electricity was a significant step in meeting EU market access requirements.  Rule Books on acquiring the status of an Eligible Customer and prescribing timetables for all customers to opt for Eligible Customer status were adopted by the FERK and RSERC over 2011–12. 6 It was learned from stakeholder interviews that, consequently, two Eligible Customers (out of a potential eight) served at 110 kV have selected alternate suppliers. Although provisions exist in the Rule Books for many more such customers, no others have yet applied for Eligible Customer status. 5 http://www.derk.ba/en/ees-bih/rulesandcodes 6 In its 36th regular session held on December 22, 2011, RSERC decided to adopt its “Rule Book” on Eligible Customers (Official Gazette of Republic of Srpska, number 131/11). Also, the “Rulebook on Eligible Customers Supply” was adopted by FERK on May 9, 2012. 15  The FERK Rule Book stipulates that customers currently served at 110 kV and 35 kV can apply for Eligible Customer status, while those served at 20 kV, 12 kV, and 10 kV will be able to do so by January 1, 2014. In the RS Rule Book, all customers currently not served at low voltage can now apply for Eligible Customer status. The difference arises from FERK’s desire to open the market gradually for different categories of customer due to a need to eliminate cross-subsidies in which 10-kV customers subsidize households. Otherwise, from a review of the Rule Books and the relevant SERC decision on market rules,7 it appears that harmonized regulations have been created and largely established at all levels to meet full market opening on January 1, 2015.  There are draft amendments to the Federation Electricity Law,8 in which REAP has had substantial involvement, to enable the Treaty requirements. Legislation in general has been held up due to parliamentary impasses within the FBiH that have impeded new legislation from passing. Although legislative backlog (among other factors) may be seen as an important impediment to market reform, the regulators, with the assistance of REAP, have adopted what they call a “bottom-up” approach in developing appropriate regulations. Of course, they are guided by Treaty requirements (which the BiH has accepted) and the draft legislation, so the “bottom-up” terminology is really a misnomer. Also, nothing in the current laws prevents the required regulations from being put into force.  Both Entity regulators noted that the EPs have been only partially unbundled. Two other beneficiary stakeholders noted resistance to change by the EPs as a problem. A significant unbundling step occurred with the creation of Transco, which effectively created a break in the EPs’ supply chain. However, this happened before the REAP project began. Further unbundling has been required (i) between the generation and distribution functions and (ii) between the distribution and supply functions. There has been resistance by the EPs to the whole process of liberalized markets and unbundling, as it leads to less centralized authority and prominence in the electricity supply industry. Thus, the process of unbundling has been slow. It is understood from the regulators that much progress has been made, although much remains to be done in this area. Other remaining work to achieve the January 1, 2015, milestone includes the resolution of certain issues related to ancillary services, which are understood to be in the process of being addressed by REAP and should be completed by the time the project is scheduled to end. One area judged to be very important by the implementer is having the necessary processes and responsibilities in place once the more competitive retail market goes into effect on January 1, 7 “On Scope, Conditions, and Time Schedule of Electricity Market Opening in Bosnia and Herzegovina,” adopted by SERC on June 26, 2006. 8 The Federation Electricity Law was passed on July 25, 2013. 16 2015. It is necessary to recognize and comprehensively address the distinction between having the appropriate regulations in place and having them operate properly. Stakeholder perceptions Interviewees were also asked questions on their perceptions of progress towards market integration. There was general consensus among beneficiary stakeholders—9.5 out of the 11 ministries, regulators, and utilities indicated in Table 19—that progress on BiH’s integration into the regional and EU electricity markets would have been at least much slower without REAP. All other interviewees shared this sentiment. All the regulators believe they have made significant progress on their end; however, there are still regulations left to implement. The regulators believe they can accomplish this on time if all parties do their part (e.g., the EPs must fully unbundle). Regarding the question of how they evaluate progress to market integration, the three beneficiaries (out of 11) who were willing to provide quantified responses gave answers ranging from 10% to 40%, with a median value of 20%. It must be noted, however, that these responders tended to be the most pessimistic of stakeholders, most of them pointing to the immaturity of the BiH electricity market as a constraint. Other beneficiary stakeholders tended to cite accomplishments as opposed to estimating progress. Two donors stated that BiH has an uneven record in progress to meeting Treaty obligations, in that BiH is ahead of some Treaty countries on meeting some obligations and way behind on others. (They may have been thinking about the gas sector.) In general, BiH is perceived by the two donors to be about “in the middle of the pack.” All stakeholders except for two agreed that much progress on setting up the regulatory framework for liberalized markets has been made. However, as measured by perceived progress towards integration (10% to 40%), there may still be a substantial amount of ground to cover. This low estimate may be based on a lack of confidence in other stakeholders’ ability to perform, since, 8 of the 11 beneficiary stakeholders gave the impression that “we are / we will be ready” when the time comes. Conclusions Table 2 summarizes BiH’s major electricity market requirements for meeting Treaty obligations as of 2007, and the status of these requirements as of July 2013. The table thus provides an indication of what has been accomplished and what remains to be done for market opening in January 2015. 9 One stakeholder group was quite divided on this question, which explains the score of 0.5. 17 Table 2: BiH Progress to Market Integration Milestone Status as of July 2013 Left to be done Implementation of EU energy market directives and regulations on cross-border network access Enabling legislation and regulations In its “Decision on Approval of the Rulebook on the Allocation of Rights for the Use of Cross-Border Transmission Capacities,” dated March 24, 2010, SERC adopted auctions for the allocation of cross-border transmission capacities, which meets the requirements of EU market directives on cross-border access. Nothing, perhaps until a regional auction market is fully established in Montenegro Functioning of a cross-border market Allocation of transmission capacities for cross-border trade from BiH is currently being undertaken on the basis of daily, monthly, and annual auctions by the ISO. Nothing, perhaps until a regional auction market is fully established in Montenegro Liberalization of the market for all non-household customers Enabling legislation and regulations Required legislation and regulations have been passed in all three BiH regulatory jurisdictions. Nothing Functioning of a liberalized market Each jurisdiction has established a timetable for allowing consumers at progressively lower voltage levels to choose their own suppliers through to 2014. Currently, all non–low-voltage customers in RS are eligible, while all customers served at 35 kV and above are eligible in the FBiH. To date, only two customers (out of hundreds eligible) in the whole of BiH have opted to choose their own supplier. Nothing, except wait for a market of lower-cost suppliers to develop Liberalization of the market for all customers Enabling legislation and regulations Required legislation and regulations have been passed in all three BiH regulatory jurisdictions. Nothing Functioning of a liberalized market Not set to happen in BiH until January 1, 2015 As this involves over one million customers suddenly eligible to choose suppliers, the required processes needed to operate and administer the market need to be in place. These will continue to be defined up until market opening on Jan 1, 2015, and probably for some time afterwards, to address any problems that may 18 arise. Unbundling of the EPs Enabling legislation and regulations Required legislation and regulations have been passed in all three BiH regulatory jurisdictions. Nothing Compliance Partial unbundling has occurred in all EPs Full functional and accounting unbundling of distribution operations and sales Requirements of the EU Third Package Enabling legislation and regulations The government of the FBiH has passed the required legislation; EU is providing assistance to draft the required legislation at the State level. RS has yet to pass the required legislation. All three regulatory jurisdictions may need to develop some secondary legislation. It should be noted that the requirements of the Third Package are not as challenging to meet for BiH as those of the Second Package, the main provisions of which have been summarized in the milestones specified above. Compliance with the Third Package The BiH power sector is already largely compliant with the directives of the Third Package, which deals mainly with the unbundling of generation and transmission functions. Minor adjustments in line with the Third Package legislation and regulations 19 The real measure of the success of BiH’s integration into the regional and EU electricity markets, as well as REAP’s contribution, will be known when full retail competition takes place, as explained in the first section of this answer to the Evaluation Question. This point has not been reached yet, but it is set to take place by January 1, 2015. Even if this date is delayed by a year or two, REAP’s contribution to achieving this will have been significant, as seen by the measures already taken, i.e., all or most of the enabling legislation, the regulatory processes required, the regulations themselves, and knowledge of precise requirements by BiH stakeholders. 1B. WHICH OF REAP’S INTERVENTIONS IN THE ELECTRICITY SECTOR HAVE HELPED IN BIH’S INTEGRATION INTO THE REGIONAL AND EU ELECTRICITY MARKET? The implementer saw the following ten accomplishments as contributing most to its mandate of helping BiH integration into the regional and EU electricity markets. It should be noted that these accomplishments are not performance indicators. They are project outputs that have helped in achieving the project objectives, but are only a means to an end, and do not represent the desired end point. They are relevant because they are essentially the implementer’s answers to the evaluation question and thus are an appropriate starting point in addressing it. 1. Development of the Internal Action Plan for meeting the requirements of the Second Package. 2. Policy Paper, which identified eight major impediments in moving towards the Second Package and designated primary responsible parties for overcoming them. 3. Regional Market Paper. 4. Gap Analysis, undertaken by a Market Working Group of 70 people to identify the problems by a hypothetical independent supplier wanting to enter the country’s electricity market. 5. Load Research Guidelines. 6. Congestion Management Rules. 7. Third Package Memo to the MOFTER summarizing requirements. 8. Energy Community Memo, recommending the adoption of the Third Package. 9. Drafting of the Federal Electrical Law. 10. Intervention in the “Transco affair,” which effectively stopped the dissolution of Transco. The above accomplishments are more or less in chronological order and represent a sequence of important steps in arriving at the desired goal of electricity market integration – first, the papers on market development, leading to a gap analysis and action plan, followed by the more specific items required (defined through the preceding steps) such as load research, congestion management and 20 drafting of legislation. Thus, all ten accomplishments can be deemed important, in no particular order.  These interventions have been largely echoed by the interviewees, whose responses to the evaluation questions have been grouped into a number of categories.10 Stakeholders worked together with the implementer on most of the above accomplishments, which allowed for an exchange of views and furthered the development and harmonization of regulations. Four out of eleven respondent beneficiaries cited this as important for them and elaborated that, previously, stakeholders, including Government, regulators, and operating companies, rarely communicated with one another. Therefore, assembling these stakeholders to discuss how they can work together in the common electricity market and to establish common rules and procedures for the market was a major accomplishment. This also included REAP’s role as a mediator in establishing these rules and procedures, which, according to these beneficiaries and the implementer, involved a great deal of negotiation and compromise between various stakeholder groups.  Expert experiences were said to be at least very useful—and, at most, invaluable—by 7 out of 15 respondents, including four beneficiaries, two donors, and one in the NGO/ media category. These experiences included expert opinions, methodologies, and case studies, which enabled stakeholders to understand in detail what has to be done for market opening and prepare the requisite regulations and implementation processes. One respondent went much further to say that he was very impressed by the willingness and motivation of the experts to come forth and share their vast expertise. While it would be desirable to break down the responses by the value accorded to specific experiences, methodologies, and case studies, this level of specificity was impossible to obtain from respondents. REAP helped identify obstacles and gaps in meeting Treaty requirements and developing appropriate plans to deal with them. This was cited by four respondent beneficiaries, with obvious reference to the “gap analysis” undertaken by REAP at the beginning of the project (and indicated above as a REAP accomplishment by the implementer), which set the stage for prioritizing areas of work to develop an appropriate regulatory framework for BiH.  Three beneficiary respondents cited drafting of legislation and valuable commentary on other legislation. 10 The protocol questions included: “What assistance from REAP has been most helpful and has contributed to BiH integration into the energy market? Can you evaluate the impact of the assistance on your agency’s work? What specifically has resulted or been accomplished or been impacted in your agency through REAP assistance?” 21  Another major theme included the intervention in the Transco affair, in which REAP assisted USAID and donors to block an initiative to pass a law in the BiH parliament to effectively dismantle Transco; this intervention prevented backsliding. Discussions with USAID and the implementer indicate that this was indeed a concerted effort that involved REAP. Though there was no evaluation question about the Transco affair, one beneficiary respondent mentioned the Transco affair during interviews, as well as one respondent in the NGO/media category. Six respondents, including one donor and five beneficiaries, mentioned the ongoing dysfunctional situation at Transco, which suggests they are well versed in the Transco affair.  Provision of a load-research methodology and analytical methods to develop customer load profiles was mentioned by four beneficiary respondents.  Support in dealings with the Secretariat of the European Community was cited by one respondent as being of substantial benefit to REAP, and has provided significant support to MOFTER in its role as the "spokesman" for BiH before the Treaty Secretariat and the EU and to understand the various Treaty requirements and directives. While it might be useful to rank implementer interventions in order of the frequency with which they were mentioned by respondents, this will not provide meaningful results, as such an analysis would be subject to considerable recall bias and selection bias. As the project is now six years old, respondents are more likely to mention less relevant recent activities (e.g., load research) than more critical activities that were undertaken years ago (e.g., the Gap Analysis, which was instrumental in identifying gaps and then defining the required steps to be taken to fully meet Treaty requirements). Also, certain items are more relevant to specific stakeholders. For example, assistance in drafting the Federal Electricity Law is no doubt an important accomplishment by REAP, but would be recognized most by the government of the Federation of BiH (and perhaps to some extent by FERK), and less by other stakeholders. Conclusions REAP was of substantial assistance in communicating approaches to meeting Treaty requirements and understanding the functioning of open electricity markets. To a large extent, the ten accomplishments (or outputs) claimed by the implementer have been echoed by the stakeholders. While the implementer’s answer to the evaluation question focused on specific work items, the other stakeholders’ answers were of a more general nature (e.g. working together, expertise); however, the implementer’s accomplishments tended to come up repeatedly in the interviews. While a statistical approach to ranking the accomplishments might be useful, recall bias and selection bias severely limit the possibility of obtaining meaningful results. However, based on the responses of stakeholders, it can be concluded that the ten major interventions cited by the implementer have indeed helped most in BiH’s movement towards integration into the regional and EU electricity market. 22 2. WHAT HAVE BEEN THE MAIN OBSTACLES IN THE IMPLEMENTATION OF REAP, IF ANY? Findings It is important to make a distinction between obstacles to the implementation of REAP’s scope of work—what falls within its “manageable interests”—and obstacles to implementing the terms and requirements of the Treaty. The team found that as far as the provision of technical assistance is concerned, REAP has not encountered serious obstacles and, according to all but two institutions consulted, the relevant institutions are advancing, following through on the recommendations made by REAP on electricity to create the legislative framework, the regulations, and implementation steps required for market opening. However, in carrying out the technical assistance, REAP, despite its neutral position, has had to negotiate political obstacles and find compromises among organizations with different perspectives and political agendas. A total of 9 of the 11 institutions consulted (plus all donors) cited “politics” as the main problem obstructing progress in the energy field. Clearly, the biggest obstacle to implementing the terms and requirements of the Treaty has been the “politicization” of energy and the use of energy laws and energy institutions as a political tactic by certain factions in the RS as part of the ongoing fight over the nature of the State and the three-way structure of its institutions. Three beneficiary institutions cited this as a direct obstacle, couched in a question about the constitutional authority of State institutions in the realm of energy, thus turning energy into a battlefield. This resulted in (i) legislative logjams; (ii) paralysis and obstruction within Transco, with company officers refusing to plan for investments in improvements to the system; and (iii) an attempt to introduce a bill in the BiH Parliament that would dismantle Transco,11 which would have disrupted any integrated national or international integration. Two beneficiary respondents and one donor remarked that the political obstruction by the RS is triggered by fear of further Entity “loss of control” beyond what was already prescribed by the DPA and sentiment against the USG (and others) for forcing the RS to accept State-level institutions in the energy realm. Other respondents, in particular donors, intimated indirectly that these political issues could only be settled by strong political will at the top. The second obstacle, which one respondent directly cited and to which others indirectly alluded, is a general lack of interest in or sense of urgency about energy as an issue at the national Parliament level. As a result, little attention is being paid to meeting Treaty requirements or to market opening, and the Government is not inclined to spend its “political 11 REAP played a helpful role in advising on ways to deflect this attempt to dismantle Transco, as well as a role in mediating the internal problems in Transco, which caused attention and resources to be directed away from other programmed activities. 23 capital” on fixing the situation. Various related issues were cited for the lack of interest. One respondent noted that the poor state of the economy means that legislators are more concerned with short-term economic issues than with the long-term consequences of falling behind or being left out of an integrated regional energy market. For one thing, energy prices in the BiH are low compared to those in neighboring countries, and politicians are keen to keep them low, both to assuage consumer pocketbook concerns in a pre-election season, and because of the effects of an energy price increase on the economy. Low prices have meant that politicians are not under pressure to obtain lower prices through increased competition in the sector. A bit of subtext to this issue, noted directly by six beneficiaries and two donors, is that politicians are exerting pressure on the regulators to hold energy prices down artificially, which is not beneficial for market opening. One respondent expressed concern that such pressures on regulators may continue past 2015 (even after the 2014 elections) when the market fully “opens.” The result may be that the default price for energy will remain low and true market competition will not take place. There were mixed opinions about the prospects of sanctions should the 2015 deadline not be met. Two donors believe the EU will use its leverage via the Instrument for Pre-Accession Assistance (IPA) fund, which could be stopped if inadequate progress is made on meeting energy-related EU accession acquis. Another donor thought that since BiH is ahead of other countries nothing would happen. Finally, a lack of urgency can be attributed to what two non-beneficiary respondents categorized as “vested special interests” (and that one labeled outright corruption), including practices such as kickbacks, nonstandard practices in procurements, rules on investments, and a reluctance to give up the energy companies as a government asset that can be manipulated for personal gain. Divergent opinions were expressed about both the Government and the public’s attitudes towards open markets. One NGO respondent stated that public sentiment is less favorable now than it was five years ago because the economy is worse and people believe that they will “lose out” through higher prices in an open market and an inadequate social safety net. An additional three beneficiary respondents mentioned that unease about market opening in the energy sector is based on observation of negative effects of privatization in neighboring countries, with fears that consumer interests may not be well served, in particular through higher prices and loss of investment in service. (These remarks were made prior to the news stories about energy price wars in Croatia.) According to three beneficiary respondents and the implementer, there is a visible lack of interest on the part of the EPs in becoming part of a competitive market, as it will require vastly changing their way of “doing business” and loss of substantial perks that they gain from their current employment status in a state agency. In Government, it was noted, the loss of energy companies as State assets is perceived as “ambivalent tending towards negative.” One beneficiary noted very loudly that a market situation will not function. Finally, one donor noted that there is no real recognition among parliamentarians of the longer-term and broader aspects of international trade and the economic benefits that will accrue to the country by making operational an integrated regional market (for which an integrated national market and market opening are but the first steps). A further obstacle to moving towards meeting the full terms of the Treaty and, in particular, achieving regional market integration, noted by four beneficiaries and all donors, is the absence 24 of progress towards a national and integrated energy strategy. This, together with a lack of clear and consistent procedures and parameters for investment, creates a serious brake on investment in electricity generation. While there is sufficient electricity being produced to meet current internal needs with some export sales, the infrastructure is old and will need replacement. Additionally, the country has the potential to generate considerably more hydroelectric power than at present to meet the new needs of a growing economy and for export to neighbors with electricity deficits, a prime objective of regional integration and for earning export revenues. Plus, in order to meet the upcoming requirements for renewable￾energy sourcing, the country may turn to other energy sources such as wind. However, any new investment depends on capital, and capital depends on certainty about authorizations. Realistic investment planning has not taken place at the national level, despite a World Bank offer to fund a national strategy and a current offer from the EU, provided that entities cooperate in the effort. Similarly, the National Energy Efficiency Action Plan proposed with USAID assistance was rejected, and the National Action Plan for opening the market has not been acted upon. While each of the Entities has an investment plan, the plans are neither realistic nor unified. Both donors and beneficiaries mentioned that investors are wary about the lack of clarity on plans and rules for new investments in energy.12 The result may be difficulty for private companies in obtaining capital for new investment until the rules of the game are more clearly established. Conclusions The main obstacle to smooth progress on meeting internal and regional energy market integration has been the dominance of politics and the use of energy by the RS as one of many political tactics to obtain larger political goals, including the obstruction of legislation and a refusal to participate in a unified energy strategy. Informants speculate that the resulting political blockages will only be removed if there is (i) strong political will at the top and/or (ii) credible intervention and/or sanctions by the EU and perhaps the USG (as happened previously in the Transco affair). Along with skepticism about the will of the Government to allow energy prices to rise in the near future, there are questions about the potential for substantial progress on the political blockages prior to elections in October 2014. A new government may have a new mandate, but the deadline of January 1, 2015, would not leave sufficient time to settle any remaining political obstacles and meet Treaty electricity requirements on time. If no progress is made to resolve political obstacles prior to elections, a new government may be able to negotiate a delay in sanctions with the EU, assuming substantial progress continues. After elections, politicians may feel less constrained about intervention in the price of electricity (although this is highly uncertain), which will allow for a freer and more competitive market and the furtherance of regional integration, an objective whose benefits to the country appear to be 12 The corruption level in BiH may also act as a brake on outside investors given the lack of transparent procedures, but only the implementing consultant cited this factor. 25 poorly understood and that may explain a lack of urgency. Also, currently unforeseen roadblocks and problems may be left unresolved for lack of a mediator when REAP assistance ends. The fear of a breakdown in communication between government, regulators, and energy companies might well materialize, especially with forces acting to undermine the concept of a unified and internationally integrated electricity market in BiH. 3. ARE THERE ANY IMPORTANT ELEMENTS OF THE SEE ENERGY COMMUNITY TREATY THAT REAP HAS NOT ADDRESSED, IN PARTICULAR THOSE PERTAINING TO IMPROVED ECONOMIC ASPECTS OF GOVERNANCE RELEVANT TO BUSINESS ACTIVITY? Findings After discussion with USAID, the phrase “pertaining to improved economic aspects of governance relevant to business activity” was interpreted to mean “to assist commercial development,” as the evaluation team had some difficulty understanding the original wording. As stated in the SOW, the question directly refers to Intermediate Result (IR) 2.2 in USAID/ BiH’s Country Development Cooperation Strategy for Bosnia-Herzegovina 2012–2016. This IR is meant to improve BiH’s business environment by helping the BiH public and private sector establish strategies and policies, then prioritize, draft, and implement regulations that stimulate economic growth, investment, and jobs. The Strategy particularly points to regulatory and policy reform assistance under this IR, as well as harmonization of regulations across Entities and capacity-building to attract investment. The energy sector is mentioned as a priority area for assistance, particularly for enhancing its attractiveness for foreign investment and continuing to assist BiH in developing transparent, functional, and responsive energy regulatory bodies. As seen in the response to Evaluation Question 1a, REAP has fully addressed the task of developing appropriate energy-sector policies (through assistance in drafting legislation) and regulations to establish an integrated electricity market in BiH—although the results are still to be realized. This assistance has been provided through considerable capacity-building, conducted through both formal training and extensive stakeholder engagement in developing the workings of the internal electricity market. In theory this should contribute to economic growth, investment, and jobs. Although not cited as a significant obstacle in response to Evaluation Question 2, it was found that two of the eight non-EP institutions interviewed (both beneficiaries) pointed to the EPs as problem areas in their unwillingness to adapt to a more liberal electricity market and that the EPs are not currently ready for markets. This was not an interview question, so it is significant that this subject came up at all during the interviews. This is a natural reaction of monopolistic, vertically integrated electric utilities to open electricity markets, no doubt encouraged by the 26 EPs’ high-level political appointee managers, who would perceive market opening and unbundling as a political football to be used as a tactic in a larger game. This attitude might have been mitigated by more REAP involvement in working with the EPs. One EP did, in fact, mention that REAP should have engaged the EPs more—and this despite EP participation at various REAP activities. Among the other EPs, one did not bring this up, while certain individuals in the meeting with the other EP were openly hostile to REAP for political reasons.13 In any case, as previously mentioned, “Restructuring and commercialization of energy companies” was a major task (among two) in REAP’s original SOW, which was seemingly less emphasized during the course of the REAP project. It should also be noted again that this part of REAP’s original SOW was out of the evaluators’ scope of work and therefore cannot be addressed in any great depth. However, the implementer, having gone through similar experiences in other jurisdictions, has the view that, with market opening, the EPs will simply be forced to adapt. In any case, the implementer stated that commercialization of the EPs was not in its mandate. As seen in Table 3 below, it was found that stakeholders were generally very satisfied with the assistance provided. Table 3: Rating of REAP Services by Number of Individuals Rating (out of 10) Number of Individuals 10.0 10 9.0 8 8.5 4 Although the interview questions differentiated between technical satisfaction and effectiveness, it was difficult to get all interviewees to distinguish between the two, so these ratings are composites. One beneficiary stakeholder seems to have summed up the general sentiment by saying, “REAP provided more than we could process.” Four respondents, all beneficiaries, said that REAP did all that was in its power to do, and that things still left undone were outside its competence to accomplish—i.e., political action, such as problems associated with Transco— even though it sought to overcome such constraints by acting as an advisor and mediator in arriving at compromises. 13 This appears to be an example of the political fighting over wider policy issues that REAP had to deal with, as discussed in the answer to Evaluation Question 2. It is understood that many of the staff of this EP actively participated in REAP. 27 In the “not addressed” or “left undone” category, stakeholders also mentioned the following:  Ancillary services (brought up by three beneficiary respondents) and social subsidies (brought up by two beneficiary respondents and one donor): With respect to ancillary services, this probably refers to ongoing issues in this area that are currently being dealt with by the project. Regarding subsidies, this is an EU requirement, although it is not directly related to electricity markets. Therefore, it is debatable whether this issue is one that REAP should address. Nevertheless, it should be noted that most stakeholders were mindful of the possibility of higher prices, leading to the requirement of an adequate social safety net.  Barriers to investment and raising capital: One issue is the lack of a standard and open tendering procedure, clear processes for approval of large energy investments, and clarity about who makes these decisions. Two respondents, neither of whom was a beneficiary, mentioned this, noting the potentially politically divisive outcome of private investors pulling out of new investments on the border with Serbia. This was not included in the REAP scope of work. The second issue in this area is that no unified and rational national investment strategy exists as a precondition for attracting private capital to invest in energy projects. It was found that various donors have been trying for several years to get approval for a least-cost power system expansion study for BiH, but they have been hindered by political forces. Conclusions REAP has addressed all the technical considerations for regulation relevant to the opening of the market and market integration. The respondents were very satisfied with the extent of assistance and generally felt that nothing had been left out. What has not been accomplished yet is due to “politics” and beyond their “manageable interests” or influence. Had it been within its influence, the whole of the BiH market might currently be functioning in accordance with Treaty requirements instead of being partway there, as described in the answer to Evaluation Question 1a, which provides an indication of the remaining work required for market opening and market integration. REAP’s work on market integration has definitely assisted commercial development in BiH’s electricity supply industry through enabling legislation and regulations. However, it should be noted that the evaluation question may be better answered through an in-depth analysis of the second major task contained in REAP’s original SOW, “Restructuring and commercialization of energy companies.” This was explicitly beyond the scope of the evaluators’ mandate. 28 4. WHAT IS THE RELEVANT STAKEHOLDERS’ ESTIMATE OF THE EFFECTS ASSOCIATED WITH USAID/BIH DISENGAGEMENT FROM BIH ENERGY POLICY ASSISTANCE AFTER REAP ENDS IN SEPTEMBER 2013? Findings The work of REAP has been highly praised by all stakeholders, except for one. In addition, the consensus of opinion seems to be that progress will slow down but not stop. According to a proverb cited by at least two different stakeholders, “The cemetery is full of irreplaceable people,” indicating that despite the cessation of outside technical assistance, progress towards market opening (and regional integration) will continue as the institutions believe they have acquired a sufficient degree of information and understanding through REAP to progress on their own. Individual comments ranged from “We would have done all of it by ourselves without REAP assistance” to the admission that while the disengagement will not be a disaster, it will leave a huge gap unfilled and will be extremely missed. This range of opinions is likely attributable to differences in levels of confidence amongst the different concerned agencies and at the same time to a recognition of the important role played by REAP as a facilitator and neutral technical expert. Four respondents mentioned that there will be a slowdown in meeting the Treaty requirements and that this will be felt both on the regulatory and legislative side. However, three respondents noted that, with the current political impasse and upcoming elections, little progress on major issues could be expected before the end of 2014. Drafting and harmonization of new and amended legislation may be affected (see remarks from MOFTER below), although EU assistance has been promised to fill the capacity gap at the State and Entity Ministry level in order to deal with obligations under the Third Package. The FBiH Ministry expects difficulty in meeting the timetable for drafting secondary legislation and detailing every step of implementation once the energy amendments pass, as they lack staff and the timetable is short. (However, this is the task of the FBiH regulator.) Respondents uniformly contrasted the EU assistance methods with those of USAID, claiming that since the EU is not hands-on and would not employ the REAP “bottom-up” approach to assistance, it is far less effective. Two respondents, including a regulator and an energy company, expressed the desire for expert assistance in the implementation of fully open markets after January 1, 2015. While they expressed confidence in their ability to proceed towards market opening based on the assistance they have already received, there are concerns about dealing with what will happen in practice after opening, and they anticipate a need for expert advice on necessary adjustments. Donors in particular were concerned with the gap left when REAP no longer is exerting its neutral leadership, facilitation, and initiation roles, as it has effectively brought together the various energy institutions to share concerns and ideas in a way that would not have happened otherwise. Two respondents thought that the leadership and facilitator role should be assumed in the future by the SERC. In any case, there was a general concern that there will be a breakdown in communication between government, regulators, and energy companies, and no 29 one group of persons well versed and up to date on the details of the sector will be left to ensure that there is continual movement towards meeting requirements of the Treaty or integrating the actions of one agency with those of another. The Scope of Work for the implementer stipulates assistance to enable the different institutions to “speak with one voice” on energy policy and regional integration to the Treaty Secretariat and others. According to MOFTER, the donors, and two other respondents, REAP assistance will be particularly missed by MOFTER in dealing with the international Treaty agreements and international markets where the State is the interlocutor. MOFTER expressed concern at its ability to respond to current infringement procedures—there are two in process, one on gas and the other on the sulphur in fuel—and a possible future process concerning the lack of energy investments by Transco. The Ministry also stated that it needs assistance on an Action Plan for renewable energy, as well as assistance for the gas and oil sectors and believes that its lack of staff (including legal expertise) leaves it in a weak position to deal with Treaty and EU requirements in the future without outside assistance. Conclusions There is no doubt, as expressed by respondents, that REAP will be missed for both its technical knowledge and facilitation capacities, and that progress will slow down when left to individual agencies. However, most of the beneficiary organizations expressed confidence in their ability to use the information provided by REAP and apply it to resolving issues related to market opening. This is in no way a negative commentary on REAP because the technical knowledge and assistance was judged highly valuable and effective. The sole organization that expressed major doubts about its ability to implement necessary technical tasks without continuing assistance is MOFTER. Of importance in considering the need for continuing assistance in the future is the looming date of January 1, 2015, when full market opening occurs; by that time, the beneficiary organizations should have finalized technical regulations and implementation steps for market opening. As noted in the answer to Question 2, beneficiaries generally believe they will be ready. Other, policy-related requirements of the Treaty outlined elsewhere in this report may well not be met by that date, as several respondents expressed skepticism about political decisions or actions until after elections in autumn 2014 at the earliest. If this holds true, additional technical assistance before that date may not be a high priority. Both donor and beneficiary respondents agreed that REAP can do nothing to resolve these issues, as they are not technical matters. If elections have not resolved the roadblocks, then high-level political pressure may be the only solution and technical assistance would be welcomed once the political issues are resolved.14 14 The current REAP contract expires at the end of September. The design and contracting of a new technical assistance project would likely take a minimum of six to nine months (meaning that assistance would not begin until early spring 2014), but the policy elements may still be pending at that point in time. 30 5. HOW WELL IS REAP’S IMPLEMENTATION COORDINATED/SYNCHRONIZED WITH OTHER DONORS’ ACTIVITIES AIMED AT INTEGRATING BIH INTO THE REGIONAL AND EU ELECTRICITY MARKET? Findings Four donors were initially found to be current players in the BiH energy sector—namely, the EU, the European Bank for Reconstruction and Development (EBRD), the World Bank, and the Office of the High Representative (OHR). It was possible to interview only two—the EU and the OHR—for reasons of availability. The OHR is not technically a donor but rather an ad hoc international institution responsible for overseeing implementation of the Dayton Peace Accord. However, in terms of outlook and purpose, it resembles a donor and provided useful input as a respondent. Regardless, all stakeholders were asked about their perceptions of donor agencies and coordination between donors. In order of mention among the six beneficiaries who were able to comment on donor funding other than USAID, four mentioned the EU as providing assistance, two mentioned the World Bank, two mentioned EBRD, and a single mention went to the British Embassy in Slovenia, referred to by the respondent as “the Norwegians.” Other donors include the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) from Germany, UNDP, and the Treaty Secretariat. Items mentioned during the course of the interviews regarding donor funding were the following:  USAID was found to be the only player in the electricity markets field in BiH at the moment, with a high reputation in competence and effectiveness, as noted in response to Evaluation Question 1b.  USAID was found to be more practically oriented to problem solving and reactive to real-time needs, and was mentioned by four beneficiaries. EU assistance is oriented towards legislation, which will be provided to meet the Third Package requirements. This was thought to be a good split of activity between REAP and EU (from two respondents, one of them a beneficiary). However, EU is generally slow, bureaucratic, restricted to its terms of reference, and inflexible (from the same two respondents).  Effectiveness of USAID assistance has been affected by lack of conditionality from other donors on policy issues (view from the USAID Mission). Other donors tend to be lenders, which USAID is not. This provides other lenders with a certain amount of clout in allocating funds, as a loan to help finance a generating plant, for example, may be accompanied by appropriate covenants in the loan agreement that deal with critical policy issues.  EBRD and World Bank have provided some good studies. For various reasons—one of them being never having had experience with donors other than USAID—five of the beneficiary respondents had nothing to say about funding from any other donors. A few others were able, for example, to vaguely recollect a one-week seminar on 31 electricity markets sponsored by the British Embassy in Slovenia. The only other donor providing help on electricity markets is the EU, and, as described above, the division of work between the EU and USAID is quite clear. Also, the interviews suggest that there has been relatively frequent contact between the EU and both REAP and the Mission. Conclusions The findings suggest that involvement by other donors in the energy sector (let alone electricity markets) is minimal. In the area of assistance provided by REAP, no issues exist in coordinating efforts because no other donors, except the EU, are addressing electricity market integration. The efforts undertaken by the two are not conflicting and appear well coordinated. RECOMMENDATIONS Considerations for Continuing Assistance A six-year commitment to the REAP: Is it enough? One would normally assume that, after six years of commitment by REAP, much of the technical assistance necessary to disseminate knowledge on the workings of open and integrated electricity markets, as well as the actions required in BiH to achieve this, would have been already provided and absorbed by the recipients. If this is the case, the following questions arise: What is the incremental value of continuing to provide technical assistance? What more will be gained? Overall, two of the three Ministries and two of the three regulatory authorities seemed confident about achieving the requirements of the January 2015 milestone. However, all seemed cognizant that total progress would slow down without REAP. The two remaining respondents seemed rather apprehensive about the absence of REAP. A member of the implementing consultant at a seminar in 2011 is quoted as having said, “Much work is still required, and it could take seven more years to achieve complete integration.” Several interviewees stated that it is too soon to drop the assistance, while others noted that the role of an expert/mediator may be crucial in the months immediately preceding and following the 2015 milestone. Having all the appropriate regulations and procedures in place for market opening is a must. Moreover, they can only be tested in a “going live” situation, which will occur on January 1, 2015. USAID must consider the incremental or marginal value to be obtained by providing additional assistance on items of unfinished “technical” business prior to January 2015, including work on implementation processes and restructuring of the EPs. The next question is the following: Subsequent to market opening, are there additional technical needs to meeting Treaty obligations where USAID assistance can make a difference that are not linked to unresolved political issues? For example, a national energy strategy and the gas laws, to name several outstanding issues, require political consensus before progress can be made on the technical front. 32 The obstacles to overcome are largely not of a technical nature; they are political. There are a number of political factors to be considered:  The first is the concept of providing assistance to a country where that assistance is actively subverted by some stakeholders because there is a substantial lack of political will towards the goals and purpose. Those who do want the assistance are hung up in political stalemates that are hindering them from moving forward and where technical assistance will not make a difference. On the whole, however, the outsider’s perception is that of a lack of political will at many levels to the policies inherent in the Treaty, for the various reasons outlined in the answer to Evaluation Question 2. Faced with such a situation, the inclination of a donor may well be to target other areas where the scarce aid money can be used for projects having less risk and more immediate social and economic benefits.  The second political factor is the role of the USG in BiH. The USG was an initiator and was heavily involved in developing the DPA, which has, as a basis, unified BiH. The USG is thus committed to a unified BiH. The USAID approach to the energy sector to date has been consistent with the premise that energy unification is a means to strengthen the unity of the State. Given the BiH political forces at work, USAID’s leaving the scene at such an important juncture in the development of BiH’s unified electricity market would not only be inconsistent with that policy; it would also send a “bad sign” (as expressed by one donor) to both REAP stakeholders and the BiH public. Also, a situation similar to the Transco affair may yet emerge again. Without any strong intervention such as that exhibited by REAP in the Transco affair, significant damage may be done to what has already been achieved.  The USG must consider whether it is desirable to spend its “political capital” in BiH on overcoming political obstacles in the energy sector rather than address the unification issues in another fashion. Donors noted that only strong political will at the top will resolve the political obstructions and that this may only come if the USG and the EU force the issue. However, the USG may not want to intervene on an SEE Treaty issue. The Treaty is meant to further a more rational, regional, and EU energy market and prepare countries for EU accession, which, in other parts of Eastern Europe, has been considered a matter between the EU and the Treaty signatory—not a primary USAID responsibility. A final consideration is the extent to which the economy of the BiH—and hence its political stability, which is at the heart of USG interests—will be assisted by developing its private sector domestic and export energy capacity. However, this, too, will proceed only after resolution of the policy disagreements about rights and responsibilities. Timing Timing of any possible continued assistance under a project similar to REAP will depend on a number of factors, some of which, unfortunately, are largely uncertain. These include:  USAID lead time required to develop and contract new projects. The timing for a new follow-on project may take up to one year, which will provide little time to prepare for 33 January 1, 2015. The assistance needed may well be required much before it is ready to be provided.  Impasse at the Government level prior to elections to be held in late 2014 and spilling over into 2015 while a new government is being formed. This may hold up required legislation. The regulators, as a consequence, may be reluctant to implement the required regulations for market opening.  The consequence of not meeting the Treaty deadline of retail choice for all customers by January 1, 2015. The EU might decide to impose penalties, but this will become less likely if the date approaches and a significant number of other signatories to the Treaty are not ready. An indirect penalty would be not to provide funding for infrastructure under the EU’s IPA program. Alternatively, BiH could request a delay in the deadline, which would fit in well with possible lead times in preparing and implementing further USAID assistance. Recommendations 1. Given that USG policy is embedded into the DPA and that the REAP work is not yet finished, some continued assistance beyond REAP is desirable but is limited to clearly technical matters where all parties are in agreement and that do not depend on the resolution of political issues. This may take the form of a smaller, more limited project designed to enhance the implementer’s role as a facilitator, mediator, and the provider of “neutral” technical expertise where disagreements might arise between the various stakeholders, leading up to 2015 and carried over into assistance implementation of market opening. To hold down costs, expensive outside technical expertise could be minimized. Continuing with the theme of restructuring and commercializing, the EPs contained in the original SOW for REAP should be explored. Although this issue is not explicitly in the evaluation scope of work, it has arisen in the answer to Evaluation Question 3. It is recognized that there might be resistance from the EPs in initiating such work. But again, it is a question of summoning the political will to carry out the required tasks to meet Treaty requirements. There is a need for such assistance, based on the results of the stakeholder interviews. The original intention of this Task, as written in the 2007 SOW, was to work extensively within the EPs, mainly to help in unbundling. This effort, subsequently undertaken by the EPs themselves, is not complete and might still take some time. Such assistance might also include other EP activities such as procurement practices. Once the timing of any proposed assistance package is known, it should be considered in line with the current status of movement towards electricity market integration in the BiH. 2. A more comprehensive assistance package, which might include support in the development of a national electricity development strategy, should be considered only in light of subsequent political developments in BiH, including the internal political will to deal with the outstanding policy issues, the political will of the EU to intervene (and the effects of such intervention), and, finally, the interests of the USG in using its political capital to influence the policy issues. Without policy changes within BiH, there are definite limitations to the incremental value of USAID technical assistance and to the field of activities where technical assistance can have an impact, both now and in the future. 34 ANNEX I: EVALUATION SCOPE OF WORK C.5 STATEMENT OF WORK Audiences and Intended Uses The primary audience for this evaluation is USAID Mission in BiH and USAID/Washington. Secondary users are local government officials with whom USAID collaborates and other donors. The intended use of this evaluation report is to provide information and lessons learned to USAID/BiH’s strategic decisions for energy sector programming, whether to remain engaged in the sector and in what ways. The evaluation report will provide information for USAID/BiH for assessing the extent to which the Mission’s involvement in the energy sector contributes to the Mission’s CDCS goal of helping BiH become more stable and closer to Euro-Atlantic integration. Evaluation Questions USAID/BiH has developed six evaluation questions for this evaluation. They are presented in order of priority: 1. 1a. To what extent has REAP contributed in a measureable way to BiH’s integration into the regional and EU electricity market? 1b. Which of REAP’s interventions in the electricity sector have helped in BiH’s integration into the regional and EU electricity market? 2. What have been the main obstacles in the implementation of REAP, if any? 3. Are there any important elements of the SEE Energy Community Treaty that REAP has not addressed, in particular those pertaining to improved economic aspects of governance relevant to business activity? 4. What’s the relevant stakeholders’ estimate of the effects associated with USAID/BiH disengagement from BiH energy policy assistance after REAP ends in September 2013? 5. How well is REAP’s implementation coordinated/synchronized with other donors’ activities aimed at integrating BiH into the regional and EU electricity market? 35 TECHNICAL REQUIREMENTS Basic Requirements Design Based on the information provided within this SOW, the Contractor will submit a written description of the proposed methodology that it will apply to carry out the Evaluation. The Contractor is expected to use a design and methodology that will generate the highest quality and most credible evidence (accurate and replicable) that corresponds to the question being asked. After the award, the Contractor will submit to USAID a detailed design and evaluation plan as a first deliverable. To carry out this Evaluation, it is anticipated that the Evaluation Team will need to carry the following tasks:  Initial orientation meetings with USAID/BiH and the project implementer;  Review of key documents and background material on BiH’s energy sector;  Review of project documentation provided by the Mission. Existing Data section of this SoW provides a list of the project documents that the Mission or project implementer will provide to the Contractor implementing the evaluation after the contract is signed;  Identification of any other relevant performance information sources;  Meet with USAID to review the evaluation objective;  Collect data from a broad range of stakeholders familiar with the BiH and energy sector, following the approved evaluation design; and  Report preparation and briefing. METHODS Data-Collection Method The Contractor will provide a detailed data collection method for each evaluation question in the Evaluation Design. It is anticipated that data will be collected through desk reviews and key informant interviews: For question 1, the Contractor will define what is meant by integration into the regional and EU electricity market, describe the level of integration at the start and end of the project, describe the interventions that are intended to affect integration, and address USAID/BiH contribution through stakeholders’ views and documentary data. The Contractor is expected to collect both qualitative and quantitative data. As a first step, the Contractor should visit the State Energy Regulatory Commission website () for all relevant BH and EU legislation and links to local and international energy bodies. For questions 2, 4, and 5, the Contractor will collect data mainly by key informant interviews. 36 To answer question 3, the Contractor will analyze the SEE Energy Community Treaty and identify the areas that are in line with USAID/BiH Intermediate Result 2.2 (improved economic aspects of governance relevant to business activity), but which REAP has not addressed. Data Disaggregation When applicable, the data collected must be disaggregated by gender (e.g. percentage of trainees that were female) Data Analysis The Contractor will provide a detailed data analysis plan in the Evaluation Design. The plan must directly address each evaluation question with specific methods for collecting and for analyzing the data that will be used to answer it. Methodological Strengths and Limitations The Contractor will disclose all methodological strengths and limitations in the Evaluation Design. C.6 DELIVERABLES The Contractor shall submit the reports, deliverables, or outputs as further described below to the COR. All reports and deliverables shall be in the English language, unless otherwise specified by the COR. All deliverables will be submitted electronically. The Final Evaluation Report must be submitted in hard copy and electronic copy. The Contractor must deliver the following: 1. Detailed Final Evaluation Plan A detailed Final Evaluation Plan shall be completed by the lead evaluator within one week of the effective date of the contract and presented to the Contracting Officer’s Representative (COR) or his/her designee. The plan will contain a Work Plan and Evaluation Design. The evaluation design will include a detailed evaluation design matrix (including the key questions, methods and data sources used to address each question and the data analysis plan for each question), draft questionnaires and other data collection instruments or their main features, known limitations to the evaluation design, and a dissemination plan. The final design requires COR approval. Unless exempted from doing so by the COR, the design will be shared with country-level stakeholders as well as with the implementing partners for comment before being finalized. The work plan will include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation team. 37 2. Draft Report The Draft Report shall be submitted electronically to USAID/BiH for review and is due two weeks after the field work completion. 3. Final Evaluation Report The final report is due two weeks after receiving USAID comments. The report shall not exceed 30 pages (excluding appendices). The evaluation final report should include an executive summary, introduction, background of the local context and the projects being evaluated, the main evaluation questions, the methodology or methodologies, the limitations to the evaluation, findings, conclusions, and recommendations and lessons learned (if applicable). The executive summary should be 3-5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include:  The Evaluation Scope of Work.  Any “statements of differences” regarding significant unresolved difference of opinion by funders, implementers, and/or members of the evaluation team.  All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides.  Sources of information properly identified and listed.  Disclosure of conflicts of interest forms for all evaluation team members, either attesting to a lack of conflict of interest or describing existing conflict of interest. The following link provides additional guidance on how to prepare evaluation reports and an evaluation report sample: The Contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within 30 calendar days of final approval of the formatted report. 4. Flash Drive Including All Instruments and Data in Formats Suitable for Re-Analysis All records from the evaluation (e.g., interview transcripts or summaries) must be provided to the COR. 38 All quantitative data collected by the evaluation team must be provided in an electronic file in easily readable format agreed upon with the COR. The data must be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. 5. Oral Briefings The Evaluation Team will conduct an exit briefing prior to departure from country. 39 ANNEX 2: INTERVIEWEE ORGANIZATIONS AND PEOPLE MET EP HZHB Mostar Zeljko Topic, Head of Sector for Metering and Relations with Operator Mirona Munetic, Sector for Market Boris Sunjic, Department for Metering Independent System Operator Omer Hadzic, Head of ISO Milodrag Košarac, Federal Ministry of Energy, Mining, and Industry, Federation of BiH Tarik Begic, Head of Electric Power Department Amira Pintul, Assistant Minister of Energy State Electric Regulatory Commission Slovodanka Milasinovac, Secretary Nikola Pejic, Commissioner Edin Zametica, Advisor to Commission Mirsad Salkic, Commissioner REAP implementers AEAI Tatjana Konjic, Consultant (short term) Jane Wilson, Head of Project (Chief of Party, first half of project) Ognjen Markovich, REAP Chief of Party Drazena Peranic, (former ) REAP Public Relations and Media Transco Dusan Mijatovic, Acting General Manager ZdenkoVukic, Executive Director for System Operation and Maintenance Cvjetko Zepinic, Chief of Department for Operation and Maintenance Planning Republic of Srpska Government, Ministry of Industry, Energy, and Mining Milan Jankovic, Head of Department, Energy Sector Association of Employers of B&H Alija-Remzo Baksic, Director Journalist Rubina Cengic, START Magazine USAID Mission Sarajevo David Barth, Mission Director Steven Major, Program Office Director Elma Bukvic Jusic, Development Assistance Specialist (M&E) 40 Ankica Gavrilovic-Altumbabic, USAID Project Management Specialist Amira Vejzagic-Ramhorst, EDO Acting Director Ministry of Foreign Trade and Economic Relations Reuf Hadjibegic, Assistant Minister, Sector of Natural Resources, Energy and Environmental Protection Mubera Bicakcic, Head of Energy Department USAID Washington Bob Archer, USAID Office of Economic Growth, Energy Team Leader European Union Sarajevo Office Amila Ibricevic Office of the High Representative Enida Hadzimuratovic EP Federation of BiH Mirsad Sabanovic, Executive Director, Supply and Trade of Electricity Mouris Dlakic Meliha Dzizic EP Trebinje Branislava Milekic, CEO Dalibor Muratovic, Head of Directorate Blagoje Supic, Executive Director for Economic/Financial Affairs Jovo Maric, Executive Director for Technical Affairs B. Branislav Koralika Kovacevic-Markov, Lead Engineer Nikola Tesanovic, Expert Associate Jagoda Lazetic, Head of Department AleksandarVukoje, Head of Legal Affairs RSERC, Trebinje Milan Kovacina Zoran Elek, Engineer Aleksandar Jegpic, Secretary and Chief of Staff Mladen Staka Nada Ceranic, Head of Market and Tariffs Department Gordana Zelenovic, Tariffs Petar Munisic Petar Duka, Engineer Milenko Cokorico FERC Mostar Sanela Pokrajcic, President of Commission Ivica Dreznjak, Secretary and Chief of Staff Risto Mandrapa, Commissioner Djulizara Hadjimustafic, Commissioner Slavica Stojcic, Head of Legal Department Julia….., Commissioner 41 ANNEX 3: DOCUMENTS REVIEWED Review of Bosnia and Herzegovina Energy Situation and USAID Assistance, November 17, 2009 PBMP – March 11, 2010 to September 30, 2013 Action Plan of the Federation of Bosnia and Herzegovina for the Restructuring and Privatization of the Electricity Sector in Bosnia and Herzegovina, 2002 REAP Activity Timetable, 2007–2011 REAP PBMP October 2011–September 2012 Project Work Plans, March 2007–March 2011 Reform Implementation Schedule, 2008–2013 Summary Implementation Schedule, 2008–2013 Action Plan of the Republika Srpska of Bosnia and Herzegovina for Restructuring and Privatization of Electricity Sector in Bosnia and Herzegovina, June 2006 Country Development Cooperation Strategy (CDCS) for Bosnia and Herzegovina 2012–2016 REAP Quarterly Reports, June 2007 to May 2013 REAP Contract and Modifications Original contract, March 2007 Modification, March 2011 Modification, April 2013 Legislation FBiH Electric Power Law, April 2002 ISO Law, published in Official Gazette of BiH, July 29, 2004 RS Energy Law, 15-04-09 42 Transco Law published in Official Gazette, July 29, 2004 RS Law on Electricity, October 2002 Law on Amendments to the Law on Electricity, published in Official Gazette of Federation of Bosnia and Herzegovina, September 28, 2009 Laws on Amendments to the Law on Electricity, published in the Official Gazette of RS, April 8, 2009; September 24, 2009; December 21, 2010 Amendments to the Law Establishing the Company for the Transmission of Electric Power in Bosnia and Herzegovina, published in Official Gazette of Federation of Bosnia and Herzegovina, September 28, 2009 Law on Amendments to the Law on Transmission of Electric Power, Regulator, and System Operator of Bosnia and Herzegovina, published in Official Gazette of Federation of Bosnia and Herzegovina, September 28, 2009 Law on Transmission of Electric Power, Regulator and System Operator of Bosnia and Herzegovina, published in Official Gazette of BiH, April 10, 2002 Law on Modifications and Amendments of the Law on Electricity, published in Official Gazette of Federation of Bosnia and Herzegovina, June 23, 2005 Law on Amendments to the Law on Transmission of Electric Power, Regulator and System Operator of Bosnia And Herzegovina, published in Official Gazette of Federation of Bosnia and Herzegovina, May 19, 2005 Law on Amendments to the Law on Transmission of Electric Power, Regulator and System Operator of Bosnia and Herzegovina, published in Official Gazette of BiH, January 10, 2011 List of USAID REAP Major Deliverables for the period March 2007 – May 2011 (pre-extension period) This includes a number of documents/notes presentations classified by task area: 1. Under “Transformation of ISO into a full MO after the Transition Phases of the Internal BiH Market”: o Draft Letter of Authorization from MOFTER to ISO o REAP presentation, “Why the Market” o REAP paper on Electricity Market o REAP pamphlet on Electricity Market o REAP presentation on experience from Hungarian Market o REAP presentation on experience from Romanian Market 43 o REAP presentation on experience from Slovenian Market o Half-Page Summary of Third Liberalization Package o Table Showing Status of Reform: Transition o Suggestion to Move Electricity Sector Forward o Draft Statement on Reform in Electricity Sector o Presentation entitled “An Idea” regarding the formation of trader in Bosnia and Herzegovina o REAP Comments on Regional Draft Market Design o Memo on Rules and Connections o Draft Complaint and Request to SERC on Initiation of Automatic Measurement for ISO o Memorandum Markovic to Wilson regarding Connection o Memorandum Wilson to USAID re: Status of Electricity Sector o Memorandum Markovic to USAID re: EC presentation on Market o Memorandum Wilson to ISO General Manager on Regulatory Strategy o Legal Memo REAP to USAID tender on automatic measurement system o REAP Memorandum to RS Ministry re: telecommunications o Paper on Congestion Management o Transco proposal technical analysis table for distribution o Note on automatic metering system by Ognjen Markovic o Research paper on market design o REAP Comments on SERC draft Connection rules o Note on data re: countries with multiple TSOs o Present situation with regard to the SERC Draft Decision on the Ancillary Services o Report on ISO management board public meeting o REAP brief on Federation Rules on Connection o Analysis of DG TREN consultation document on ITC Mechanism and tariffs o JW Notes on DG TREN Gas Working Group meeting o Road map for implementation of directive on security of supply o Memorandum re: status of network and TSO o Only impediments and preconditions for successful elect. Market functioning final o Comparative Market Opening Table I o Comparative Market Opening Table II o REAP Comparative Table on the proposed rule on Capacity Allocation o REAP Comparative Table on Capacity Allocation Rules o Explicit Auction procedure o Draft REAP comments on Dokic proposed amendments to the Rule on Marketing Opening o REAP proposed rule provisions o Explicit auction procedure o Comparative table Capacity Allocation Rules o Memorandum Markovic to Wilson need for market operator and auctions (congestion management) o Diagram Measurement for System Balancing 44 o Memorandum from Markovic to Wilson on the need for a market operator and auctions o Summary table on who does what in the proposed capacity allocation rule o Rulebook on Capacity Allocation, redline and clean o Annex 1 to the Rulebook on Capacity Allocation (Statement of Acceptance) o Annex 2 to the Rulebook on Capacity Allocation (Guarantee of Payment) o Table who does what in the capacity allocation rule o ISO Redraft of Rulebook on Capacity Allocation redline o REAP redlined comments on ISO Redrafted Rulebook on Capacity Allocation o ISO final Redraft of Rulebook on Capacity Allocation redlined o Draft Contract for use of interconnect o REAP GANTT Schedule for capacity allocation o Legal Issues to the Establishment of the coordinated auction office in South Eastern Europe o REAP legal notes on congestion management o Memorandum Pierce Atwood to REAP BiH System Market operator o Markovic note on Dokic and Salkic documents on capacity allocation o REAP paper on congestion management, amended and simplified for distribution 2. Under “Unification of the Regulators”: o Note on REAP Approach and Activities to Preserve FERC’s Role in FBiH Power Sector o Memorandum Wilson to USAID on Proposed Federation Legislation o Memorandum Markovic to Wilson regarding FERC and Supply o REAP comments on RSERC Connection Rule o REAP comments of FERC eligible customer technical hearing o REAP final comments on RSERC Rule on qualified producers o Report from SERC hearing on Tariff Methodology Amendments o REAP Comments on SERC Draft amendments to rule on tariff methodology o Presentation on Advanced Excel 2008 by Navigant o Tariff rates for JP EPHZHB model o REAP to FERC memo on power balance Art 12 Tariff Methodology 3. Under “Regulators and ISO Efficiently Communicate with Decision Makers and General Public” o Presentation to the ISO Board o Article in Oslobođenje on ISO opening ceremony o REAP to ISO PR, draft ISO “Who We Are” pamphlet o REAP Media & PR Training – Developing Reputation o REAP Media & PR Training – Communication o REAP Media & PR Training – PR – persuasion o REAP Media & PR Training – Public Relations o REAP Media & PR Training – Radio o REAP Media & PR Training – Crisis Management o REAP Media & PR Training – Television 45 4. Under “Decision Makers are Acquainted with Energy Reform”: o REAP handout for Parliamentarians o REAP handout for Parliamentarians o Activity Plane for the Federation Parliament training/presentations o MOFTER Parliamentary presentation edited by REAP o FERC Parliamentary presentation edited by REAP o Presentation by REAP on internal Action Plan o Success story on sector presentation in FBiH Parliament o Success story on PR Training o Success story on Parliamentarians and draft fed. Eneg. Law o Pamphlet “Who We Are” o Draft Parliament Conclusions for the presentations o RSNA sector presentation, Transco presentation – Mijatovic o RSNA sector presentation, Transco presentation – Mijatovic, new slide o RSNA sector presentation, ISO presentation o RSNA sector presentation and RS Reaction o Presentation to the State Parliament o Report on Energy Sector Presentation in BiH Parliament 5. Under “NGOs Have Knowledge to Support Proposed and Effected Reform”: o REAP to NGO Center for Ecology and Energy, comments on commentary of FBiH Strategic plan o REAP article published in Fondeko on climate and energy o REAP Report from Sarajevo NGO Conference o Presentation to Belgrade NGO Conference o Report on participation on Belgrade Conference 6. Under “Media Outlets Consistently, Accurately, and Comprehensively Inform Public on Energy Reform”: o First Article resulting from REAP influence o Draft II Energy Sector Facts and Messages o Press Release o Radio Show produced by AlterMedia Radio Network o REAP Scrapbook on USAID REAP Conference in Neum o Prism Survey Results of Public Awareness of Electricity Sector o Press Release o Note on Public Opinion Survey o Press Release o Paid Advertisement Draft “Who Determines Electricity Price” o Press Release on upcoming gas regulatory training o Paid Ad re: who determines the price of electricity o Paid Ad in Dnevni Avaz re: who determines the price of electricity o Paid Ad in Dnevni List re: who determines the price of electricity o Paid Ad in Nezavisne novine re: who determines the price of electricity 46 List of USAID REAP Major Deliverables for the period May 2011 – present (post-extension period) This includes a number of documents/notes/presentations classified by task area: 1. Under “Harmonize regulatory work of the three commissions to enhance performance of the market”: o Presentation for the Joint Regulators’ Meeting o Memorandum Wilson to Lieberman on FERC Tariffs o Possible Pricing Rules for FERC Default Supplier Mechanism o Comments to RERS Eligible Customer Rule o Comments on FERC Eligible Customer Rule o Activity Plan for Regulators o Case Study Presentation – Making and Terminating Contracts o Case Study for Technical and regulatory preparation for market opening 2. Under “Technical and regulatory preparation for market opening” o Presentation on Case Study o REAP Scope of Work for Case Study o Time Schedule for the Case Study o Case Study Metering o Case Study Recent Developments o Case Study Presentation – Framework for Retail Market o Case Study Presentation – Switching o Case Study Presentation – Metering Process o Case Study Presentation – Balancing Settlement Process o Case Study Presentation – Customer and Supplier Information o Case Study Presentation – Billing o Case Study Presentation – Concluding Remarks o Case Study Presentation – Licensing o Paper on Metering Process o Paper on Scheduling Process o General Conclusions from Gap Analysis o BiH Electricity Market Gap Analysis – Master Version o Presentations from Case Study Workshop on Gap Analysis o Presentation from Case Study Workshop on Hierarchy of Documents o Presentation from Case Study Workshop on Conclusions o Draft Scope of Work for Load Research o Time Schedule to Perform Load Research Technical Assistance o Load Research Kick-Off Meeting Presentation o Sample Selection Outline for EPs – First Step o Load Research Sampling Presentation o Load Research Questionnaire Presentation o Draft Load Research Questionnaire 47 o Load Research Manual o Load Research Guidelines o Load Research Balancing Settlement o Load Research Recent Developments o Presentation on Load Research – Sample Selection o Presentation on Load Research – Sample Implementation o Presentation on Load Research – Data Transfer o Presentation on Load Research – Data Analysis o Presentation on Load Research – Data Processing o Presentation on Load Research – How to Calculate o Presentations from LR Workshop – Portuguese Experience o Presentations from LR Workshop – Analysis of Guidelines o Presentations from LR Workshop – Analysis of EP Reports o Presentations from LR Workshop – International Practice o Presentations from LR Workshop – Load Research Guidelines o Presentations from LR Workshop – Load Research Process o Presentations from LR Workshop – Conclusions o Presentations from LR Workshop – Using Load Profiles o Speech in FBiH Parliament o Presentation LR Meeting – Influence of Small-Consumption Customers o Presentation LR Meeting – Where to Use LR Results o Minutes from LR Meeting in Tuzla o Conclusion from LR Meeting in Tuzla o REAP Subgroup Proposal for Duration of Transitional Period o Presentation for Regulators on Finalization of Transition Period o REAP Research on Default Supplier o Transition Period Paper with Attachments o Framework for the Development of the Action Plan for Wholesale Market o Paper on Transitional Period and Default Supplier o Consolidated Paper on Default Supplier Market Opening o Memorandum to Ms. Gavrilović regarding the Default Supplier o Memorandum REAP to USAID on the Federation Vulnerable Customer Program o Memorandum REAP to MOFTER on World Bank Wholesale Study o REAP Comments on the Federation Electricity Law o Memorandum Changes Needed in Federation Electricity Law 3. Under “Strengthen the State Government’s role in the energy sector to facilitate BiH speaking with one voice in the internal and international liberalization of the energy sector” o Presentation – Summary Third Liberalization Package o REAP Analysis of Regional Action Plan and National Action Plan o Two Question for MOFTER for the Athens Forum o Memorandum USAID on Transco on 110 network re-conveyance o Memorandum to USAID – Facts on Transco o Memorandum on Report on Implementation of the Acquis o Conclusions from 18th Athens Forum 48 o 10th Ministerial Council Meeting Decisions o Report from 8th Annual EU Energy Law Conference o MOFTER Task List o Memorandum REAP to MOFTER regarding Ministerial Council Decisions Websites consulted State Electricity Regulatory Commission (SERC) – http://www.derk.ba/en Regulatory Commission for Electricity in Federation of Bosnia and Herzegovina – http://www.ferk.ba Regulatory Commission for Energy of Republic of Srpska – http://www.reers.ba/en Energy Community Regulatory Board (ECRB) – http://www.energy-community.org/portal/page/portal/ENC_HOME 49 ANNEX 4: EVALUATION METHODOLOGY AND LIMITATIONS PARTICIPANT SELECTION AND SELECTION BIAS Key informant interviewees were selected non-randomly based on availability and engagement with program activities. As there is a small universe of stakeholders for this activity it was not possible to randomly select interviewees. The initial list of important people and institutions and individuals came from USAID, with some addition from the contract implementer. The team has held interviews with every organization that was a recipient or object of the REAP project, plus several whose own interests and work coincide with the REAP project. These organizations included Ministries, regulators, energy companies, and donors—organizations with highly divergent viewpoints. However, most interviewed beneficiaries were direct participants in the project and were recommended by either USAID or the contractor, so, as expected, they said positive things about the project. There was a problem finding people who could be simply classified as observers, mainly because the nature of the product (e.g., power sector legislation and regulations) is such that it was hard to find a neutral, detached party who would have enough interest in the subject matter to have formed a strong opinion. In order to minimize the bias imposed by the small number of organizations and persons directly involved, we included several groups not on the USAID list who represent “public stakeholders” with a different “causal distances” from the activities (an NGO and a journalist as well as two donors) to qualitatively triangulate/check information from beneficiary organizations and who cannot be perceived as “friendly” or biased stakeholders but who have sufficient knowledge to comment. Selection bias in this case was mitigated partly by the tangible nature of the results themselves, e.g., the fact that legislation and regulations were drafted, trading entities were established, and certain progress has been towards meeting the obligations of the ECT and subsequent EU Directives. A final check on potential selection bias is the comparison of the very real tangible results with expectations of the contract through the deliverables and indicators. Additionally, each institution was free to invite whom they wished to participate in the interviews. In all but a few cases, there was representation from both top and senior-level persons who had direct exposure to the project and the TA and who were able to comment on it. Numbers of participants at interviews ranged from two to nine (in addition to the three evaluation team members). The respondents represented those who participated in trainings, seminars, meetings, and other project activities; however, we noted that more junior staff members often did not speak up in interview sessions, leaving the most senior person present to “speak for the group.” Of the 52 respondents, 19 were women. This breakdown may accurately represent the male gender predominance in high positions in the electricity sector. The team was not able to 50 control for gender parity in meetings, as the criteria for interview participation was direct experience with the REAP program of TA. RECALL BIAS To minimize recall bias, the interview questions were designed to address specific activities/skills/knowledge that the organization or the individual acquired from the technical assistance and training, and how this specifically was applied by the organization. Recall bias was minimal as the project was actually still being carried out at the time of the interviews, although in its very final stages. Also, all interviewees were very much involved in the REAP tasks. However, even with repeated questioning, it was difficult to get interviewees to cite specific examples such as which particular case studies, reports, or trainings were of value that would enable the evaluators to present numerical data to substantiate broad statements. Interviewees preferred to keep their answers general. INTERVIEWEE RESPONSE BIAS All participants come with biases, especially in BiH with its political and ethnic divisions. Our sample includes organizations with different ethnic and political points of view, as they play different roles within the energy space. As expected, there were diverse perspectives, especially concerning the success of preparation for market entry, the appropriateness of roles of different players, and the cause of blockages to moving forward on energy integration, where ethnic disagreements over the structure and roles of state and entity institutions in BiH have turned the energy issue into a political tactic. Some respondents avoided directly addressing these issues, citing “politics” as the problem with no further specification; others expressed a very clear point of view, based to a large extent on their ethnic/entity identification, with some finger-pointing at other institutions responsible for failures. In the case of two organizations in particular, there was extreme bias in the responses (against USAID and the USG) and in one case the interview was curtailed, both based on political agendas. Where significant agreement was found in responses we can safely assume there is no bias. An undoubtedly larger problem that is difficult to assess is that of deliberately inaccurate responses because the respondent wants to maintain a good relationship with the implementer and USAID. All the organizations interviewed would like the continuation of technical assistance and hence may be reluctant to provide any negative responses. In such a case, a respondent might feel compelled to not say anything detrimental in order not to jeopardize the chances of receiving similar help in the future. A related issue was bias by organizations towards what they found valuable and useful to them and downplaying the other aspects of the project assistance. This shortcoming was mitigated through triangulation of what was said by different organizations. QUESTION CHANGES Questions were asked in roughly the same order and following the original interview protocols throughout. However, we discovered that questions pertaining to Evaluation Question 3 were 51 poorly understood and did not elicit coherent responses. After discussion with USAID staff, we altered the wording of the question. Nonetheless, participants were never able to clearly respond to this question, and the implementer claimed it was not relevant to its scope of work. Additionally, over the course of the interview weeks, we eliminated several redundant questions in which the subtle differences between them were not perceived by the interviewees, which resulted in confusion. In a few cases we skipped over a number of questions because the political atmosphere was highly charged and given previous replies, the remaining questions became irrelevant. ABILITY TO GENERALIZE FROM THE DATA The method of organizing the data by question and respondent in an Excel workbook allowed for the grouping of responses to questions according to recurrent themes and opinion agreements and disagreements. This has allowed us to develop the areas where the data clearly justify generalizations. Where possible, we have quantified the responses, although the quantification is subject to interpretation since the respondents’ answers were often imprecise. (These were responses to “open-ended questions” that were mainly opinion-related.) The development of recurring themes and concurrence across different types of respondents has therefore provided the best source of data from which to formulate findings and conclusions that are supported by specific examples and quotes. The ten major themes are provided below. Theme #1: Mixed perceptions by stakeholders of the progress on meeting Treaty obligations. Theme # 2: TA as a major support to writing legislation, by-laws and secondary legislation, and preparing for market regulations. Theme #3: The importance of the implementers as facilitators (bringing people together from all institutions to find common approaches in a non-judgmental and impartial atmosphere). Theme# 4: Increased understanding of market functioning, the roles of regulators, and defining specific regulations to make markets work. Theme# 5: Major political and structural obstacles (that are due to the ethnic/political conflicts over the role of the RS and Federation institutions and the role of state institutions) qualified as “politics.” Theme # 6: Low level of public understanding of, and interest in, energy issues, as well as fears about future price rises. Theme #7: The effects of market opening on energy prices. Theme# 8: Lack of progress on unified national and entity energy-investment strategies and action planning as a brake on investment in energy generation and the effects on regional integration. 52 Theme# 9: January 2015 deadline and the ability to meet remaining obligations by that time. Theme# 10: Bottom-up and practical approach of REAP assistance compared to EU’s static and bureaucratic approach. 53 ANNEX 5: REAP PERFORMANCE BASED MANAGEMENT PLAN (PBMP) Indicator 1 Number of energy agencies, regulatory bodies, utilities, and civil society organizations undertaking capacity-strengthening as a result of USG assistance Number of energy agencies, regulatory bodies, utilities, and civil society organizations Baseline 2011 2012 2013 Total Unit of Measurement: Number of institutions Method of Data Collection/Calculation: Counting Target 9 9 12 12 Indicator Description: The number of energy and concerned agencies, utilities, etc. where USG programs are directly supporting strengthening of institutional capabilities. Data Source: Implementing partners, Ministries of Energy, energy regulators, energy companies/utilities Data Quality Issues: N/A Schedule/Frequency of Data Collection: Cumulatively for Quarterly Reports Data Analysis Plan: N/A Responsible Person: COP Report Date Responsible Entity Quarterly Every quarter REAP team Comments: In the total, REAP has not double-counted institutions; however, it is dealing with at least the regulators, ministries, and EPs on a yearly basis. 54 Indicator 2 Number of people receiving USG-supported training in energy-related policy and regulatory practices Number of people receiving USG supported training Baseline 2011 2012 2013 Total Unit of Measurement: Number of people Method of Data Collection/Calculation: Counting Target 30 30 50 110 Indicator Description: Number of public/private professionals that are provided with USG-supported training (formal coursework, on-the-job, seminars) on issues concerning energy policy and regulation. Data Source: Implementing partners, Ministries of Energy, energy regulators, energy companies/utilities Data Quality Issues: N/A Schedule/Frequency of Data Collection: Cumulatively for Quarterly Reports Data Analysis Plan: N/A Responsible Person: COP Report Date Responsible Entity Quarterly Every quarter REAP team Comments: 55 Indicator 3 Number of people receiving USG-supported training in technical energy fields Number of people receiving USG supported training Baseline 2011 2012 2013 Total Unit of Measurement: Number of people Method of Data Collection/Calculation: Counting Target 20 20 20 60 Indicator Description: Number of public/private professionals that are provided with USG supported training (formal coursework, on-the-job, seminars) on technical areas in the energy sector. Data Source: Implementing partners, Ministries of Energy, energy regulators, energy companies/utilities Data Quality Issues: N/A Schedule/Frequency of Data Collection: Cumulatively for Quarterly Reports Data Analysis Plan: N/A Responsible Person: COP Report Date Responsible Entity Quarterly Every quarter REAP team Comments: 56 Indicator 4 Number of policy reforms/regulations/administrative procedures analyzed to enhance sector governance and/or facilitate private sector participation and competitive markets as a result of USG assistance Number of policy reforms / regulations / administrative procedures Baseline 2011 2012 2013 Total Unit of Measurement: Number of policies Method of Data Collection/Calculation: Counting Target 20 10 10 40 Indicator Description: The number of reform measures, regulations, rules and procedures, etc. that have been analyzed as part of a USG program with the objective of improving overall sector governance, the enabling environment, private sector investment and management, competition within the sector, etc. Data Source: Implementing partners, Ministries of Energy, energy regulators, energy companies/utilities Data Quality Issues: N/A Schedule/Frequency of Data Collection: Cumulatively for Quarterly Reports Data Analysis Plan: N/A Responsible Person: COP Report Date Responsible Entity Quarterly Every quarter REAP team Comments: 57 Indicator 5 Number of legal, regulatory, or institutional actions taken to improve implementation or compliance with international trade and investment agreements due to support from USG￾assisted organizations Number of legal, regulatory, or institutional actions Baseline 2011 2012 2013 Total Unit of Measurement: Number of actions Method of Data Collection/Calculation: Counting Target 5 5 5 15 Indicator Description: Number of legal, regulatory, or institutional actions taken to improve implementation or compliance with international trade and investment agreements partly or fully due to analysis or advocacy by USG-assisted organizations. Data Source: Implementing partners, Ministries of Energy, energy regulators, energy companies/utilities Data Quality Issues: N/A Schedule/Frequency of Data Collection: Cumulatively for Quarterly Reports Data Analysis Plan: N/A Responsible Person: COP Report Date Responsible Entity Quarterly Every quarter REAP team Comments: 58 Indicator 6 Number of USG-supported training events held that related to improving the trade and investment environment Number of USG supported training events held Baseline 2011 2012 2013 Total Unit of Measurement: Number of trainings Method of Data Collection/Calculation: Counting Target 1 1 1 3 Indicator Description: Number of USG-supported events held that provided training on topics related to improving the trade and investment environment. Data Source: Implementing partners, Ministries of Energy, energy regulators, energy companies/utilities Data Quality Issues: N/A Schedule/Frequency of Data Collection: Cumulatively for Quarterly Reports Data Analysis Plan: N/A Responsible Person: COP Report Date Responsible Entity Quarterly Every quarter REAP team Comments: 59 Indicator 7 Number of participants in trade and investment environment trainings Number of participants Baseline 2011 2012 2013 Total Unit of Measurement: Number of participants Method of Data Collection/Calculation: Counting Target 30 30 50 110 Indicator Description: Number of participants trained on improving the trade and investment environment. Data Source: Implementing partners, Ministries of Energy, energy regulators, energy companies/utilities Data Quality Issues: N/A Schedule/Frequency of Data Collection: Cumulatively for Quarterly Reports Data Analysis Plan: N/A Responsible Person: COP Report Date Responsible Entity Quarterly Every quarter REAP team Comments: 60 USAID BOSNIA AND HERZEGOVINA ROBERTA C. FRASUREA 1 71000 SARAJEVO, BIH TELEPHONE: +387 FAX: +387 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523