EVALUATION FINAL PERFORMANCE EVALUATION OF THE USAID/ARMENIA MARKET LIBERALIZATION AND ELECTRICITY TRADE (MLET) PROGRAM SUBMITTED: MARCH 26, 2023, REVISION 1: JUNE 2, 2023 This document was produced for review by the United States Agency for International Development. It was prepared by International Development Group LLC under the USAID Technical Assistance Project for Economic Growth (TAP EG). FINAL PERFORMANCE EVALUATION OF THE USAID/ARMENIA MARKET LIBERALIZATION AND ELECTRICITY TRADE (MLET) PROGRAM DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Cover photo of Yerevan by Levon Vardanyanon on Unsplash. Activity Title: Final Performance Evaluation of the USAID Market Liberalization and Electricity Trade (MLET) Activity Sponsoring USAID Office: USAID/Armenia Under Contract: Technical Assistance Project for Economic Growth (TAP EG) Contract Number: 47QRAD20DD1072/7200AAN00006 Contractor: International Development Group LLC (IDG) Submission Date: March 27, 2023 Revision 1: June 2, 2023 ABSTRACT iii ABSTRACT The USAID Market Liberalization and Electricity Trade (MLET) is a five-year activity implemented by Tetra Tech ES, Inc. aimed at improving the electricity market and supporting electricity trade with Georgia by helping the Government of Armenia (GOAM) and energy institutions adopt legal-regulatory reforms, accelerate implementation of market liberalization mechanisms, and strengthen Armenia￾Georgia dialogue on cross-border trade. This final performance evaluation of the activity, conducted by the Technical Assistance Project for Economic Growth (TAP EG) and implemented by International Development Group LLC (IDG), focuses on five evaluation areas of inquiry: 1) coherence, 2) effectiveness, 3) efficiency, 4) impact, and 5) sustainability. Data collection efforts, including 23 key informant interviews with 38 stakeholders and a focus group discussion, were conducted between November 2022 and March 2023. MLET has been a largely successful activity, as it will have completed almost all planned tasks by mid￾2023 and made significant gains in its three component areas of electricity market development, energy supply diversification, and promotion of cross-border electricity trade with Georgia. The range of achievements is impressive as MLET has provided extensive support in restructuring the electricity market in Armenia, increased planning capability, and has played a significant role in increasing electricity trade with Georgia. USAID stands out as a reliable, credible, and trusted partner in assisting Armenia to move forward with its power sector reform agenda as it strives to meet requirements to participate in the European electricity market by 2029. It is the only donor addressing the Government of Armenia’s energy priorities in a significant and practical way. The activity’s accomplishments have been noted together with several areas where outcomes might have been better, some of which were outside of MLET’s control. This includes the construction delay of a crucial new high voltage transmission line and back-to-back link with Georgia. This, in turn, resulted in cross-border trade benefits not materializing, which would have otherwise occurred in the presence of the link. However, it must be noted that even without the link, cross-border trade has increased substantially in 2022-2023 over previous years. Certain methodological weaknesses were observed in the development of a least-cost energy development plan and a ten-year network development plan for the power system, which were partly beyond the mandate of MLET. However, due to the comprehensiveness of the analyses, the overall effects of these weaknesses on the results were likely minimal. While MLET made substantial progress towards the ultimate goals of market liberalization, movement to more diverse sources of energy supply and increased electricity trade with Georgia, several areas for further work remain and continued support from USAID can capitalize on the momentum that MLET has generated. CONTENTS iv CONTENTS ABSTRACT III ACRONYM LIST VI EXECUTIVE SUMMARY I Evaluation Purpose and Evaluation Questions i Findings and Conclusions i Recommendations v Lessons Learned vii PURPOSE OF EVALUATION AND EVALUATION QUESTIONS 1 Purpose of Evaluation 1 Evaluation Questions 1 How the Report Is Organized 2 BACKGROUND 3 Overview of The Armenian Power Sector 3 EVALUATION DESIGN AND METHODOLOGY 6 Methodology 6 Qualitative Research and Analysis 6 Quantitative Analysis 6 Evaluation Activities 6 Evaluation Limitations 7 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 8 Evaluation Area 1—Coherence 8 Evaluation Area 2—Effectiveness 10 Evaluation Area 3—Efficiency 22 Evaluation Area 4—Impact 23 Evaluation Area 5—Sustainability 27 LESSONS LEARNED 31 Task 1 31 Task 2 31 Task 3 31 ANNEXES 32 Annex I: Timeline 33 Annex II: Evaluation Statement of Work 34 Annex III: Evaluation Methods and Limitations 40 Annex IV: Data Collection and Analysis Tools 45 Annex V: Sources of Information 50 Annex VI: Disclosure of any Company Interest 51 Annex VII: Evaluation Team Members 63 Annex VIII: Statements of Difference 67 Annex IX: Documents Reviewed 68 CONTENTS v LIST OF FIGURES Figure 1. MLET Evaluation Areas 2 Figure 2. Key informant interviews 6 Figure 3. Armenian Electricity Market Development (% of Sales) 24 ACRONYM LI ST vi ACRONYM LIST AAF Annual Adequacy Forecast AEX Armenia Electricity Exchange ANPP Armenian Nuclear Power Plant ANRA Committee on Nuclear Safety Regulation CEPA Comprehensive and Enhanced Partnership Agreement DO Development Objective DSO Distribution System Operator EAEU Eurasian Economic Union EE Energy Efficiency ENA Electric Networks of Armenia ENTSO European Network of Transmission System Operators for Electricity EPSO Electric Power System Operator EQ Evaluation Question ESDSP Energy Sector Development Strategic Program ET Evaluation Team EU European Union FGD Focus Group Discussion GOAM Government of Armenia HHI Herfindahl-Hirschman Index IEA International Energy Association JWG Joint Working Group KI Key Informant LCEDP Least Cost Energy Development Plan MEINR Ministry of Energy Infrastructure and Natural Resources MLET Market Liberalization and Electricity Trade MMS Market Management System MO Market Operator MOU Memorandum of Understanding MTAI Ministry of Territorial Administration and Infrastructure MW Megawatts PPA Power Purchase Agreement PSRC Public Service Regulatory Commission RE Renewable Energy SC Settlement Center CJSC SCADA Supervisory Control and Data Acquisition SOW Statement of Work TAP EG Technical Assistance Project for Economic Growth TYNDP Ten-Year Network Development Plan US United States USAID United States Agency for International Development WEM Wholesale Electricity Market EXECUTIVE SUMMARY i EXECUTIVE SUMMARY The USAID Market Liberalization and Electricity Trade (MLET) is a five-year activity implemented by Tetra Tech ES, Inc. aimed to improve the electricity market and support electricity trade with Georgia by helping the Government of Armenia (GOAM) and energy institutions to adopt legal-regulatory reforms, accelerate the implementation of market liberalization mechanisms, and strengthen Armenia-Georgia dialogue on cross-border trade. MLET activities are organized under three tasks: i) support energy market development; ii) promote energy supply diversification; iii) facilitate cross-border trade. The activity builds on USAID’s work over the past decade in Armenia to facilitate the unbundling of the electricity and gas sectors, creation of an independent regulator, penetration of substantial renewable energy (RE) resources, large-scale privatization of electricity generation and distribution assets, and improving the quality of services. GOAM passed legislation in 2017 to drive greater liberalization of the market. With assistance from the MLET activity, changes under the law are currently being phased in, with a hybrid model operating during a transition period, whereby existing power purchase agreements are continuing alongside a competitive wholesale market and a regulated market. The intention is to introduce competition among electricity suppliers, which is meant to break up utility control over distribution throughout Armenia. Since the MLET activity started, Armenia has made considerable progress in enhancing regional market integration. The country signed and ratified the Comprehensive and Enhanced Partnership Agreement (CEPA) with the European Union (EU) in 2021 that includes a timetable for the alignment of Armenian laws and regulations to relevant EU laws over the next few years, and by 2029 at the latest. EVALUATION PURPOSE AND EVALUATION QUESTIONS USAID/Armenia engaged the Technical Assistance Project for Economic Growth (TAP EG) to conduct an independent final performance evaluation of the MLET activity to: (i) inform USAID of the degree to which the MLET activity, implemented by Tetra Tech ES, Inc., achieved its intended goal and objectives; and (ii) capture key lessons to inform USAID/Armenia’s design or implementation of future activities. This final performance evaluation focuses on five evaluation areas: i) coherence, ii) effectiveness, iii) efficiency, iv) impact, and v) sustainability. FINDINGS AND CONCLUSIONS Evaluation Area 1—Coherence The three tasks of the MLET SOW align perfectly with the USAID definition of energy security. The three main tasks of the MLET Statement of Work (SOW) are:—(i) Energy Market Development, (ii) Promote Energy Supply Diversification, and (iii) Facilitate Cross-Border Trade.—These tasks dovetail with the definition of energy security used by USAID/Armenia; that is, market liberalization that will support investments, supply diversification, and integration with Georgia. Thus, the tasks are relevant and contribute very closely to USAID’s strategic intermediate result “energy security increased.” MLET has contributed significantly to all five strategic areas of GOAM’s efforts to increase energy security in Armenia. The GOAM Energy Sector Development Strategic Program (ESDSP) to the year 2040 cites five priorities: i) development of RE potential, ii) energy efficiency (EE) across all sectors, iii) nuclear power, iv) transmission interconnections, and v) gradual electricity market liberalization. An examination of the detailed subtasks revealed that EXECUTIVE SUMMARY ii MLET: i) assisted in the preparation of regulations pertaining to net billing for distributed RE sources and provided transaction advisory support for managing the procurement process of the Ayg-1 200 megawatt (MW) solar project; ii) prepared an Energy Efficiency Gap Analysis Report; iii) aided in the assessment of various nuclear power generation scenarios; iv) provided support for cross-border trade through assistance to the Joint Working Group on trade with Georgia; and v) led GOAM efforts to design and implement a transitory Wholesale Energy Market (WEM) that will eventually align with EU requirements. Evaluation Area 2—Effectiveness MLET has been very effective from a beneficiary-KI perspective, with eight of nine beneficiary-KIs giving scores of 9 or 10 (10 being the best) to MLET effectiveness, satisfaction, and contractor performance. Thus, it may be concluded that almost all KIs were extremely satisfied with MLET and that the activity was very effective from a beneficiary-KI perspective. MLET has been effective in supporting legal and regulatory reforms of the Armenian electricity sector. MLET provided input for legal and regulatory reforms toward market liberalization by: i) drafting several strategic and analytical documents supporting market development and an effective electricity trading mechanism, ii) drafting several pieces of crucial secondary legislation, iii) assisting in the preparation of the new Electricity Law, iv) developing, procuring, and putting into operation the customized Armenian Electricity Exchange (AEX) Market Management System (MMS), v) providing extensive support to all energy sector entities participating in the new electricity market, vi) supporting procurement and establishment of the MMS control room and training center, and vii) providing extensive capacity-building and trainings to energy sector entities and MPs. Important gaps remain if Armenia is to arrive at a liberalized electricity market that functions in accordance with CEPA, specifically: i) full compliance with the commitments made under CEPA is yet to be achieved, ii) no clear path exists as to when and how the CEPA commitments are to be achieved— which is part of a greater governance issue within GOAM, due to a lack of capacity to design policies and assess their impacts through strategic planning exercises, as well as to implement them, and to monitor, assess, and adapt these policies in line with monitoring results, iii) although the MMS software serves the current and near-term Armenian electricity market well, it does not meet EU standards, which will be critical if trading begins with the EU, and iv) guides and pertinent reference material for the Electric Power System Operator (EPSO) and Settlement Center (SC)—to efficiently operate in an increasingly complex electricity market and in accordance with EU standards—are not readily available. Additionally, several functional areas of the new electricity market require some focus over the short term for market liberalization to proceed efficiently: i) the EPSO’s Supervisory Control and Data Acquisition System (SCADA)/ energy management system does not yet cover the whole Armenian power system, ii) a data gathering process for short￾term planning and Ten-Year Network Development Plan (TYNDP) development has not been formalized, which has implications on quality and credibility of the data used, iii) provision is yet to be made for certain market-based ancillary services that EPSO would be normally be expected to manage (for example, frequency control, voltage regulation) nor congestion management, and v) flexibility resources to help manage demand in an environment of increasing variable renewable energy (VRE) penetration are not yet present, but will likely become more critical as VRE adoption increases. The Transmission Network Code assigns to EPSO the responsibility to develop the TYNDP in cooperation with the system’s users. However, explicit regulatory approval is not required. This is not in compliance with EU directives. EXECUTIVE SUMMARY iii While MLET produced satisfactory results in supporting the least-cost energy development program (LCEDP) and TYNDP—developed for the energy sector and power network respectively— the credibility of their results for power system planning could be further enhanced with a more conventional approach that uses an optimization model to develop the TYNDP. MLET updated the LCEDP twice, which was an input to GOAM’s Energy Sector Development Strategic Program (ESDSP). In doing so, MLET helped upgrade the MARKAL/TIMES/VEDA energy model and provided necessary training to the staff of the Scientific Research Institute of Energy. MLET also aided in developing the TYNDP using an annual system simulation model, however, other models such as the Wien Automatic System Planning (WASP) or PLEXOS would have been more suitable because they are oriented towards producing an optimal generation expansion plan. The model used by MLET—MARKAL/TIMES/VEDA—is a higher-level model generally employed by governments as a planning tool for the whole energy sector, encompassing areas such as electricity, transportation and energy efficiencies across all sectors (for example, building construction). It has also been used to help in the setting of National Determination Contributions of carbon emissions prescribed under the Paris Agreement. It is generally not used as a power sector planning tool, but is evidently capable of allocating power sector resources in some “optimal” manner, albeit not under the criteria and methodological rigor that other, more power sector-oriented programs are designed to do. Additionally, an inherent limitation in the LCEDP is that it does not consider the 1,000 MW interconnection with Iran, which could pose a significant challenge as the peak demand of the Armenian system is about 2,000 MW. While Iran’s exclusion is in accordance with the U.S. government’s foreign policy stance, the absence of this input limits the technical validity of the results. However, this problem was likely overcome through the development and analyses of multiple scenarios to the extent that the implications of adding the 1,000 MW load to the system were sufficiently examined and addressed. MLET has been effective in increasing cross-border trade, although results are less than anticipated due to factors beyond MLET’s control. MLET organized regular meetings of the Armenia-Georgia Joint Working Group (JWG). Five such meetings were held over the course of MLET’s implementation in which cross-border trade issues were discussed. Efforts were not as effective as originally expected as progress on the long-awaited back-to-back 400 kV interconnection was slower than anticipated and the capacity to increase cross-border trade to anticipated levels was thus not in place. A higher level of transmission capacity could have led to more trade (with proper trading arrangements). Despite the transmission constraint, however, trade with Georgia increased significantly in 2022 along existing lower-capacity existing connections—from 242 gigawatt-hours (GWh) over the whole ten-year period 2012 to 2021 to 365 GWh in 2022. Preparedness for upcoming significantly higher levels of cross-border trade could be improved by undertaking additional measures, notably: i) development of fundamental market-based mechanisms for cross-border trade such as capacity allocations, ii) clearly defined congestion management procedures in Armenia conducted on an economic basis, iii) increased familiarity with the rules and procedures of the European Network for Transmission System Operators (ENTSO-E), iv) development of a comprehensive regulatory framework for cross-border trade, including specific responsibilities assigned to Armenia’s Public Service Regulatory Commission (PSRC), and v) more Armenian traders involved in all cross-border transactions (currently one is involved, despite the remarkable increase in cross-border trade). Several key factors that enabled and hindered achievement of MLET outcomes were identified. Identified enabling factors included: i) commitment to the program by all stakeholders, ii) development of the MMS trading platform, and iii) MLET’s ongoing EXECUTIVE SUMMARY iv availability, responsiveness, and support to energy entities and MPS throughout the project’s duration. Identified hindering factors were: i) GOAM’s reassessment of project support after the 2018 elections, where the newly elected government undertook an extensive review of the previous regime’s policies, slowing down progress during the review period, ii) a consistent slowness by GOAM to act on key issues, iii) COVID-19, and iv) the conflict with Azerbaijan. Two significant identified geo-economic and sector developments include: i) Russian economic influence in the Armenian power sector, and ii) GOAM elections in 2018, in which a new government was elected, resulting in restructured ministries and policy reviews on matters such as power sector reform. MLET was able to continue carrying out the project in both instances. Evaluation Area 3—Efficiency The MLET intervention delivered results in an economic and timely way. The project has been carried out in accordance with the originally envisaged timeline. In addition, resources were diverted to other ongoing activities from activity allocations that either did not materialize or were out of MLET’s control. Evaluation Area 4—Impact Main impacts perceived by interviewed stakeholders include: (i) “huge” impact on moving the market liberalization process forward; (ii) transparency, fairness (specifically in prices); (iii) fewer disputes. Potential negative impacts include excessive market concentration by a few players and higher prices in the longer-term. Key informant interviews yielded a variety of positive effects (listed here the in order of number of responses): (i) large/ “huge” impact in moving the market liberalization process forward, (ii) transparency, fairness (specifically in prices), (iii) fewer disputes, (iv) better prices, (v) new 1 Which would be in contrast to the stated positive effect of better prices by interviewed stakeholders. professional opportunities, (vi) greater awareness of electricity supply, (vii) better planning/ certainty, (viii) movement to “best international practice”, (ix) improved safety, (x) better power system discipline, (xi) more stimulating discussions on tariffs, (xii) remarkable increase in cross-border electricity trade. Stated negative effects included: (i) more “work”/ problems to resolve, and (ii) prices might increase. Other potential negative impacts identified by the evaluation team based on the desk review were: i) an exceedingly high market concentration by a few customers as well as ownership in the market of multiple facilities (that is, generation and distribution), which might lead to market abuses and (ii) the possibility of increased electricity prices dictated by market forces.1 MLET interventions moved forward Armenia’s desired energy sector reforms at a pace that otherwise would not have been possible. To the question of what would have happened without MLET’s technical assistance, none of the respondents in a position to answer this (USAID and Tetra Tech excluded) stated that the reforms would have continued at the same pace or quicker. While a few respondents stated that assistance in undertaking reforms probably would have come from elsewhere, the resulting reform would, at best, have been at a slower pace, and at worst, would have stopped. Evaluation Area 5—Sustainability The liberalized market is in its nascent stages and cannot be sustained without further steps, which include: (i) passage of the Electricity Law—an important step that will lay the legal foundation for compliance with CEPA; this has been delayed due to problems encountered in drafting the law by the GOAM working group originally entrusted with this task, (ii) ongoing evolution of the EU market design, a dynamic “moving target” that Armenia will eventually need to meet in the future, (iii) continuing development and/or conversion of existing MMS EXECUTIVE SUMMARY v software to EU-compatible standards, and (iv) development and implementation of effective market monitoring mechanisms. Potential risks, such as a change in government and international political risks due to Armenia’s geographic location, have not been a factor in sustainability of MLET results so far. The current government, elected in the midst of the 2018 Armenian parliamentary “revolution”, has been supportive of the reforms begun under the previous government, albeit after a period of intense scrutiny. The identified political risk is that GOAM’s “go￾slow” approach to market liberalization and its relatively close relationship with Russia (for example, through the Eurasian Economic Union), tempered with some public support, might somehow lead to abandonment of the power sector reform program. However, MLET has survived the 2018 revolution as well as the political unrest following the 2020 war with Azerbaijan—and GOAM’s approach to market liberalization so far seems to be working, given the high percentage of participation in the recent voluntary market opening. All key informants interviewed, including policymakers, sector entities, MPs, NGOs, and media expressed very strong support for the MLET activity. Identified possible spoilers (through interviews with the supporters) include GOAM technocrats and other vested interests in the power industry that could stand to lose because of market reforms. RECOMMENDATIONS The following is a summary of recommendations, elaborated in the report. For the liberalized electricity market Implement the Action Plan provided in the Electricity Market Gap Analysis to achieve full compliance with the commitments under CEPA, to shift from current status (partial compliance) to full compliance with the requirements of the Third Package of EU internal market legislation (specifically, Directive 2009/72, Regulation 714/2009, Directive 2005/89). Future design of USAID programs in the energy sector (and more broadly) would benefit from an assessment of how support could be provided to GOAM to implement contemporary (for example, OECD) good governance principles, and, in particular, on energy matters. Simultaneous governance support may help enhance GOAM capability to design policies and assess their impacts (strategic planning, including modeling), to implement them, and to monitor, assess, and adapt these policies in line with monitoring results. This might begin with technical assistance in the form of advisory services to the highest levels of the Ministry of Territorial Administration and Infrastructure (MTAI), which is the GOAM entity in charge of energy policy. The MMS software should eventually be upgraded to fully support the European Network for Transmission System Operators (ENTSO-E) standards (cyber security included) and be consistent with EU trading. This upgrade will be costly—a lower-cost option would be to share the cost of the software with Georgia and integrate power exchanges, including capacity allocations (through joint auctions) onto this platform. Introduce Capacity Allocation and Nomination (ECAN) documents (for example, Rights documents, Total Allocation Results Documents, Control Area Exchange Documents, and Environmental Appraisal Report Documents) to the MMS. These should be introduced in line with ENTSO-E documents such as ENTSO-E ECAN Implementation Guide, ENTSO-E General Code List For Data Interchange, P2 (Policy 2): Scheduling and Accounting and ENTSO-E Settlement Process (ESP) Implementation Guide. As more consumers are given the opportunity to select their own electricity supplier, a Price Comparison Tool should be introduced to enable them to compare offers from all suppliers on a single EXECUTIVE SUMMARY vi platform. This is a common feature of most liberalized electricity markets.2 The EU Regulation on Wholesale Energy Market Integrity and Transparency should be adopted for reporting and preventing wholesale energy-market abuse. For System Operations Integrate the Armenian power system into the EPSO’s SCADA/energy management system to allow EPSO complete coverage to operate more efficiently and in accordance with EU directives. Increase data providers’ capacity during short-term planning and TYNDP development through formal reporting procedures. This would improve the quality and credibility of the data and information submitted, leading, in turn, to more credible short￾term planning and timely TYNDP. Procedures on provision of and payment for ancillary services, as well for congestion management, should be developed for inclusion in the WEM Rules. Flexibility resources should be introduced to the market—for example, storage, demand response, energy communities, prosumers, aggregators—to facilitate increased VRE that will arise in the future. For Cross-Border Trade In preparation for the completion of the 400 kV transmission link and back-to-back stations that will allow greater cross-border trade, PSRC should be designated as the sole authority in Armenia to regulate access to cross-border facilities, including setting market rules for cross-border electricity trading, allocation of cross-border transmission capacity, and congestion management. Through the JWG, PSRC should cooperate with regulatory authorities of neighboring systems and other national entities such as MTAI to coordinate 2 An example of such a tool in the United Kingdom can be found at https://www.energylinx.co.uk/. development and monitoring of congestion management rules and allocation of cross-border capacities. Congestion management arrangements should be established so they are conducted in accordance with non-discriminatory, market-based solutions. EPSO should join ENTSO-E for collaboration on key issues to move the Armenian market closer to the European market and allow electricity trading to take place soon after completion of the new interconnection facilities, the construction of which is expected to begin soon. To prepare for free market electricity trading through Georgia over the 400 kV connection and back-to-back stations, EPSO should be involved in developing or submitting to PSRC congestion management procedures for cross-border trade. To adequately prepare for expanded cross-border connection, USAID should continue to promote/support dialogue through the JWG, as well as closely monitor events and act to provide any other support that may be required to push forward the cross-border trade agenda. Efforts should be made to increase the number of traders on cross-border export/import transactions from the single existing trader. For Power Sector Planning To be able to produce a robust least-cost power sector expansion plan, consider conducting a least￾cost generation expansion study for the power network using software meant for that purpose (for example, WASP or PLEXOS), as opposed to the current practice of using the energy sector-oriented MARKAL/TIMES/VEDA model and then “fine-tuning” for the TYNDP through year-by-year simulations. With Armenian beneficiaries having been provided with the training and tools to undertake LCEDP EXECUTIVE SUMMARY vii analyses, interconnections with neighboring countries not previously included can be assessed with the use of a dummy entity that takes into account associated electricity flows. PSRC should be allowed to request changes and monitor implementation of the TYNDP, preferably through the Electricity Law and changes to the Transmission Network Code; the EU Directive emphasizes that the TYNDP must include, inter alia, the list of investments decided and executed over the next three years. Impact Unless moderate market concentration can be achieved after February 1, 2025, additional measures may be required, such as lowering the threshold value of annual consumption for consumers of the deregulated market from one million kWh. Otherwise, market concentration and potential for manipulation may become a problem. Assistance may be required to further analyze how market concentration, or its effects, may be mitigated. Similarly, to avoid the potential for market abuse, the distribution system operator (DSO) must be obliged to preserve the confidentiality of commercially sensitive information to eliminate potential favoritism to affiliates. USAID can ensure that such a provision is included while assisting with draft regulations. Sustainability Continue to assist in the development of the new Electricity Law, as this is a critical link to lead to an electricity market that is fully compliant with EU requirements. A GOAM working group set up to draft this law was not able to advance development significantly, causing delays. The law is presently being drafted with MLET support, which should continue until the law is passed. Future assistance should continue monitoring the development of the EU electricity market model and rules and supporting GOAM with possible timely amendments to the adopted legislation. As significant changes were implemented after the preparation of MLET’s Review of the Electricity Market Rules and Market Monitoring Solutions, a gap analysis of the status of market monitoring may be considered (that is, compare current status with what was recommended by MLET in 2020). Although MLET has successfully navigated several adverse situations, given the somewhat volatile political situation in Armenia and the Caucasus over the past decade, the political situation should be closely monitored in future projects so that adverse events can be anticipated and USAID/ project implementers can act quickly to adjust and mitigate their effects. LESSONS LEARNED Task 1 Although successful, any custom phased-in approach to electricity market liberalization similar to the one adopted by Armenia can be slow. Based on experience to-date, the made-in-Armenia electricity market solution appears to be serving the country well, with successful implementation of successive transitional phases on February 1, 2022 and February 1, 2023. However, this lesson learned comes with the caveat that the move to a fully liberalized market in line with EU standards is far from complete, as only 12 to 14 percent of total sales are presently taking place on the WEM. Armenia has set a target of 2029 to become fully compliant with EU electricity market requirements. In drafting legislation, large working groups do not appear to work well. Before switching to an alternate modus operandi, where the task was eventually allocated to MLET activity, the creation of a large working group to draft the new Electricity Law did not work at all, causing significant delays. Task 2 The power sector planning methodology followed with MLET support is not consistent with conventional approaches that first seek to optimize EXECUTIVE SUMMARY viii the generation expansion sequence from a more technical (and purely power sector) perspective. This would entail the use of optimization software such as WASP or PLEXOS, rather than first employing a rather high-level energy model for the entire energy sector (MARKAL/TIMES/VEDA) and then trying to “fine tune” for the power network through a year-to-year simulation. This casts an element of doubt on the results, although the actual approach used with MLET support to develop the TYNDP may well have led GOAM to develop the same ESDSP as they would have with the optimization software. Task 3 Basing a significant portion of a program budget on the assumption of completion of a major infrastructure project can potentially have negative consequences if the infrastructure project does not materialize. Fortunately, in the case of MLET, funds budgeted for much-increased cross-border trade as a consequence of the 400 kV transmission line and back-to-back facilities were easily reallocated because of its large size and different components when the anticipated infrastructure project did not materialize as expected. PURPOSE OF EVALUATION AND EVALUATION QUESTIONS 1 PURPOSE OF EVALUATION AND EVALUATION QUESTIONS PURPOSE OF EVALUATION USAID/Armenia engaged the Technical Assistance Project for Economic Growth (TAP EG)—a buy-in mechanism through USAID’s Bureau of Development, Democracy, and Innovation (DDI) Center for Emerging Markets and Development (EMD) implemented by International Development Group LLC (IDG) — to conduct an independent final performance evaluation of the Market Liberalization and Electricity Trade (MLET) activity. The purpose of this final performance evaluation is: (i) to inform USAID of the degree to which MLET, implemented by Tetra Tech ES, Inc., achieved its intended goal and objectives; specifically, how it contributed to power market liberalization, electricity supply diversification, and expanded electricity trade and (ii) to capture key lessons and recommendations for future programming in the sector. Specifically, the evaluation focuses on MLET’s performance on the following tasks: 1. Contribute to electricity market development 2. Promote electricity supply diversification 3. Facilitate cross-border trade with Georgia. USAID/Armenia intends to use the findings and recommendations of this evaluation to inform future programming to effectively help achieve the US government’s strategic objectives in Armenia, particularly in advancing energy security in the country. USAID may share the evaluation report, or its main findings, with the key stakeholders, such as GOAM, MTAI, PSRC, and energy institutions. EVALUATION QUESTIONS This final performance evaluation focused on five evaluation areas: 1) coherence, 2) effectiveness, 3) efficiency, 4) impact, and 5) sustainability. Figure 1 below provides a summary of each evaluation area. Evaluation Area 1: Coherence EQ 1a: To what extent and in what specific ways did MLET contribute to USAID’s strategic intermediate result “energy security increased”? Why? EQ 1b: How has the activity contributed to wider efforts of GOAM to increase energy security in Armenia? How significant and/or relevant were these contributions? Evaluation Area 2: Effectiveness EQ 2a: How effective has MLET been in achieving intended outcomes of developing Armenia’s energy market, promoting energy supply diversification, and facilitating cross-border trade? Which interventions have been most and least effective in achieving the intended outcomes and why? EQ 2b: What were the key factors that enabled and/or hindered the achievement of outcomes? Which of these factors were within and outside of the MLET’s control and manageable interests? EQ 2c: To what extent was MLET able to steer its strategic approach in response to the unfolding of key geo-economic and sector developments during the implementation period? Evaluation Area 3: Efficiency EQ 3: To what extent did the intervention deliver results in an economic and timely way? Evaluation Area 4: Impact EQ 4a: To what extent did the intervention generate or is expected to generate significant positive or negative, intended, or unintended, higher-level effects? EQ 4b: To what extent did the MLET interventions move forward Armenia’s desired energy sector reforms? PURPOSE OF EVALUATION AND EVALUATION QUESTIONS 2 Evaluation Area 5: Sustainability EQ 5a: Which elements of the intervention approaches and results achieved by MLET are most likely to be taken further and sustained beyond the close of the program? Why? EQ 5b: What are the political, geopolitical, and geo￾economic risks most likely to impact the sustainability of the results of the MLET program? Who are the supporters and spoilers of continued market liberalization, increase in the uptake of renewable energy technologies, and increased electricity trade, including regional? HOW THE REPORT IS ORGANIZED The evaluation report is organized as follows: Section two (Background) provides an overview of the project and the Armenian power sector. Section three (Evaluation Design and Methodology) outlines the evaluation methodology, data collection efforts, and limitations. Section four (Findings, Conclusions and Recommendations) presents the findings, conclusions, and recommendations by evaluation area. Section five presents Lessons Learned. FIGURE 1. MLET EVALUATION AREAS Will the benefits last? What difference did the program make? ow well were resources used? Is the program achieving its intended outcomes? ow well does the program fit with the USAID strategy and GOAM efforts? BACKGROUND 3 BACKGROUND MLET is a five-year, $9.99 million activity aimed at improving Armenia’s electricity market and supporting electricity trade with Georgia by helping the Ministry of Energy Infrastructure and Natural Resources (MEINR),3 PSRC, and energy institutions to adopt legal and regulatory reforms, accelerate the implementation of market liberalization mechanisms and to strengthen Armenia-Georgia dialogue on cross-border trade. The activity builds on USAID’s work over the past decade in Armenia to facilitate significant progress in macro-level reforms, including unbundling of the electricity and gas sectors, creation of an independent regulator, exploiting substantial RE resources, large-scale privatization of electricity generation and distribution assets, improving the quality of services, and overcoming various difficulties that the country’s energy sector has faced following independence in 1991. The five-year MLET activity is implemented by Tetra Tech ES, Inc. under the contract #72011118C00001, beginning June 6, 2018, and ending June 5, 2023. The activity is viewed as fundamental to improving Armenia’s energy security and compliance with EU standards and approaches. Energy sector reform is also essential to enhance transparency, reduce corruption, and advance the market opening for sector competitiveness and diversification, in addition to increasing energy trade with Georgia. As indicated in the MLET SOW, the activity supports USAID/Armenia’s 2013-2017 Country Development Cooperation Strategy Development Objective (DO) 1, “More inclusive and sustainable economic growth,” and specifically, Sub-Intermediate Result 1.1.3, “More strategic management of energy and water resources.” 3When the MLET contract was issued, the key counterpart was the Ministry of Energy Infrastructures, and Natural Resources (MEINR), but it was consolidated OVERVIEW OF THE ARMENIAN POWER SECTOR Armenia’s energy sector relies on imported oil and natural gas to a considerable extent. It does not have domestic fossil energy resources. Imports, mostly from Russia, contributed to approximately 77 percent of total energy consumption in 2020.4 This energy insecurity is exacerbated by Armenia’s partial regional isolation, landlocked borders, and need for further energy sector reforms to increase transparency, accountability, and competitiveness to attract investments and increase trade. Armenia’s lack of an open national market, limited integration with regional markets, and an outdated law on energy together exacerbate the problem. GOAM amended the Law on Energy in 2017 to drive greater liberalization of the market. With assistance from the MLET activity, the changes under the law are currently being phased in. GOAM is using a hybrid model during the transition period, whereby existing power purchase agreements are continuing alongside competitive wholesale and regulated markets. The amendments in the law provide for the introduction of competition among electricity suppliers. This is intended to break up the control exercised by one firm, Electric Networks of Armenia (ENA), over distribution throughout Armenia. Segments of ENA’s customer base can now purchase electricity on a wholesale market from trading entities who, in turn, can purchase electricity from a designated deregulated generation market. In addition, large wholesale consumers are now able to enter the market to purchase and consume electricity generated from outside Armenia. Mechanisms to implement the amendments went into gradual effect from February 1, 2022, with with the MTAI in the GOAM reorganization completed in June 2019. 4 https://www.iea.org/countries/armenia BACKGROUND 4 “Qualified Customers”5 being able to choose their own suppliers. The number of Qualified Customers was expanded on February 1, 2023. The principal bodies involved in energy sector governance in Armenia are: (i) MTAI, which is responsible for overall energy policymaking; (ii) the Ministry of Environment; (iii) PSRC; and (iv) the Committee on Nuclear Safety Regulation (ANRA). In January 2021, Armenia adopted the 20-year Energy Sector Development Strategic Program (ESDSP), updated last in January 2023. ESDSP is intended to build a transparent, diversified, and energy-efficient system for sustainable development. ESDSP will identify new power generation facilities, electricity transmission systems, and institutional issues related to market liberalization and associated legislative gaps. On May 2, 2022, the U.S. and Armenia signed a Memorandum of Understanding (MOU) Concerning Civil Nuclear Cooperation. Such MOUs are a diplomatic initiative for the U.S. to better help its partners build their own infrastructure for nuclear energy and technology, adopt exacting standards of nuclear safety, security, and independent regulatory oversight. The MOU provides a framework for the U.S.-Armenia cooperation on nuclear energy security and strengthens economic relationships between the countries.6 Since the MLET activity started, Armenia has made considerable progress in enhancing regional market integration. The country has signed and ratified the CEPA with the EU, which came into force in March 2021. CEPA includes a timetable for the approximation of Armenian laws and regulations to relevant EU laws over the next few years, and by 2029 at the latest. Armenia is also a member of the Eurasian Economic Union (EAEU), which aims to 5 A Qualified Customer is any customer that meets PSRC criteria to be eligible to purchase electricity on the wholesale electricity market, generally restricted to those who can purchase large amounts. establish common EAEU gas and electricity markets by 2025. Armenia has sufficient electricity-generating capacity to meet current domestic needs, but electricity demand is projected to grow by up to 3 percent annually. Electricity in Armenia is generated primarily by the Armenian Nuclear Power Plant (ANPP), hydroelectric plants, and thermal plants. The country’s grid is currently synchronous with that of Iran while maintaining “island” connections with Georgia. Planned increased interconnections with Georgia via a set of back-to-back high-voltage direct current stations will help facilitate trade in the region. The ANPP, with an operating capacity of 440 megawatts (MW), meets just less than 40 percent of Armenia’s demand for electricity. owever, Armenia is under international pressure to decommission this plant for safety reasons, which GOAM is reluctant to do in order to maintain a diversity of energy sources. GOAM expects to obtain an extension of its operating license through 2026. According to its Strategic Program, GOAM is prioritizing a further extension of the operational lifetime of ANPP into 2036. Beyond that, GOAM plans to maintain nuclear power in the country’s energy mix. Hydroelectric plants generally provide an additional 30 percent of Armenia’s electricity demand, but the levels of generation depend on seasonality and rainfall patterns. Thermal power accounts for the bulk of the remaining generation. Solar energy provides less than 2 percent of current power generation. From 2006 to date, ENA has been the sole electric power distributor in the country. It is Armenia’s largest employer and serves approximately 985,000 electric utility customers. ENA’s owner, Tashir Group, has recently announced that, in cooperation 6 https://www.state.gov/the-united-states-of-america-and-the￾republic-of-armenia-sign-a-memorandum-of-understanding￾concerning-strategic-civil-nuclear-cooperation BACKGROUND 5 with international financial institutions, it will invest about $900 million in upgrading its network infrastructure. The High Voltage Electric Networks—a state monopoly operated as a closed joint stock company—manages the assets of Armenia’s transmission network, performs operational and maintenance functions, and implements necessary investment programs. EPSO is responsible for day￾to-day and strategic functioning of Armenia’s power system. PSRC establishes the rules and procedures for the workings of the electricity market and sets and reviews tariffs. The “Settlement Center CJSC” (SC) has been issued a license by PRSC to serve as Market Operator (MO) under a partially liberalized power market. Thus, SC conducts commercial settlements between generators and purchasers, manages electricity imports and exports, and monitors metering and billing services to wholesale market participants (MPs). Under current WEM rules for the transitional market, generating entities in the Armenian electricity market are classified into three “market components”: (i) plants for which long-term power purchase agreements had been signed prior to moving towards market liberalization, (ii) plants (or portions thereof) whose tariffs are regulated, and (iii) plants whose output must be sold on the (liberalized) non-regulated market. In the first instance, plant output is sold to the “Universal Supplier” (in this case, ENA), which then distributes and sells to regulated consumers. In the second instance, plant output is sold to a wider group of customers, including ENA and Qualified Customers, who, at present, can only “qualify” by virtue of being a large consumer connected directly to the transmission system. In the third instance, electricity trade is conducted on a bid-ask basis in an open market, where the MPs include non-regulated generators and the previously mentioned parties plus electricity traders, who can then sell to any Qualified Customer as a “Supplier”. Over time, the intention is to move the threshold of being a Qualified Customer, from transmission-level supply (together with consuming large amounts of electricity) down to smaller-consumption customers served at distribution voltage level, who would then be able to purchase electricity from any supplier on the market. The current target date of achieving this fully liberalized electricity market is February 1, 2025. EVALUATION DESIGN AND METHODOLOGY 6 EVALUATION DESIGN AND METHODOLOGY METHODOLOGY This evaluation was conducted using a mixed￾methods design, combining qualitative and limited quantitative analysis. The approach of using complementary methods helped strengthen the validity of findings, by allowing the evaluation team (ET) to triangulate between different data sources and data collection methods. Findings that supported each other strengthened the ET’s confidence in their validity. In cases where findings did not support each other, the ET raised new questions, which they clarified and sought to resolve by going back to the original sources throughout data collection. Data collection methods applied by the ET are described below. QUALITATIVE RESEARCH AND ANALYSIS The qualitative evaluation was conducted primarily through the following methods. Document and Literature Review: The ET reviewed documents related to the MLET activity, including Quarterly and Annual Reports, Annual Work Plans, and other relevant documentation produced by the activity. A complete list of documents reviewed is provided in annex V. Key Informant Interviews (KIIs): The ET collected data from 23 key informant (KIs) interviews across Tasks 1, 2, and 3. During the KIIs, 38 KIs were present. The key informants were USAID Armenia, the implementing partner staff, government officials, the power system operator, the market settlement center, the distribution utility (ENA), generation entities, large consumers purchasing on the WEM, the regulatory authority (that is, PSRC), international financial institutions, non-governmental organizations (NGOs), and media. Information was collected through in-depth, semi-structured KIIs, conducted mainly in-person in Yerevan, but also remotely. A list of KIs is provided in annex V. Focus Group Discussion (FGD): A focus group discussion was also undertaken in Yerevan with seven wholesale MPs who own small hydropower facilities. QUANTITATIVE ANALYSIS A quantitative analysis was conducted, specifically to respond to EQ 4a on Impact, where market concentration was assessed through the Herfindahl– Hirschman Index (HHI). Another quantitative analysis included three quantifiable questions, where KIs were asked to provide a rating or ranking response. These questions were helpful in getting stakeholders to summarize their perceptions of an issue when followed by a request to explain their rating/ranking on a scale of 1 to 10 (10 being the best). The questions pertained to EQ1, Effectiveness, which received 9 responses. The difference in the number of responses and KIs is because some participants declined to answer, some lacked sufficient knowledge about the activity and, in other cases, the interview did not cover the issue. The mean scores were tabulated. FIGURE 2. KEY INFORMANT INTERVIEWS EVALUATION ACTIVITIES A Work Plan, detailing the evaluation methodology, was prepared in advance of the field activities and approved by USAID before the ET began formal 23 9 0 5 10 15 20 25 No. of KIIs Conducted No. of KIIs Responding EVALUATION DESIGN AND METHODOLOGY 7 interviews with KIs. The schedule for the Work Plan is contained in annex I, while an Evaluation Matrix showing each EQ, the corresponding evaluation method used for data collection and analysis, and data sources is contained in annex IV. Three remote interviews were conducted with USAID and Tetra Tech before field activities were undertaken in Yerevan, where the ET spent two weeks interviewing 35 stakeholders (28 KIs and 7 stakeholders in the FGD). This was followed by four further remote interviews. EVALUATION LIMITATIONS The MLET evaluation, like all evaluations, has several limitations. These include: • The evaluation takes place before the end of the activity. This means that: a) not all activities will have been completed; and b) insufficient time will have elapsed to be able to assess the sustainability of the intervention, a common challenge for any “final evaluation” that is conducted before the activity’s end. This limitation has been addressed, to the extent possible, by assessing general factors known to contribute to sustainability as they would relate to the electricity supply industry, such as institutionalization of changes, demand for capacity, and change in capacity. • Time and resources to collect primary data were limited and access to all key stakeholders has not been established (for instance, the generating plant owned by the Tashir Group). This may have limited the ability to draw firm conclusions in certain areas, such as representativeness of stakeholder views. • Selection bias, as some key informants may have declined to be interviewed or were not available to be interviewed. Those respondents who chose to be interviewed might differ from those who did not in terms of their attitudes and perceptions, affiliation with government/non-government structures, and socio-demographic characteristics and experience. • Recall bias, since several questions raised during the interviews dealt with issues that took place in the past. The MLET program, together with predecessor projects financed by USAID, has a history going back to the mid-2000s, and therefore, some respondents may have difficulty recalling relevant information specific to MLET. • Absence of a counterfactual scenario. Due to the absence of a counterfactual, the evaluation design will not be able to collect evidence that attributes observed results to interventions by MLET. The evaluation design attempted to compensate for several of the above limitations through the following methods: • Triangulating evidence from different qualitative and quantitative data sources helped improve the credibility of findings via validation by multiple data sources. • A certain degree of selection bias is unavoidable because the evaluation relied partly on Tetra Tech and USAID to obtain contact information of key informants. To mitigate this bias, the evaluation team selected key informants to be interviewed. During the KIIs, stakeholders were asked to recommend other key informants (snowball method) to verify that data is being obtained from the most relevant sources. Regarding qualitative findings, the evaluation team was careful to report only those findings that recur with relative frequency across multiple stakeholders. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 8 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS EVALUATION AREA 1—COHERENCE EQ1a. To what extent and in what specific ways did MLET contribute to USAID’s strategic intermediate result “energy security increased”? Why? FINDINGS The International Energy Association (IEA) defines energy security as “the uninterrupted availability of energy sources at an affordable price.” Given this general definition, energy security can have many aspects. According to the USAID Contracting Officer’s Representative (COR), energy security is further defined as a longer-term goal dealing mainly with timely investments to supply energy in line with economic developments. In the Armenian context, this includes market liberalization that will support investments, supply diversification, and integration with Georgia. The MLET SOW is divided into three main tasks: (i) Energy Market Development, (ii) Promote Energy Supply Diversification and (iii) Facilitate Cross￾Border Trade. Task 1. Energy Market Development. Sub-tasks include: (i) legal and regulatory reforms to support market liberalization and an effective electricity trading mechanism, (ii) develop and implement the market model for Armenia, and (iii) strengthen the market and system operators and MPs. The sub￾tasks align with the market liberalization component of the energy security definition. MLET addressed all three sub-task areas respectively by: (i) providing input on the drafting of laws and regulations, (ii) supporting the movement from a “single-buyer” electricity market towards the implementation of a competitive electricity market, and (iii) providing assistance in the development of the MMS software as well as support, advisory services, and capacity building to the PSRC, market operator, system operator, and other MPs to implement the new system. Thus, significant progress in the market liberalization aspect of energy security was made. Task 2. Promote Energy Supply Diversification. This task by definition aligns with the supply diversification component of energy security definition. MLET addressed this issue by supporting the update of the LCEDP to consider technological developments in nuclear power and renewables as well as assistance to GOAM in developing its ESDSP. GOAM’s plan is thus in line with supply diversification, moving away from aging conventional resources towards more modern and diverse sources. However, not all considerations analyzed in the LCEDP have been adopted by the ESDSP, notably options related to the replacement of the ANPP. However, interviews with two stakeholders revealed that these options are seriously being considered. Task 3. Facilitate Cross-Border Trade. The SOW specifically targets Georgia to facilitate cross-border trade. Thus, the task aligns with the “integration with Georgia” component of the energy security definition. MLET addressed this by organizing regular meetings of the JWG and a workshop on trade between Armenia and Georgia, as well as the preparation of regional studies on electricity generation and transmission. Trade with Georgia increased remarkably in 2022, from 242 GWh over the whole ten-year period between 2012 to 2021 to 365 GWh in 2022. MTAI’s forecast for 2023 is about 500 GWh. On the other hand, the extent of increased energy security due to cross-border cooperation in the future may be less than envisaged because of transmission constraints and delays in the construction of a crucial Armenia-Georgia transmission link and back-to-back stations. CONCLUSIONS The three tasks of the MLET SOW align perfectly with the definition of energy security. MLET contributed significantly to USAID’s strategic FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 9 intermediate result “energy security increased” in the area of market liberalization, but to a lesser extent in terms of supply diversification and cross￾border trade. However, the findings suggest that energy security related to these areas will eventually materialize with the adoption of the preferred nuclear development path and eventual completion of a transmission link to Georgia. RECOMMENDATIONS Given their alignment with USAID’s strategies of market liberalization and increased cross-border trade, continued assistance should be provided in line with the potential additional progress that can be achieved. EQ 1b. How has the activity contributed to GOAM’s wider efforts to increase energy security in Armenia? How significant and/or relevant were these contributions? FINDINGS The GOAM ESDSP to 2040 cites five priorities: i) development of RE potential, ii) EE across all sectors, iii) nuclear power, iv) transmission interconnections, and v) gradual electricity market liberalization. In the area of RE, revisions to the secondary legislation came into force on May 1, enabling new options for net metering and billing for autonomous power producers and market-based renewable energy integration. MLET assisted in the preparation of these regulations. MLET also provided transaction advisory support to the Renewable Resources and Energy Efficiency Fund (R2E2) of Armenia to manage the procurement process to prepare, organize, execute, and complete a tender for the Ayg-1 200 MW solar project. MLET supported negotiations with the investor in the project (Masdar, Abu Dhabi) and its lenders until the project achieved financial close. This development of RE resources responds to the energy security criterion of energy diversification. Regarding EE, MLET is currently supporting GOAM efforts to develop Armenia’s energy efficiency potential by comparing its legislation to the energy efficiency elements of EU legislation. Before the end of the activity, MLET is to develop an Energy Efficiency Gap Analysis Report that reviews EU and Armenian legislation and provides recommendations on amendments to Armenian legislation. This activity supports the path to EU integration and thus, to increased energy security. Regarding nuclear power, MLET is supporting GOAM to address options for nuclear power development by aiding in the assessment of various nuclear power generation scenarios in the LCEDP. However, the desired direction stated in the SOW to develop alternatives to continued operation of the ANPP has been slow to materialize (GOAM’s latest ESDSP is still considering extending the life of the ANPP to 2036). Two of the interviewed stakeholders mentioned that the introduction of small modular reactors (SMRs) to replace the ANPP is the best path to follow, which is the stated position of USAID in the SOW. This development of nuclear power capability and options responds to the energy security criterion of energy diversification. The MLET activity provides an opportunity for Armenia to be a bridge in engaging in more global electricity markets as well as taking advantage of opportunities for mutually beneficial electricity trade with neighboring countries. Moreover, full implementation of this project will increase the reliability and security of the Armenian power system. MLET’s Task 3 in support of cross-border trade directly addresses the energy security criterion of energy diversification. As previously stated, MLET has been involved in organizing regular meetings of the JWG and a workshop on trade between Armenia and Georgia, as well as the preparation of regional studies on electricity generation and transmission. Regarding market liberalization, Armenia has already commenced this process and will transition to a new liberalized model in the coming years, which will be subject to limited competition. Certain milestones towards full liberalization have been established till February 1, 2025, with subsequent steps envisaged FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 10 so that Armenia may eventually comply with all EU directives. This will then allow active participation by Armenia in the EU electricity market. MLET has been instrumental in providing input on the drafting of laws and regulations, supporting the movement from a “single buyer” electricity market towards the implementation of a competitive electricity market, and providing the necessary assistance, advice and capacity building to all stakeholders in implementing the new market. CONCLUSIONS MLET has contributed significantly to all five strategic areas of GOAM’s efforts to increase energy security in Armenia, including: i) development of RE potential, ii) EE across all sectors, iii) nuclear power, iv) transmission interconnections, and v) gradual electricity market liberalization. The largest contribution to energy security by far has been in the area of market liberalization. RECOMMENDATIONS Given the evident alignment of the MLET activity with GOAM strategies and the potential for greater achievements in these areas, the follow-on USAID energy activity in Armenia, Energy Secure Armenia, should continue to expand on the work done under MLET. For instance, the groundwork for cross-border trade with Georgia has been laid with the facilitation of the JWG and other analyses, and future programming in this sector can take this work further. EVALUATION AREA 2—EFFECTIVENESS EQ 2a. How effective has MLET been in achieving intended outcomes of developing Armenia’s energy market, promoting energy supply diversification, and facilitating cross￾border trade? Which interventions have been most and least effective in achieving the intended outcomes and why? FINDINGS Findings by task and subtask are provided below. First, however, it may be noted that three general questions regarding effectiveness were posed to all beneficiary-KIs. These KIs were asked to rate the effectiveness of the technical assistance, satisfaction with the technical assistance, and the contractor’s work, on a scale from 1 to 10, with 10 being the highest. Nine KIs out of the 38 KIs interviewed were in a position to respond to these questions. All except one KI gave a rating of 9 or 10 to each of the three questions, with one “outlier” responding with scores of 6, 7.5 and 8 for effectiveness, satisfaction and contractor, respectively. Mean scores were 9.2, 9.4 and 9.4 respectively. TASK 1: ENERGY MARKET DEVELOPMENT The MLET SOW Task 1 is Energy Market Development. Sub-tasks as stipulated in the SOW include: (i) legal and regulatory reforms to support market liberalization and an effective electricity trading mechanism, (ii) develop and implement the market model for Armenia and (iii) strengthen the market and system operators and MPs. Subtask 1.1: Legal and regulatory reforms to support market liberalization and an effective electricity trading mechanism. The following findings arose after an examination of MLET Annual Reports and interviews with KIs. • MLET provided/ supported drafting of several strategic and analytical documents to the Armenian stakeholders supporting the market development, including ”The Action Plan to MLET has contributed significantly to all five strategic areas of GOAM’s efforts to increase energy security in Armenia. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 11 Ensure Implementation of The Republic of Armenia Energy Sector Development Strategic Program”, “The Report on Draft Methodology and Procedure for Setting (Revision) of Tariffs in the Electricity System”, the Reliability and Security Indicators, “Gap Analysis for Policy, Legal, Regulatory and Institutional Framework for Development of Renewable Energy in Armenia”, “Electricity Market Gap Analysis” , “Review of the Electricity Market Rules, and Market Monitoring Solutions”, Annual Adequacy Forecast (AAF), “Recommendations to Improve Secondary Legislation for Renewable Energy”, “Ten Year Network Development Plan”, and the “Market Monitoring Guidelines”. All documents were adopted by Armenian authorities or endorsed by the project governance mechanism. • MLET drafted several pieces of secondary legislation, such as the new Licensing Rules, several model licenses, the Wholesale Electricity Market (WEM) Rules, WEM Contracts, Retail Electricity Market Trading Rules and Contracts, Transmission Network (Grid) Code, and the Distribution Network Code. All documents have been endorsed by appropriate stakeholders. • The activity supported drafting amendments to the Energy Law (2001), providing the necessary legal basis to enhance the balancing mechanism and establish the ancillary services market. The amended Energy Law is in force. • The Activity supported drafting the outline of the new Electricity Law and assisted the MTAI working group activities on drafting the new Electricity Law. This task has not been finalized within the reporting period due to an “ineffective working group”. MLET has attempted to overcome this constraint by mobilizing a small team of lawyers to start drafting the new law chapter by chapter.7 7 One possible lesson to be learned from another country (Serbia) is that large working groups were shown to be significantly less efficient in comparison to small working Drafting of the amendments to the Energy Law and the new Electricity Law took more time than estimated in both GOAM plans and the project timetable. The COVID-19 pandemic and the conflict with Azerbaijan may have adversely affected the dynamics and priorities of the MTAI Working Group (WG), leading to inefficiencies in the establishment and operational performance of the WG, which was out of MLET’s control. According to the MLET’s “Fourth Annual Report: October 2021–September 2022,” MLET “tried unsuccessfully to begin drafting the new Electricity Law with MTAI’s working group.” According to the same report, “the working group mechanism did not prove to be functional.” Finally, GOAM extended the deadline for finalizing of the Electricity Law to end-December 2023, while MLET “mobilized a small team of lawyers to start drafting the law by chapter and will share the chapters with MTAI working group for comments and suggestions”. owever, the activity succeeded in supporting the legislation necessary to launch the new electricity market arrangements (new wholesale market model, direct contracts, balancing mechanism), while work on the Electricity Law continued. It has been noted that GOAM has not adopted the elements of EU Third Package of internal market legislation word by word, but has attempted to make adjustments to take into account specificities of the Armenian energy sector, which has affected the dynamics of the transition process. Such a lengthy evolution of legislation introducing electricity market liberalization is not unusual, as experienced in other jurisdictions. It should be noted that the EU reform process started in the late 1990s and is again under reconstruction. For context, drafting the energy laws in Serbia in 2004, 2011, and 2014 took two years per act on average. On the other hand, GOAM’s concerns that the specificities of Armenia should be taken into consideration while transposing EU legislation are groups or consultants followed by a proper public consultation. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 12 not fully unjustified. The Third Package was designed for the EU market of 500 million customers, in which effective competition is supported by a huge number of market participants, common institutions supporting market governance, and common mechanisms supporting cross-border trade. Thus, the legal framework of the Energy Community, comprising contracting parties from Western Balkans and Black Sea regions directly connected with Continental Europe (with exception of Georgia), includes provisions adapting to the special circumstances of the various contracting parties. Subtask 1.2. Develop and implement the market model for Armenia. Examination of MLET Annual Reports and interviews with beneficiaries, including MTAI, PSRC, EPSO, SC, four generators, and two trading entities yielded the following findings. Market design • MLET supported the establishment and/or principles for electricity markets and related required mechanisms as (for example, day-ahead, balancing, and ancillary services) that are properly regulated, operated, and utilized by all MPs. • MLET supported PSRC in revising tariff calculations. • MLET supported the development of data transparency requirements, including revised regulatory reporting forms to strengthen market monitoring. • MLET developed a brief report on EU practices on licensing procedures for electricity undertakings in a competitive market environment that was provided to an MTAI working group for review, comments, and recommendations. • Principles of balancing and ancillary services (which are essential for a fully functional WEM) were introduced in the Energy Law amendments of 2018, in the WEM Rules, and in the MMS. However, the following should be noted: — The MLET SOW stipulated development of an ancillary services market along with other activities. While ancillary services are mentioned in the definitions section of the WEM Rules and in other documents, WEM Rules do not define procedures for the provision of and payment for ancillary services such as voltage control. — Articles 262 to 268, among other articles of the Power System Transmission Network (Grid) Code of the Republic of Armenia, describe congestion and congestion management measures to be undertaken by the system operator and refer to WEM Rules. Meanwhile, WEM Rules do not explicitly prescribe procedures and responsibilities for congestion management. • MLET is developing the market monitoring report and guidance to effectively monitor the market. For this, MLET is supporting PSRC, SC, and EPSO to develop bi-monthly feedback reports on market monitoring and will prepare a market monitoring guide to support PSRC and SC to effectively monitor the market. Implementation support • MLET developed, procured, and put into operation the custom-tailored AEX MMS. During the development and pilot phases, the AEX was shaped in accordance with the specifics of WEM and metering infrastructure, completing it in time for the launch of the new transitional electricity market. This began with a limited deregulated WEM in February 2022, but from February 2023, the regulated component market also becomes operational. • After eleven billing periods (that is, till immediately before February 2023), the AEX platform had generated about 3,000 successfully cleared and paid invoices for 250 MPs, who had concluded about 30 long-term contracts on different market components. • Currently, 45 small hydropower plants are participating in the competitive market. Over the years to come, this number will increase as other long-term contracts concluded with GOAM expire. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 13 • From February 2023, PSCR allowed the larger￾sized ANPP and Severn Hrazdan Cascade power plants to sell portions of their energy on the day￾ahead market. • As of February 2023, there were more than 16 Qualified Customers, four active Suppliers and five traders at the AEX. System operations MLET assisted EPSO to develop the AAF, which was designed to ensure the use of proper and calibrated baseline Armenian power system models to support EPSO’s functions under the new electricity market environment. This also entailed a series of on-the￾job training on the use of these models. The final AAF methodology was approved by PSRC and adopted by the EPSO in developing the first AAF in December 2022. However, key gaps in the development of the AAF provided in Annual Reports and confirmed by interviewers include: • Required data were scattered across different data sources and authorities. • The Armenian power system is not yet well integrated into EPSO’s SCADA/energy management system; therefore, the hourly dispatch simulation for the whole year using the least-cost dispatch principle is not credible and the collection, storage, verification, and transfer of data across multiple platforms requires further optimization and process automation. • MLET is harmonizing the template forms EPSO needs for the AAF and helping develop procedures to collect and analyze data. With MLET’s support, EPSO was able to prepare and disseminate the first AAF by December 2022. Other Sub-task areas MLET designed and implemented a communications campaign to inform and educate the Armenian population on electricity market reforms. MLET supported GOAM efforts to develop Armenia’s EE potential by: (i) comparing its legislation to the EE elements of the EU acquis, (ii) developing an Energy Efficiency Gap Analysis Report, and (iii) preparing a report Recommendations on Amendments to Proposed Armenian Primary Legislation (Energy Saving and Renewable Energy Law) along with the respective action plan. Other findings Although the MLET activity has been effective in supporting legal and regulatory reforms, gaps remain for Armenia to arrive at a liberalized electricity market that functions in accordance with the CEPA Agreement, specifically: • With an emphasis placed on a “made-in-Armenia” transitional market, full compliance with the commitments made under CEPA is yet to be achieved. Also, no clear path exists as to when and how the CEPA commitments are to be achieved. GOAM, when left on its own to move forward in this area (for example, develop the Electricity Law), has not made adequate progress. • Although the MMS software serves the current and near-term Armenian electricity market well, it does not meet EU standards, which will be critical if trading begins with the EU area. • Guides and pertinent reference material for EPSO and SC to efficiently operate in an increasingly complex electricity market and in accordance with EU standards are not readily available. Additionally, several gaps have been noted, which may be addressed as the market matures: • EPSO’s SCADA/energy management system does not yet cover the whole of the Armenian power system. • A data gathering process for the purpose of short￾term planning and TYNDP development has not been formalized, which has implications on quality and credibility of the data used. • No provision has yet been made for certain market-based ancillary services that EPSO would normally be expected to provide (for example, frequency control, voltage regulation) nor congestion management. • Little thought has yet been given to flexibility resources to help manage demand in an environment of increasing VRE penetration. These FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 14 are not yet present, but may become more critical as VRE adoption increases. Subtask 1.3. Strengthen the market and system operators and market participants. Findings were: All energy sector entities, particularly MTAI, PSRC, EPSO, and SC, participated extensively in the implementation of the new electricity market. Also, the capacity of all MPs was strengthened through workshops and seminars linked to the pilot phase of MMS operation. • MLET prepared a compendium of adopted legislation, which was printed and distributed among MPs to make a handbook for everyday use and discussion. • MLET supported procurement and the establishment of SC’s control room and training center. The control room was furnished with modern electricity, voice, data, and access control infrastructure. • MLET trained potential Qualified Customers on the main principles of the new WEM Rules and MMS. • A training center was established for MPs to prepare them for the market opening that occurred in February 2023. • MLET managed, maintained, and operationalized IT systems at SC, including staff capacity-building. • A need was expressed by practically all KIs for more training and capacity building, particularly once the market is fully in operation from February 1, 2023. • MLET strengthened the capacity of all the above energy sector entities by supporting their extensive participation in the implementation of the new electricity market, as well as the capacity of all MPs through workshops and seminars linked to the pilot phase of MMS operation. • A need was expressed by almost all KIs for additional capacity building and workshops with EU market operators. The COVID-19 pandemic curtailed much of these originally envisaged capacity-building activities. TASK 2. PROMOTE ENERGY SUPPLY DIVERSIFICATION The SOW describes this task as “update the LCEDP and support the update of the Armenian Energy Strategy to address realistic options for energy alternatives potentially available within the ten-year ANPP replacement horizon, such as western￾researched SMR and new renewable technologies, as well as to support transition from the Markal software to Times.” Effectiveness in achieving intended outcomes is described below by sub-task. Subtask 2.1. Update the LCEDP annually to consider technological developments in SMR and renewables, etc. Findings were: • Updating the then-existing software to MARKAL/TIMES is presumably included under this sub-task. This was undertaken during 2019, along with the development of a baseline scenario of the LCEDP. The first version of the LCEDP was issued in November 2019. This was subsequently updated in June 2022, which suggests that the proposed annual updates of the LCEDP as per the SOW was perhaps too ambitious. This may be due to a few factors, including COVID-19, and a certain slowness of GOAM to comment/accept the recommendations arising from the LCEDP. In addition, after having gone through the process of model updates and training on MARKAL/TIMES, MLET continued to develop the LCEDP through addition of the TIMES/VEDA module, which required more training and thus involved another prolonged learning curve on running the new software. • The Evaluation Team notes that the development of the LCEDP did not consider the 1,000 MW interconnection with Iran, which could pose a significant challenge as the peak demand of the Armenian system stands at about 2,000 MW. While Iran’s exclusion is in accordance with the U.S. Government’s foreign policy stance, the absence of this input may have implications for the long-term power system plan for Armenia. This becomes particularly important as Armenia plans to further augment this interconnection, and its FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 15 long-term power system plan must be consistent with the higher-level LCEDP. This issue, however, is outside the scope of MLET. Although analysis of over 20 LCEDP scenarios may have led to a comprehensive understanding of the Armenian power network, and the probable impact of the interconnection, its full impact on Armenia’s long￾term power system plan remains unclear. • There is no evidence of a least-cost plan power system having been developed using more effective software, such as, WASP8 or PLEXOS, 9 which are designed to find the economically optimal generation expansion sequence for an electric power system within user-specified constraints. MARKAL/TIMES/VEDA simulates the energy sector at a higher level across multiple sectors, which does not necessarily produce an optimal generation expansion plan. The TYNDP was developed using Antares, which is an open￾sourced tool that can simulate power system operations on an hourly basis given specific configurations. However, this tool is not oriented towards producing an optimum generation expansion plan. Subtask 2.2. Assist the Energy Working group in updates of the Energy Strategy and development of the ten-year energy plan based on the LCEDP. Findings were: • Running the software multiple times, whether MARKAL/TIMES or TIMES/VEDA, produces results for various scenarios that can then be taken as inputs for developing a strategic plan for the energy sector. MLET presented the results of these scenarios and recommendations for a strategic plan to stakeholders in both 2019 and 2022, thus advising GOAM in its formulation of the ESDSP. 8 Wien Automatic System Planning (WASP) software, International Atomic Energy Agency, https://www.iaea.org/publications/6327/wien-automatic￾system-planning-wasp-package-a-computer-code-for￾Subtask 2.3. Support the Armenian Nuclear Regulatory Agency, MOEINR, and PSRC in developing a regulatory framework, planning, and environmental studies as well as capacity building for new nuclear SMR and/or renewable technologies. The evaluation team’s finding was the following: • This activity did not occur as originally envisaged. GOAM has not yet officially adopted a policy of shutting down the ANPP in favor of SMRs. Subtask 2.4. Assist PSRC in developing new tariff methodology and tariff structure to support investments for electricity generation. The evaluation team’s finding was the following: • This activity included the drafting of procedures and methodologies for setting tariffs, attendance of PSRC staff at a course in Budapest, and assistance in developing a transmission level tariff. This SOW task has been interpreted by MLET and PSRC as relating to procedures in the development of tariffs within the liberalized market, principally transmission tariffs, with no detailed analyses of the tariffs themselves nor of the tariff structure. TASK 3. FACILITATE CROSS-BORDER TRADE. Sub-tasks in this area include: (i) strengthen Armenia-Georgia electricity dialogue through support to the JWG; (ii) promote export and import transactions in wholesale markets; (iii) support regional studies on electricity generation and transmission; (iv) assist in the development and implementation of non-discriminatory cross-border access to all eligible traders; and (v) support development of cross-border trade mechanisms, including joint dispatch of power exchange, development of bilateral contracts, day-ahead market and balancing. Findings were: • The principal activity in this area led by MLET was regular meetings of the JWG, where five such meetings were held over the course of the power-generating-system-expansion-planning-version-wasp￾iv 9 PLEXOS, Energy Exemplar, https://www.energyexemplar.com/plexos FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 16 activity. At the beginning of the activity, topics of discussion included: — Future possible market concept designs. — Reserve sharing between Armenia and Georgia identified under the USAID-funded Black Sea Regional project study led by the US Energy Association and the National Association of Regulatory Utility Commissioners. — Potential for recovery of the 110 kV Lalvar cross-border transmission line. Status of the planned 400 kV direct current back-to-back interconnection project. • Proposal to establish an Armenia-Georgia￾Moldova-Ukraine transmission system operator. Of the above issues, by far the most crucial in terms of trade enhancement is the 400 kV interconnection. Despite being acknowledged as an important development since an agreement was first signed in 2010, the trade enhancement has often been delayed for various reasons. This, in turn, has delayed progress on resolving cross-border trade issues since efforts have focused more on attempting to move forward the implementation of the interconnection, which MLET has had little control over. Regarding the promotion of import and export transactions in wholesale markets, the remarkable increase in cross-border trade has already been noted. Whether this was due more to a distinct effort to focus on trade or whether it is a consequence of the market liberalization that has taken place is difficult to say. “Regional studies” are said to have been undertaken as part of the LCEDP, where the Georgia interconnection (but not others) has been considered as an integral component. MLET also coordinated with the USAID-funded Black Sea Regional Transmission Planning activity for sub￾regional simulation optimization modeling and regional generation and transmission studies. Assistance in the development and implementation of non-discriminatory cross-border access to all eligible traders resulted in only one trading entity having transacted with Georgia since 2022. However, this one entity is different from the single entity that worked under the MTAI in executing cross-border trades prior to 2022. Finally, support in the development of cross-border trade mechanisms has been limited because of the limited progress in the development of interconnection facilities. Nevertheless, discussions in this area have taken place in JWG meetings. MLET has drafted an agreement between Armenia and Georgia on the regulation of cross-border power flow imbalances and on emergency supply of electricity to regulate deviations during commercial deliveries of electricity. CONCLUSIONS With eight out of nine beneficiary-KIIs giving scores of 9 or 10 to MLET effectiveness, satisfaction, and contractor performance, it may be concluded that almost all KIs were extremely satisfied with MLET and that MLET can be evaluated as very effective from a beneficiary-KI perspective. SUBTASK 1.1 (LEGAL/ REGULATORY) MLET has been effective in supporting legal and regulatory reforms of the Armenian electricity sector in close collaboration with main Armenian stakeholders—MTAI, PSRC, MO, SO. This conclusion corresponds with experience in other emerging markets; that is, drafting rules is usually the more successful part of the energy sector reforms supported by international development organizations when compared to their actual implementation. MLET has been successful in providing the regulatory framework necessary to launch the new market model for Armenia in 2022 based on the 2018 amendments to the Energy Law. However, further work is needed to improve the efficiency of state Almost all KIs were extremely satisfied with MLET. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 17 administration in order to avoid lengthy legislative processes (for example. drafting the Electricity Law, which is still a work in progress). No clear path exists as to when and how the CEPA commitments are to be achieved, which is part of a greater governance issue within GOAM, where a lack of capacity exists to design policies and assess their impacts through strategic planning exercises, to implement them, and to monitor, assess, and adapt these policies in line with monitoring results. This is exemplified by its lack of progress in developing the Electricity Law. SUBTASK 1.2 (DEVELOP AND IMPLEMENT THE MARKET MODEL) As the principles of balancing and ancillary services were included in the Energy Law amendments and further elaborated in the WEM Rules and in MMS, the ET concludes that an essential balancing market has been successfully developed and serves as a solid ground for further development as the WEM evolves. The AAF addresses a key requirement in the WEM Rules by ensuring the least-cost dispatch of all available resources in the Armenian power system. It acts as a baseline for the closer-to-real-time operational procedures defined under the short￾term planning section of the Transmission Network Code. The EPSO is required to collect annual data to develop the AAF and to continue and complement its data collection for annual, monthly, and day-ahead periods. Despite the cited data gaps, the first AAF was produced in December 2022, which indicates success in implementation. MLET developed, procured, and put in operation the custom-tailored AEX MMS. During the development and pilot phases, the team shaped the AEX in accordance with the specifics of the WEM and metering infrastructure, completing it in time for the launch of the new transitional electricity market. This is enough evidence to conclude that a functional day-ahead market was introduced. At the current incipient stage of the deregulated electricity market development, the identified shortcomings related to ancillary services and congestion management do not have a critical impact. However, the importance of the above procedures will be increasing as the share of the deregulated electricity market unfolds, especially as electricity trade between Armenia and Georgia increases. SUBTASK 1.3 (STRENGTHENING). The MLET activity was effective in strengthening the market and system operators and market participants. There was a strong (unanimous) consensus among stakeholders that MLET delivered knowledgeable and sound operational support, that MLET was available and very responsive to all stakeholders (especially as they were venturing into largely unknown territory), and that the training and workshops were valuable in transitioning to the WEM. TASK 2 (ENERGY SUPPLY DIVERSIFICATION) Given that the introduction of SMRs has been somewhat of a cornerstone of the energy supply diversification task (it is mentioned upfront in MLET’s SOW), adoption of SMRs by GOAM in its ESDSP to replace the aging ANPP has not occurred, consequently putting into question the effectiveness of this aspect of the task. The SMRs are not mentioned in the ESDSP. The development of nuclear regulations and tariff methodology/tariff structure support mentioned in the SOW either did not materialize (nuclear power regulations) or took a slightly different path than originally envisaged (the tariff methodology/structure support dealt only with the tariff setting procedure/ process). This was due to the PSRC not expressing a need for specific support. TASK 3 (CROSS-BORDER TRADE) The effectiveness of this task is best assessed through MLET’s support to the JWG, mainly in terms of keeping the dialogue going as the start of construction of the 400 kV facilities is becoming FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 18 more imminent. The five JWG meetings that have occurred over the life of MLET have been the most important in this regard. The draft agreement on emergency supply and dealing with commercial supply deviations is also an encouraging sign of progress. Although the task may be regarded as effective on this basis, future effectiveness will depend on implementation of the interconnection. In assessing the effectiveness of Task 2, which primarily focused on LCEDP development, it is important to consider two key factors: (i) the absence of a viable alternative to replacing the aging ANPP, such as the adoption of SMR technology, as originally outlined in the MLET SOW; and (ii) the absence of a least-cost power sector expansion analysis to formulate an optimal TYNDP that complements the higher-level LCEDP derived from the MARKAL/TIMES/VEDA program. Notwithstanding these gaps in analysis, Task 2 did furnish pertinent outputs that GOAM took into account while developing its ESDSP. Therefore, Task 2 can be regarded as effective, with limitations in the technical approach taken. Based on the above analysis, it is concluded that MLET has been most effective in carrying out Task 1 (energy market development) and Task 3 (cross￾border trade), with Task 1 determined to be a huge success and the future effectiveness of Task 3 being heavily dependent on the Georgia interconnection. The effectiveness of Task 2 (energy supply diversification) is open to some questions because: (i) GOAM has not yet adopted a suitable alternative to the ANPP, (ii) the LCEDP does not consider all planned interconnections, and (iii) a power sector expansion plan that adheres to conventional approaches to develop optimal least-cost plans has not been formulated. RECOMMENDATIONS FOR THE LIBERALIZED ELECTRICITY MARKET Implement the Action Plan provided in the Electricity Market Gap Analysis to achieve full compliance with the commitments under CEPA; to shift from current status (partial compliance) to full compliance with the requirements of the Third Package of EU internal market legislation (Directive 2009/72, Regulation 714/2009, Directive 2005/89). Future design of the USAID program in the energy sector (and perhaps more broadly) would benefit from an assessment of how support could be provided to GOAM to implement contemporary (for example, OECD) good governance principles (in particular, on energy matters within the MTAI). Simultaneous governance support may help enhance GOAM capability to design policies and assess their impacts (strategic planning, including modeling), to implement them, and to monitor, assess, and adapt these policies in line with monitoring results. This might begin with technical assistance in the form of advisory services at the highest levels of MTAI. One of the higher-placed KIs noted that “The government ability hinges on the TA the project provided.” The capacity gap is supported by the experience of the delayed work on the Electricity Law, where it was first left to GOAM working groups to develop. This is a fairly broad recommendation although the need to improve GOAM capacity to govern the energy sector has been specifically identified. Governance is usually an overarching (not sectoral) issue in emerging economies and is best addressed in that way, although the recommendation in this case applies only to the energy sector. MMS software should eventually be upgraded or transitioned to the Nord Pool software currently being used by Georgia, with full support of ENTSO-E standards (cyber security included) so it is consistent with EU trading. This upgrade will be costly—a lower-cost option would be to share the cost of software with Georgia and integrate power exchanges, including capacity allocations (through joint auctions) onto this platform. Introduce Capacity Allocation and Nomination (ECAN) documents (for example, Rights documents, Total Allocation Results Documents, Control Area Exchange Documents and Environmental Appraisal Report Documents) to MMS. These should be introduced in line with ENTSO-E documents such as FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 19 ENTSO-E ECAN Implementation Guide, ENTSO-E General Code List For Data Interchange, P2 (Policy 2): Scheduling and Accounting and ENTSO-E Settlement Process (ESP) Implementation Guide. As more consumers are provided with the opportunity to select their own electricity supplier, a Price Comparison Tool should be introduced to enable them to compare offers from all suppliers on a single platform. This is a common feature in most liberalized electricity markets. The EU Regulation on Wholesale Energy Market Integrity and Transparency should be adopted for reporting and preventing wholesale energy market abuse. FOR SYSTEM OPERATIONS Integrate the Armenian power system into the EPSO’s SCADA/energy management system to allow EPSO complete coverage so it can operate more efficiently and in accordance with EU directives. Increase data-provider capacity during short-term planning and TYNDP development through formal reporting procedures. This would improve the quality and credibility of the data and information submitted, leading, in turn, to more credible short￾term planning and timely TYNDP. Procedures on provision of and payment for ancillary services as well as congestion management procedures should be developed for inclusion in the WEM Rules. Flexibility resources should be introduced to the market —for example, storage, demand response, energy communities, prosumers, aggregators—to facilitate increased VRE that will arise in the future. FOR CROSS-BORDER TRADE In preparation for completion of the 400 kV transmission link and back-to-back stations that will allow for a much greater degree of cross-border trade, PSRC should be designated as the sole authority in Armenia to regulate access to cross￾border facilities, including setting market rules for cross-border electricity trading, allocation of cross￾border transmission capacity, and congestion management. Through the JWG, PSRC should seek to cooperate with regulatory authorities of neighboring systems and other national entities such as MTAI to coordinate development and monitoring of congestion management rules and allocation of cross-border capacities. Congestion management arrangements should be established so they are conducted in accordance with non-discriminatory, market-based solutions. EPSO should join the European Network for Transmission System Operators (ENTSO-E) for collaboration on key issues to move the Armenian market closer to the European market and allow electricity trading to take place. EPSO should be involved in developing or submitting to the regulator congestion management procedures for cross-border trade to prepare for free market electricity trading through Georgia over the 400 kV connection and back-to-back stations. To adequately prepare for the expanded cross￾border connection, USAID should continue to promote/ support dialogue through the JWG, as well as closely monitor events and act to provide any other support that may be required to push the cross-border trade agenda forward. Efforts should be made to increase the number of traders on cross-border export/import transactions from the single existing trader. FOR POWER SECTOR PLANNING To be able to produce an unquestionable least-cost power sector expansion plan, consideration should be given to conducting a least-cost generation expansion study for the power network using software meant for that purpose (for example, WASP or PLEXOS), as opposed to the present practice of using the energy sector-oriented MARKAL/TIMES/ VEDA model and then “fine￾tuning” for the TYNDP through year-by-year simulations. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 20 With Armenian beneficiaries having been provided with the training and tools to undertake LCEDP analyses, interconnections with neighboring countries not previously included can be analyzed with “dummy entities” that take into account the potential electricity transit. PSRC should be allowed to request changes and monitor implementation of the TYNDP, preferably through the Electricity Law and changes to the Transmission Network Code the EU Directive emphasizes that the TYNDP must include, inter alia, the list of investments decided and to be executed over the next three years. EQ 2b. What were the key factors that enabled and/or hindered the achievement of outcomes? Which of these factors were within and outside of MLET’s control and manageable interests? FINDINGS Enabling factor. All KIs working in some way within the electricity market, including MTAI, SC, EPSO, all generators, and ENA showed a commitment to work on the market liberalization process. This was demonstrated by unanimously positive responses among these KIs to the question of what constitutes success. There were no suggestions about moving back or that the whole process may have been a mistake. Enabling factor. Development of the MMS was cited by three KIs (without any prompting) as having been a key enabling factor in market development. This is easy to comprehend as this is where all trades on the WEM are transacted. Enabling factor. A final key enabling factor, cited by all KIs, was the support, accessibility and responsiveness provided by MLET in answering questions and resolving problems, stretching all the way down to MPs, such as, small hydropower producers who were provided MLET contact information (a single producer who contacted MLET with a query was promptly answered). Training provided to all KIs was also unanimously cited as useful. Hindering factor. According to the MLET Annual Reports, GOAM restructuring at the beginning and re-examination of whether to proceed with both market liberalization and the Georgia interconnection adversely influenced the dynamics of the project. With the uncertain political situation, there were some difficulties in ensuring continued focus on and continuity of steps toward reform by key sector stakeholders, which affected GOAM although not to the extent that could endanger the majority of outputs. Hindering factor. A factor somewhat related to the above hindrance is the tendency of GOAM to act slowly on key action items required to move the reform process forward. Two examples of this are: the slow evolution of (i) the new Electricity Act, and (ii) formal policy adoption of SMRs to replace the ANPP. Hindering factor. The Annual Reports also cite COVID-19 as having been a challenge for the MLET activity, addressed by effectively organizing remote work during the lockdown period and developing a risk-mitigation plan and flexible work schedules afterwards. No severe adverse impacts were noted and MLET was able to manage this unforeseen change in the operating environment. Hindering factor. From MLET Annual Reports and two KI interviews, conflict in the Nagorno-Karabakh region—that is, the military emergency situation and subsequent uncertain political situation—affected GOAM involvement in the MLET activity, mainly due to GOAM shifting focus to the crisis, thus influencing the dynamics of the activity to a certain extent. However, no severe adverse impacts were noted and MLET continued to communicate with key stakeholders on issues related to the activity. The commitment by all local key “implementers” to move forward was/is a key factor in enabling market liberalization. FINDINGS , CONCLUSIONS, AND RECOMMENDATIONS 21 Hindering factor. The issue of Russian influence in Armenia arose only in one interview. It was recognized that Russian influence is strong and that certain Russian parties have been opposed to the market opening. An indication of this power is the Tashir Group, which owns several generating stations in Armenia and is planning to build more. Tashir Group also owns ENA (that is, the transmission/distribution company). Although Tashir Group was seemingly against the market liberalization at first, the greater worry now is the potential for market abuse. CONCLUSIONS The commitment by all local key “implementers” to move forward was/is a key factor in enabling market liberalization. Uncertainty around reforms were introduced at the beginning of the MLET activity with the GOAM turnover in elections and a subsequent reexamination of reform program. However, this doubt was addressed, to an extent, through commitment to the program by MTAI and sector stakeholders. Another significant enabling factor was the development and implementation of the MMS, which cannot be understated. All the work performed in arriving at a liberalized, functioning WEM is encapsulated in this trading platform. Finally, MLET’s support to all KIs interviewed can be summarized as being exceptional, with nearly all unresolved issues in any stakeholder’s mind answered. Negative influences on the activity, including COVID-19, Nagorno-Karabakh, and Russian influence, have not had major impacts. Slow movement on certain key action items by GOAM such as the Electricity Act and SMR adoption may be a source of some frustration but does not necessarily indicate abandonment of a particular policy. The best example is the adoption and implementation of the AEX, which did happen after a somewhat prolonged period of assessment by GOAM. RECOMMENDATIONS Transition to a completely liberalized market is not yet complete. To overcome possible future obstacles, a certain focus should be placed on the following: • Development of the new Electricity Law, which will fill gaps in existing legislation, such as the introduction of a market-based cross-border capacity-allocation methodology. • Continue market design to enable PSRC, SC, EPSO, generators, and consumers to take full advantage of new approaches. • Assist in the development and usage of market software. • Support PSRC in monitoring the regulated and unregulated market players. • Continue providing consultations and advisory services to PSRC, SC, EPSO, and other market participants to implement the new market rules and improve the functioning of the internal electricity market. • Capacity-building activities, including training, study tours to the extent possible to support PSRC, MOEINR, and market participants, and consumers. EQ 2c. To what extent was MLET able to steer its strategic approach in response to the unfolding of key geo-economic and sector developments during the implementation period? FINDINGS Key geo-economic and sector developments have been previously identified as hindering factors to which MLET was able to respond. For instance: The first half of 2019 saw a significant focus on the process of reorganization of GOAM (completed in June), but the MLET Activity was able to collaborate with key technical-level counterparts to advance needed actions directed towards market liberalization. For example, an MOU was signed in June between USAID, MTAI, and PSRC for support FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 22 for MMS software development. While this was being developed and negotiated, MTAI itself was being reorganized; however, the MLET team was able to ensure effective and timely commitment to the process by MTAI. In September 2020, GOAM had to declare another emergency in Armenia—a military situation and general mobilization—which entailed several restrictions, such as the mobilization of citizens, organization, and participation in rallies and strikes and dissemination of information on military operations. USAID was regularly updated on the status and outcomes of these developments and their potential to impact MLET activities. Snap elections in June 2021 resulted in a win for the incumbent party, but structural changes in GOAM remained a possibility for a certain period. Government reorganization after the elections may have affected market implementation by slowing the process. Adapting to the COVID 19 situation, described above. Although strong, Russian influence in the Armenian energy sector was never mentioned as a constraint or obstacle by any stakeholder. Presumed pressure from the Russian government resulted in the Hrazdan 5 Thermal Power Plant signing a 10-year power-purchase contract with GOAM in 2019. Although Russian interests appear to be against market liberalization, it has not caused MLET’s strategic approach to change, nor is there any indication that GOAM will change its desire for market-oriented reform. As previously mentioned, Russian-owned ENA has supported market liberalization. CONCLUSIONS MLET was able to strategically influence GOAM at the beginning of the project when it was restructuring and reassessing the market reform program. MLET was able to maintain this influence to help GOAM to stay the course towards market liberalization. As described earlier, MLE. T went through several challenges and emergency situations such as the COVID-19 pandemic and a war, successfully adapting by mitigating their effects and uncertainties through the various methods outlined above. EVALUATION AREA 3—EFFICIENCY EQ 3. To what extent did the intervention deliver results in an economic and timely way? FINDINGS As concluded above, the evaluation found MLET to be largely effective in the conduct of Tasks 1 and 3, and somewhat effective in Task 2. Based on conversations with USAID/Armenia, MLET is on track to have completed this work on time and within budget. In addition, it may be noted that for a five-year activity, planned activities can change. The most obvious in this case are the efforts to promote electricity trade with Georgia. If there had been an interconnection as originally envisaged, MLET would have devoted more time and resources to facilitating cross-border trade. As such, electricity trade did increase with a lower level of MLET input, with allocated resources transferred to other activities (according to the COR of USAID/Armenia). As mentioned previously, GOAM adopted a “go￾slow” approach to market liberalization, preferring a “Made-in-Armenia” solution, which has influenced the speed at which Armenia will transition to the terms of the CEPA. Thus, the efficiency of the process is lower, but is perhaps to the advantage of Armenia (as explained in the upcoming “impact” section of this report). With respect to the MLET activity, the development of a transition process has affected the efficiency at which the work could normally be carried out. However, MLET has thus far successfully executed the transition to the Made￾in-Armenia solution. The development of a unique MMS, as opposed to off-the-shelf software used extensively in EU￾compatible countries, illustrates the lack of efficiency FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 23 inherent in the Made-in-Armenia solution, more so by the fact that the MMS will eventually need to be abandoned in favor of a new off-the-shelf MMS. CONCLUSIONS As the intervention delivered results in an economic and timely way, the evaluation team found the MLET activity to be largely efficient. Inefficiencies in implementing the liberalized market were due to factors beyond MLET’s control, such as, the longer time it took GOAM to implement the made-in￾Armenia solution; and the inefficiency of the WG initially set up to draft the Electricity Law. RECOMMENDATIONS Future USAID efforts in electricity market liberalization in Armenia may consider trying, to the extent possible, to streamline or even entirely avoid Made-in-Armenia solutions as they can be costly; for example, development of custom MMS software, as opposed to purchasing off-the-shelf software, will, in the end, be more costly because it will eventually need to be either upgraded to EU standards or scrapped entirely in favor of off-the-shelf software. This would have been apparent through a cost￾benefit analysis conducted upfront. EVALUATION AREA 4—IMPACT EQ 4a. To what extent did the intervention generate or is expected to generate significant positive or negative, intended or unintended, higher-level effects? FINDINGS The state of the Armenian electricity market since its formal opening in February 2022 is presented in figure 2 below. It is worth noting that the market shares of Qualified Customers and Suppliers is expected to increase from February 2023 due to improvement in the market as a result of formally introducing the balancing mechanism. In February 2023, electricity sales to Qualified Customers and Suppliers reached 7.8 percent and 4.5 percent respectively, while electricity sales by ENA were 87.6 percent of the total. Responses on impact-related effects were received from 13 of the stakeholders interviewed (counting the FGD as one). Several stakeholders cited multiple effects. POSITIVE EFFECTS • Large/“huge” impact in moving the market liberalization process forward • Transparency, fairness (specifically in prices), less disputes • Better prices • New professional opportunities • Greater awareness of electricity supply • Better planning/certainty • Movement to “best international practice” • Improved safety • Better power system discipline • More stimulating discussions on tariffs. NEGATIVE EFFECTS • More “work”/problems to resolve • Prices could potentially go up in the future. It may be noted that every respondent who cited a negative effect also provided at least one positive effect. All “more work” comments (comprising the majority) except for one were given as a statement of fact, and not as a complaint. Only one respondent complained about the extra work. Regarding the “better prices” responses, this came from a sample of two large consumers and four producers. This includes FGD participants as a single producer group of seven, all of whom were unanimous in their view of better prices. Thus, the evidence is overwhelming that producers on the market are receiving better prices. The fact that two producers did not mention “better prices” is due to the fact that all, or most of the production from these producers, is at a regulated price. On the consumer side, the sample was less robust, but it showed that the two large consumers have FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 24 experienced lower bills. This apparent dichotomy of both sides getting “better” prices may be explained by the fact that large customers whose tariffs were previously regulated were overpaying in relation to the cost of supply (and thereby cross-subsidizing residential customers), while producers were being underpaid relative to the market value of their production. On a task-by-task basis: Task 1, Energy Market Development, has changed Armenia’s energy market significantly, from a single￾buyer regime in 2018, where all energy was sold to ENA, to a partial wholesale market in February 2023, in which 12 to13 percent of all electricity sales are conducted. Even with this relatively modest proportion of total sales on the WEM (illustrated in figure 3), the electricity market structure has changed, and will continue to change until a significantly larger proportion of total sales originates on the WEM. Given that Armenia wishes to integrate with the EU market, the effect would be positive. On the other hand, market concentration as measured by the Herfindahl–Hirschman Index (HHI)10 is currently very high, with the current WEM having an HHI of about 7600. Section 6 of the Retail Electricity Market Trading Rules and Contracts lays out transitional provisions for the WEM up to February 1, 2025, which progressively lowers the minimum voltage level of supply for Qualified Customers but maintains a minimum required level of one million kWh consumption annually. Given this high threshold and current level of the HHI, it might be difficult to lower the HHI significantly so that it shows an efficient WEM with a sufficient number and 10 The Herfindahl–Hirschman Index is a commonly accepted measure of market concentration, calculated by squaring the market share of each firm competing in the market and then summing the resulting numbers. It takes into account the relative size distribution of the firms in a market. The index approaches zero when a market is occupied by a large number of firms of relatively equal size and reaches its maximum of 10,000 points when a market is controlled by a single firm. The HHI increases both as the number of firms in the market decreases and as the disparity in size between those firms size of MPs. This concern may be mitigated through increased trade with Georgia, as long as the exporters are not affiliated with the Universal Supplier (ENA). It may be useful to carry out an exercise that reveals HHIs at various threshold levels for deregulated consumers together with various scenarios of electricity trade. FIGURE 3. ARMENIAN ELECTRICITY MARKET DEVELOPMENT (% OF SALES) Another potential negative effect is on tariffs, which all stakeholders seem to be ignoring for the time being because tariff changes have thus far been favorable for all market participants. Only one stakeholder pointed out that tariffs might eventually increase. Tariff increases might arise from two easily identifiable sources: (i) elimination of cross-subsidy from “over-paying” consumers to “under-paying” consumers, meaning that the current regulated subsidy to under-paying consumers will need to be financed somehow in the absence of higher tariffs; and (ii) market forces, as in Europe over 2022. increases. Agencies such as the US Department of Justice and the Federal Trade Commission generally consider markets in which the HHI is between 1,500 and 2,500 points to be moderately concentrated, and consider markets in which the HHI is in excess of 2,500 points to be highly concentrated. Transactions that increase the HHI by more than 200 points in highly concentrated markets are presumed likely to enhance market power. Formula: HHI = S1 2 + S2 2 + S3 2+ …+ Sn 2 where Sn is the market share percentage of firm n expressed as a whole number, not a decimal 0% 20% 40% 60% 80% 100% 120% 1/23 2/23 3/23 4/23 5/23 6/23 7/23 8/23 9/2310/2311/23 Qualified Customers Suppliers ENA Customers FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 25 Potential negative impacts may arise from the following factors that do not appear to have been addressed yet: • No legal barriers exist for the creation of suppliers affiliated with ENA. Any affiliation with ENA may create a notable advantage for a supplier, violate fair competition, and further increase the real HHI. The current ownership structure of the Armenian power supply industry makes this a particularly valid concern. • Currently the Energy Law has a confidentiality requirement for the licensees but does not provide for a separation of confidential data between the distribution system operator (DSO) and a supplier. Impacts of Task 2 are relatively modest compared to Task 1. The main impact is the likely future adoption of SMR technology (as mentioned previously). Although GOAM’s strategic plan explicitly mentions extending the life of the ANPP to 2036 with no mention of SMRs, this option has been considered seriously in both the LCEDP and by GOAM. The Task 2 activity also included updating the MARKAL/TIMES model, which is used to develop the LCEDP in Armenia. The updated model now allows a better analysis of the deployment of VRE technologies such as solar, which can be a challenge given RE production characteristics. The impact of Task 3 is difficult to assess because the remarkable increase in cross-border trade may be more due to Task 1, which has allowed Armenian traders to take the initiative to transact with parties in Georgia. On the other hand, the impact of continuing the JWG meetings and discussions under Task 3 cannot be discounted as having an influence on the increase in cross-border trade. The fact that funding was transferred from Task 3 to Task 1 over the course of the MLET activity in response to a lack of progress on the Georgia interconnection might mean that the impact of higher cross-border sales was due more to Task 1 activities. CONCLUSIONS The paramount achievement of MLET is that a part of the Armenian electricity market is liberalized and operational. MLET succeeded in equipping the Armenian power sector with a set of necessary regulations and tools to enable operation of the market in its current development stage. As might be expected at this early stage, with Qualified Customers restricted to those taking power at high voltage level and having annual consumptions in excess of one million kWh, market concentration by a few customers is very high. However, this situation is expected to change after the transitional provisions (as laid out in Section 6 of the Retail Electricity Market Trading Rules and Contracts) expire and/or with increased electricity trade with Georgia. Current regulations allow for a relatively high potential for market abuse, most notably by the Universal Supplier (ENA). This factor may be exacerbated under the current ownership structure of the Armenian power industry, where the DSO and several generating sources have the same owner. From a stakeholder perspective, positive effects of the MLET activity, both realized and expected, far exceed the actual and possible expected negative effects. For the time being, there appears to be a certain satisfaction with how events are unfolding based on stakeholder responses. Perceived effects are overwhelmingly positive. Implementation of the wholesale market under Task 1 is fundamentally changing the structure of the electricity market in Armenia, a positive effect given that this is necessary for Armenia to participate in the European market. A potential negative impact of Task 1 is higher tariffs in the long term. Task 2 (planning) had a more modest impact on sector planning and, as previously mentioned, the methodology could be further improved in The paramount achievement of MLET is that a part of the Armenian electricity market is liberalized and operational. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 26 accordance with best practices for long-term planning of the power sector. Task 3 (cross-border trade) taken together with Task 1 has had a very positive impact. RECOMMENDATIONS Unless moderate market concentration can be achieved after February 1, 2025, additional measures may be required, such as lowering the threshold value of annual consumption for consumers of the deregulated market from one million kWh. Assistance may be required to further analyze how market concentration, or its effects, may be mitigated. The DSO must be obliged to preserve the confidentiality of commercially sensitive information to eliminate potential favoritism to affiliates. In assisting with drafting appropriate regulations, USAID can ensure that such a provision is included. EQ 4b. To what extent did the MLET interventions move forward Armenia’s desired energy sector reforms? FINDINGS Responses to the question “What would have happened without the technical assistance? Would it have happened anyway?” were received from nine stakeholders, as certain stakeholders such as producers and consumers were not really in a position to provide an informed answer. Without MLET, stakeholders answered that: • There would have been little or no change from the status quo in 2018, which is when the MLET activity began (seven responses). • Assistance to proceed with the reforms would have come from elsewhere, albeit resulting in a slower pace in moving the process forward (two responses). This is probably due to USAID’s prior involvement in the energy sector (going back 10 years), resulting in a certain familiarity with the sector as well as trust built up over that period with stakeholders. The sole reason given for the above responses is that there is a distinct lack of capacity in Armenia to undertake power system reform and to do it rapidly. It may also be considered that liberalized markets, such as those in Europe, have been evolving over 30 years (since the early 1990s when concepts of unbundled power sectors and third-party access to power systems were first introduced). Outside assistance to “catch up” to decades of reforms is required to move forward the reform agenda. It should be considered that USAID has been the only international donor agency involved in the Armenian power sector that is focusing on electricity market reforms. Other donors are active in the power sector, but they have tended to provide funding for infrastructure projects. The project that best compliments MLET is the current plan promulgated by KfW (Kreditanstalt für Wiederaufbau, Germany) to construct transmission facilities capable of eventually carrying 1050 MW to Georgia. However, to benefit from the use of these facilities, an enabling environment must exist, which is starting to develop with the MLET project. It is very telling that cross-border electricity trade with Georgia in 2022 exceeded all electricity trade over the previous ten years, all due to the MLET program allowing traders for the first time in Armenia to also explore cross-border opportunities. Thus, the two projects complement each other—the KfW project needs MLET to stimulate cross-border trade to take advantage of the capacity on the transmission line, while MLET needs the facilities to be able to realize the goal of considerably expanding cross-border trade. It may well be that another donor will move into the MLET space if it is vacated by USAID, but this would take time to materialize. In the meantime, MLET has been moving the reform agenda forward at a pace that would be difficult to replace and at a point in time that coincides with unfolding events such as the movement towards EU power market, integration by Georgia, and the development of infrastructure facilities that will allow for this to happen. It may also be noted that Armenia’s “go-slow” approach to market liberalization appears to match the schedule of the often-delayed transmission link FINDINGS, CO NCLUSIONS, AND RECOMMENDATIONS 27 to Georgia—or may be ahead now, as a distinct possibility exists that capacities demanded for transfers to/from Georgia will soon exceed the line￾carrying capacity of existing facilities. CONCLUSIONS MLET has moved Armenia’s desired energy sector reforms forward at a pace that is consistent with events such as the movement towards EU power market integration and a substantial transmission interconnection with Georgia. In the absence of MLET assistance, it is largely perceived by stakeholders that power sector reform and market liberalization in Armenia would not have advanced beyond the single-buyer model of 2018, or at most, at a very slow pace. EVALUATION AREA 5—SUSTAINABILITY EQ 5a. Which elements of the intervention approaches and results achieved by MLET are most likely to be taken further and sustained beyond the close of the program? Why? FINDINGS The extent to which the intervention approaches and results achieved by MLET can move forward and be sustained may depend on several factors: ELECTRICITY LAW The Armenian power sector is currently governed by the Energy Law of 2001, which has since become outdated, as evidenced by the more than 3011 amendments that have been passed since then. Although the existing Energy Law with all amendments and anticipated transitional provisions has allowed for liberalized market operations, it does not provide for sustainable development of the liberalized market, nor does it meet the EU market requirements. The new Electricity Law was drafted and submitted to GOAM for review and comments. According to MLET, the draft law is fully harmonized with EU 11 https://www.arlis.am requirements. It is expected that the internal discussions with GOAM, energy sector entities, and stakeholders will be concluded by the end of 2023 and the draft will be submitted to the parliament in early 2024. EVOLUTION OF EU ELECTRICITY MARKET DESIGN In the wake of the difficulties in the EU energy market seen in 2022, with particularly high and volatile prices and serious concerns about the security of supply, EU heads of government called on the European Commission to work swiftly on structural reform of the electricity market. The structural reform has the dual objective of securing European energy sovereignty and achieving climate neutrality. Following a public consultation in early 2023, the Commission presented a proposal on March 14 to revise the rules for electricity market design and to improve EU protection against market manipulation in the wholesale energy market. It aims at making the EU energy market more resilient and for the energy bills of European consumers and companies to be less dependent on fluctuations in the short-term market price of electricity. This can be accomplished through longer-term contracts, such as power-purchase agreements, and appropriately structured investment support. The aim is not only to better protect consumers and to accelerate deployment and improved integration of renewables, but also to enhance protection against market manipulation, ensuring stability and predictability of the cost of energy and contributing to the competitiveness of the EU industry. The proposal has now passed to the Council and the European Parliament for debate and negotiation under the normal legislative procedure. The above development comes in the wake of formal adoption by the EU of the Fourth Package in 2019 and of a proposed Fifth Package in 2021, while CEPA extends only up to the Third Package. These new EU requirements will need to be adopted by all countries acceding to the EU electricity market. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 28 Thus, the issue then becomes not whether the results of the MLET interventions can be sustained, but whether the results have sufficient in-built dynamism to continue evolving alongside EU developments. MMS SOFTWARE Specific MMS software was created for the AEX rather than purchasing the available off-the-shelf products compatible with ENTSO-E requirements.12 While providing for the current transitional phase of the AEX, the MMS software will likely fall short in the next stages of market development. The AEX MMS software does not fully meet EU requirements and standards, nor has it been tested for interoperability with the MMS software being used in Europe or the Nord Pool software in Georgia. ENTSO-E has supported its members in MMS development.13 MARKET MONITORING A “Review of the Electricity Market Rules and Market Monitoring Solutions” was prepared by MLET in June 2020. The Review identifies and recommends solutions to key issues concerning the monitoring framework pertinent to the proposed market rules and energy law. As such, the review evaluates the tools that the current legal and regulatory framework in Armenia entrusts to PRSC, SC, EPSO, transmission owner, and DSO. However, proper market monitoring mechanisms have not yet been implemented. CONCLUSIONS A legally binding framework and the “rule of law” are fundamental for the sustainability of market liberalization. Sustainable liberalized market development and further integration of regional markets cannot be guaranteed without adoption of the new Electricity Law and respective changes to the legal framework. For this, adoption of the Electricity Law will likely sustain reforms well into the future. Without this law, there is a risk that the 12https://www.ge.com/digital/applications/transmission/advan ced-market-management-system-amms lack of guidance and support by independent experts can result in a final version that is distorted by interference of market liberalization opponents from within GOAM or from elsewhere. With respect to the evolution of the EU electricity market, the conservative, phased transition of the Armenian market in recognizing the realities in Armenia seems to have served the country well. While justifiably limiting expectations, the selected approach has allowed for achieving timely realistic goals in transitioning a segment of the Armenian electricity market from a single buyer model to direct contracts. One advantage of the phased transition approach is that it has allowed for certain flexibility for further development in case of changes to the EU and/or Georgian markets. MMS interoperability and data interchange will be of considerable importance for further integration of Armenian and Georgian electricity markets. New MMS software or a considerable overhaul of the existing MMS will be required for this. Otherwise, this component of market liberalization is not likely to be sustained. Appropriate market monitoring has yet to be implemented in full and certain capacity building in this area is still required. Further effort is required to develop and implement the market monitoring mechanisms and tools put forth in MLET’s Review of the Electricity Market Rules and Market Monitoring Solutions. RECOMMENDATIONS Supporting GOAM to adopt EU requirements with a compliant Electricity Law should be continued. This will ensure a sound legal basis for sustainable development of the liberalized market. Future assistance should continue monitoring the development of the EU electricity market model and rules with the goal of supporting GOAM with 13 https://www.entsoe.eu/digital/common-information￾model/cim-for-energy-markets/ FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 29 possible timely amendments to the adopted legislation. Support will likely be required at some point in transitioning the current MMS to a compatible European model. As significant changes were implemented after the preparation of MLET’s Review of the Electricity Market Rules and Market Monitoring Solutions, a gap analysis of the status of market monitoring may be considered (that is, compare the current status with what was recommended by MLET in 2020). Also, market monitoring mechanisms may be further developed to meet the requirements of Regulation (EU) 2019/943. EQ 5b. What are the political, geopolitical, and geo-economic risks most likely to impact the sustainability of the results of the MLET program? Who are the supporters and spoilers of continued market liberalization, increase in the uptake of renewable energy technologies, and increased electricity trade, including regional? FINDINGS Over the course of the MLET activity implementation, two elections were held—in 2018 and in 2021. The first election in December 2018, was the most crucial for MLET, as it resulted in a new government being formed, which chose to carefully examine and reassess policies implemented by its predecessor. Throughout the second half of 2018, despite a clear political focus on preparing for elections in December and the subsequent proposed reorganization of government, key GOAM counterparts for the MLET activity at MTAI, SC, EPSO, and PSRC were able to maintain momentum to achieve all deliverables planned in the GOAM Action Plan in a timely fashion. Similarly, while the first half of 2019 saw significant focus on the process of reorganization within GOAM and a certain 14 MLET’s 2020-2021 Annual Report expresses this concern, but there is no mention of this at all in the 2021-2022 Annual Report. amount of reflection on whether/how the reform program should be carried out, MLET was still able to collaborate with key technical and policy-level counterparts to advance actions toward market liberalization. A positive example of this is the MOU signed in June between USAID, MTAI, and PSRC on support for MMS software development. While this was being developed and negotiated, MTAI was being reorganized. The MLET team was still able to execute its planned activities. After certain civil turmoil arising as a result of the war with Azerbaijan in 2020, snap elections were called. The elections resulted in a return to power of the incumbent government. Although MLET was expecting certain structural changes in GOAM to possibly affect its activities because of the election, this did not happen.14 Over the course of the MLET activity, structural changes of the government have continually been identified as risks for implementation of the market, or at least slowing down the process. Armenia is a member of the Eurasian Economic Union, comprising Russia, Armenia, Kazakhstan, Kyrgyzstan, and Belarus. This relationship may cause some concern, but it was never brought up as such by any KI. Based on interviews with GOAM, PSRC, EPSO, and SC, support for MLET has been very strong among all market “implementers,” with not one dissenting voice among any of them. Similarly, all MPs, including generators and consumers interviewed, have also been unanimously positive about the market changes, although this might have more to do with the better prices thus far. A crucial unknown, however, is the public/smaller consumers who make up the vast majority of customers, and whose tariff will fall under the regulated regime until at least February 1, 2025. In interviews, some KIs noted a lack of public knowledge about market liberalization. This was first FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 30 recognized by MLET in the 2020-2021 Annual Report, in which it was noted that only government agencies and energy sector companies really understood the upcoming electricity market reforms and that other groups, including customers, were largely unaware of the changes. Accordingly, MLET initiated a communications campaign, set to begin early in 2022, to increase public awareness of Armenia’s electricity market reforms. A range of media tools such as television broadcasts, short videos, infographics, and animations, as well as training of target audiences were planned for this task. The communications campaign was undertaken late 2022/early 2023. It is too early to gauge the effectiveness of this task relative to the KI interview period in February 2023. The communications campaign, if conducted well, could be a significant source of public support (or lack thereof). Although no other significant sources of “spoilers” of reform could be identified during interviews, brief mention was made in single interviews of the following possibilities: • GOAM technocrats. For example, GOAM technocrats, trained in Russian nuclear technologies, may feel threatened with the introduction of western-style SMRs. • Other vested interests in the power industry might stand to lose in the market “shake-out”. The motive might be financial (for example, Tashir Group) or it might concern employees of certain entities concerned that they may lose their jobs. CONCLUSIONS Although structural changes in GOAM have continually been identified as risks for implementation of the market, no evidence exists that this has had an effect, except perhaps for a brief period in 2018 when GOAM changed and the desirability of the MLET Activity was closely reexamined. However, MLET was able to proceed with its agenda and eventually gained GOAM’s full support. The relatively friendly relationship of GOAM with Russia was never cited as a concern. This can be attributed more to Armenia’s geographic position and having to get along with its neighbors. No other significant political risk nor potential spoiler of the MLET Activity’s results could be identified. RECOMMENDATIONS Although MLET has successfully navigated several adverse situations, given the somewhat volatile political situation in Armenia over the past decade and Armenia’s geographic position, the political situation should be closely monitored in future projects so that adverse events can be anticipated, and USAID/project implementers can act quickly to adjust and mitigate their effects. LESSONS LEARNED 31 LESSONS LEARNED TASK 1 Based on experience to-date, the Made-in-Armenia electricity market solution appears to be serving the country well, with successful implementation of successive transitional phases on February 1, 2022, and February 1, 2023. However, this lesson learned comes with the caveat that the move to a fully liberalized market in line with EU standards is far from complete, as only 12 to 14 percent of total sales are presently taking place on the WEM. In drafting legislation, large working groups do not appear to work well. Before switching to an alternate modus operandi, the formation of a large working group to undertake the task of drafting the new Electricity Law caused significant delays. TASK 2 The development of an optimal least-cost development plan for a power sector should be undertaken using more conventional techniques than first employing a rather high-level energy model for the entire energy sector and then trying to “fine￾tune” for the power network through a year-to-year simulation. This casts an element of doubt on the results. TASK 3 Basing a substantial portion of a program budget on the assumption of completion of a major infrastructure project can potentially have negative consequences if the infrastructure project does not materialize. Fortunately, in the case of MLET, funds were easily reallocated because of its large size and different components under the activity. ANNEXES 32 ANNEXES ANNEX I: TIMELINE 33 ANNEX I: TIMELINE November December January February March April May June Activity 12 19 26 3 10 17 24 31 7 14 21 28 4 11 18 25 3 10 17 24 31 7 14 21 28 5 12 19 26 2 Preparation Kick-off meeting with USAID Desk review of available documents Prepare Evaluation Design and Work Plan Submit Draft Evaluation Design and Work Plan to USAID Finalize Evaluation Design and Work Plan Submit Evaluation Design and Work Plan to USAID Data collection Identify and contact participants for KIIs Conduct virtual KIIs Conduct KIIs in Armenia Interim meeting with USAID in Yerevan– data collection update Out brief presentation in Yerevan Data analysis and report preparation Analyze data Prepare Draft Evaluation Report Submit Draft Evaluation Report to USAID Conduct Presentation to USAID on the Draft Evaluation Report to discuss feedback Incorporate feedback Submit Final Evaluation Report Draft final evaluation PowerPoint slide deck* Submit final evaluation PowerPoint slide deck* *Timing will depend based on the feedback received on Revision 1 of the Draft Evaluation Report. The Evaluation Brief and the Final Deck Presentation will be drafted once the Draft Evaluation Report receives approval from USAID. ANNEX II: EVALUATION STATEMENT OF WORK 34 ANNEX II: EVALUATION STATEMENT OF WORK Introduction USAID/Armenia intends to conduct a final performance evaluation of its Market Liberalization and Electricity Trade (MLET) Program implemented by Tetra Tech ES, Inc. under the Contract #: 72011118C00001. The Activity runs from June 6, 2018, through June 5, 2023 with a total estimated cost of $9,989,195. The award is administered by USAID/Armenia. The purpose of the final evaluation is to: • Determine the degree to which MLET achieved its intended goal and objectives, specifically how it contributed to power market liberalization, electricity supply diversification, and expanded electricity trade, and; • Capture key lessons and provide conclusions that will be applied by USAID/Armenia in the design and/or implementation of future activities and/or be used for directions the mission may wish to explore regarding energy security. The evaluation will focus on assessing MLET’s performance against the following tasks: • Contribute to electricity market liberalization • Promote energy supply diversification • Facilitate cross-border trade with Georgia USAID/Armenia will use the findings, analysis and conclusions of this evaluation to inform future programming to help effectively achieve U.S. government strategic objectives in Armenia. Additionally, USAID will share the evaluation report with the key stakeholders: the Government of Armenia (GOAM), Ministry of Territorial Administration and Infrastructure (MTAI), Public Service Regulatory Commission (PSRC), and energy institutions. Background Over the past decade, USAID/Armenia has facilitated significant progress in macro-level reforms, including unbundling of the electricity and gas sectors, creation of an independent regulator, penetration of substantial renewable energy resources, large-scale privatization of electricity generation and distribution assets, improving the quality of services, and overcoming various difficulties that the country’s energy sector faced following independence in 1991. owever, Armenia remains energy insecure and is among the most dependent on energy imports in the region. Through the five-year Market Liberalization and Electricity Trade program, USAID aimed to assist in developing and implementing liberalized market rules, which will support energy market compliance with western approaches and facilitate the electricity trade with Georgia. To build off of USAID’s previous efforts, the contractor supported the Government of Armenia in the development of Armenia’s Energy Strategy and Action Plan, which were based on the Least Cost Energy Development Plan conducted by MLET. The project also promoted Market Liberalization and Promotion of Cross-Border Trade, as well as assisted in the development of Regulatory Road Map, and PSRC’s concept paper. ANNEX II: EVALUATION STATEMENT OF WORK 35 To make energy market reforms successful, this contract focused on assisting in the establishment of the necessary legal regulatory framework, creating market rules, procedures, and software, and providing necessary training and consultancy especially to the newly established institutions like Market Operator, as well as those carrying out new functions like System Operator to support a smooth transition to an open market. The contract also supported the Armenia-Georgia Electricity Working group, harmonization of regulatory approaches with those in Georgia, and the development of trade mechanisms that are critical for promotion of electricity trade. Overall, the contract assisted in the creation of an investor-friendly environment to accelerate competitiveness, power sector development, and regional trade and, in a broader context, improve the country’s economic development and minimize the potential for corruption. By supporting the implementation of Armenia’s Energy Strategy and Action Plan for Market Liberalization and Promotion of Cross-Border Trade, through the development and harmonization of regulatory approaches, Armenia will successfully create a sustainable liberalized power market and expand electricity trade with Georgia. The specific tasks and subtasks under the MLET are: • Task 1: Energy Market Development will ensure that legal and regulatory reforms to support electricity market liberalization and an effective electricity trading mechanism are developed and that Armenia’s Government, including MTAI and Market Operator (MO), as well as PSRC develop and implement a liberal electricity market model for Armenia. The MLET Activity will work to strengthen the MO and System Operator (SO) and Market Participants (MPs) by supporting continued revision of market operating procedures and advisory services to MO and MPs. • Task 2: Promote Energy Supply Diversification will support MTAI’s efforts to transition Armenia’s Least- Cost Energy Development Plan (LCEDP) model from the MARKAL software to TIMES, update the LCEDP to incorporate recent changes in renewable energy and nuclear technology, and update the Armenian 2036 Energy Strategy and Action Plan to reflect the findings of the newly updated LCEDP. • Task 3: Facilitate Cross-border Trade will strengthen Armenia-Georgia electricity dialogue through support to the Joint Working Group, promote export and import transactions in wholesale markets, support regional studies on electricity generation and transmission, assist in the development and implementation of non-discriminatory cross-border access to all eligible entities, and support development of cross border trade mechanisms. Evaluation Questions The following are key lines of inquiry the evaluation should address. Evaluation questions will be finalized in collaboration with the evaluation team when the evaluation design is finalized: • Coherence — To what extent and in what specific ways did MLET contribute to USAID’s strategic intermediate result “energy security increased”? Why? ANNEX II: EVALUATION STATEMENT OF WORK 36 — How has the activity contributed to wider efforts of GOAM to increase energy security in Armenia? How significant and/or relevant were these contributions? • Effectiveness — ow effective has MLET been in achieving intended outcomes of developing Armenia’s energy market, promoting energy supply diversification and facilitating cross-border trade? Which interventions have been most and least effective in achieving the intended outcomes and why? — What were the key factors that enabled and/or hindered the achievement of outcomes? Which of these factors were within and outside of the MLET’s control and manageable interests? — To what extent was MLET able to steer its strategic approach in response to the unfolding of key geo-economic and sector developments during the implementation period? • Efficiency — To what extent did the intervention deliver results in an economic and timely way? • Impact — To what extent did the intervention generate or will generate in the long run significant positive or negative, intended or unintended, higher-level effects? — To what extent did the MLET interventions move forward Armenia’s desired energy sector reforms? • Sustainability — Which elements of the intervention approaches and results achieved by MLET are most likely to be taken further and sustained beyond the close of the program? Why? By who - at institutional and champion levels? — What are the political, geopolitical and geo-economic risks most likely to impact the sustainability of the results of the MLET program? Who are the supporters and spoilers of continued market liberalization, increase in the uptake of renewable energy technologies, and increased electricity trade, including regional? Methodology It is anticipated that a mix of methodological approaches including quantitative and qualitative methods will be needed to answer the evaluation questions outlined above and ensure multiple levels of triangulations. The emphasis will be on collecting reliable empirical data and/or objectively verifiable evidence, as opposed to anecdotal evidence. Suggested data collection methods include: • Desk Review: The evaluation team will conduct a desk review of the available documents including background documents, MLET work plans, performance monitoring plans, and reports, relevant GOAM policy documents, and third-party research reports. • Data Analysis: The evaluation team will suggest a robust quantitative and qualitative data analysis plan with emphasis on how FGD and KIIs will be transcribed and analyzed to generate findings and draw ANNEX II: EVALUATION STATEMENT OF WORK 37 conclusions. The analysis plan should include illustrative versions of the tables and graphs that will be produced. The plan should be comprehensive enough to provide analysis detail for each question. • Key Informant Interviews (KII) and Focus Group Discussions (FGDs): The evaluation team will conduct in-depth interviews and group discussions with project staff, partner organizations, stakeholders, development partners, non-government organizations, and other community members on their views and perceptions on MLET and the kind of changes that have resulted from the program intervention. Evaluation Limitations The offerors must disclose any limitations to the evaluation and how they plan on mitigating them. Summary Evaluation Design Matrix The evaluation design matrix should include a data analysis plan for each evaluation question for evaluators to complete. Evaluation Questions Data Source(s) Data Collection Methodology Data Analysis Methodology Current MLET indicators that would help answer the evaluation question Deliverables USAID/Armenia anticipates that the evaluation will take approximately 12 weeks to complete. This includes approximately five weeks of desk research, three weeks in-country followed by approximately four weeks in the evaluator organization’s eadquarters/ ome Office completing the draft and final reports. Offerors should provide a detailed timeline with their proposal. The evaluation team will serve under the technical direction of USAID/Armenia’s MEL Specialist who will also act as the USAID activity manager of this evaluation. Coordination of all meetings with GOAM Representatives and implementing partner staff will be accomplished through USAID/Armenia staff. The evaluation team will arrange visits to the interview sites in consultation with the USAID Activity manager. A list of key stakeholders and partners that the evaluation should interview will be provided and these interviews should be conducted in-person whenever possible. USAID will facilitate introductions of key stakeholders selected for interviews to the evaluation team. The following deliverables are required: • Kick-off meeting with USAID/Armenia: At the launch of the Evaluation, the evaluation team will have a virtual kick-off meeting with USAID/Armenia for introductions and to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan. The in-brief meeting between USAID and the evaluation team will allow both parties to clarify evaluation expectations. • Evaluation Design: Within 15 business days following the kick-off meeting, the evaluation team will deliver an evaluation design plan that describes the conceptual framework for the evaluation and the ANNEX II: EVALUATION STATEMENT OF WORK 38 justification for selecting that approach, detailed methodology, as well as the final work plan that details key evaluation tasks, timeline, and data collection instruments. The plan will also include a brief section on the intended audience for the final evaluation and a dissemination plan for the evaluation findings. USAID/Armenia must provide its approval of the evaluation design proposal before the evaluation team begins in-country data collection. • Interim Meeting: During the in-country data collection phase, the evaluation team will organize a briefing session to provide USAID/Armenia with an update on data collection progress and discuss potential challenges and emerging opportunities. If necessary, additional virtual briefings may be arranged. • Out brief/Presentation of Preliminary Findings: The team will make a presentation of key preliminary findings of the evaluation to USAID/Armenia at the close of fieldwork and before the team departs Armenia. The debriefing must include a discussion of findings and conclusions. The evaluation team leader will share the draft PowerPoint slides for USAID review prior to the presentation. • Draft Evaluation Report: Within 15 days of the presentation of preliminary findings, the evaluation team will submit the draft evaluation report to USAID/Armenia. The report should separately and comprehensively address each of the objectives and evaluation questions listed in the Scope of Work as well as the findings and conclusions, which should be clearly supported by the collected and analyzed data. Findings should be presented graphically where feasible and appropriate, using graphs, tables and charts. The draft evaluation report must contain at least the following: • An evaluation abstract (not to exceed half a page) that discusses the evaluation purpose, key questions, and key findings and conclusions. • Executive Summary: This section should be up to five pages in length and describe the purpose, activity background, evaluation design and methodology including the evaluation questions, and key findings and conclusions. The executive summary should accurately represent the report as a whole. • Background: This section will provide a brief description of MLET activity that highlights its scope, design features including the Results Framework and development hypothesis, and activities undertaken. It will also describe the purpose of the evaluation and the evaluation questions to be answered. • Evaluation Design and Methodology: This section will detail the overall evaluation design and methodology and related research protocols undertaken in conducting the evaluation, including the relevant data collection and analysis methods, sampling approach, and related challenges or limitations encountered during the evaluation and mitigation approaches employed. • Findings: This section will present findings collected from the evaluation relevant to each evaluation question. The evaluation findings must be presented as analyzed facts, evidence, and data and not be based on hearsay. The findings must be specific, concise, and supported by the quantitative and/or qualitative evidence analyzed through scientifically plausible methodologies. • Lessons Learned: To serve the objective of USAID/Armenia to utilize the evaluation to inform its future programs, the evaluation team will develop a section on lessons learned from the implementation of the MLET. This section will describe both what went well and what could be ANNEX II: EVALUATION STATEMENT OF WORK 39 replicated in a follow-on project, as well as what went wrong and what lessons could be applied into the follow-on project. • Conclusions: The evaluation report will present evaluation conclusions that are interpretations and judgments based on the findings described and must logically follow from the gathered data and findings and be explicitly justified. If necessary, the evaluation team will state its assumptions, judgments, and value premises in presenting a conclusion so that readers can better understand and assess them. Final Report: Following receipt of all USAID comments on the draft evaluation report, the evaluation team will have 10 days to prepare and submit a final version that incorporates and responds to USAID/Armenia feedback. The final evaluation report should contain the same sections as noted above for the draft evaluation report and should also include: — References: This section should include a list of all documents reviewed, including background documentation. — Annexes: All the annexes listed in the draft report description above will be included in the final evaluation report. The evaluation report should be formatted in accordance with USAID’s general branding guidelines and meet the requirements described in ADS 201, USAID Evaluation Report Requirements, the How to Note on Preparing an Evaluation Report (https://usaidlearninglab.org/sites/default/files/resource/files/how￾to_note_preparing_evaluation_reports.pdf) and Criteria to Ensure the Quality of an Evaluation Report (ADS 201maa Criteria to Ensure the Quality of the Evaluation Report A Mandatory Reference for ADS Chapter 201 (usaid.gov). All members of the evaluation team should be provided with USAID’s mandatory statement of the evaluation standards they are expected to meet. Finally, the evaluation team will upload a copy of the final evaluation report in English, to USAID’s Development Experience Clearinghouse (DEC) within 90 days of COR approval to post it on the DEC. Final Presentation: The final report is to be accompanied by a virtual PowerPoint presentation that aims to debrief selected stakeholders of the findings and conclusions from the evaluation. A draft of the final deck should be submitted to USAID/Armenia prior to finalization and the virtual presentation. ANNEX III: EVALUATION METHODS AND LIMITATIONS 40 ANNEX III: EVALUATION METHODS AND LIMITATIONS Detailed Methodology and Data Collection and Analysis Plan This section presents the conceptual approach and data collection and analysis plan by evaluation question. COHERENCE • To what extent and in what specific ways did MLET contribute to USAID’s strategic intermediate result “energy security increased”? • How has the activity contributed to wider efforts of GOAM to increase energy security in Armenia? How significant and/or relevant were these contributions? CONCEPTUAL APPROACH To answer the first question, USAID’s energy strategy for Armenia will be reviewed, through an examination of project documentation and interviews with USAID Mission personnel. Current energy security as per criteria defined by USAID will be compared to the situation at project inception and also with respect to current plans in Armenia to grow and diversify energy supply. To answer the second question, accomplishments in each of MLET’s task areas will be assessed against perceived increases in energy security since 2018 arising from the first question. Also, GOAM will be queried about its overall efforts and policies to grow/ diversify energy supply over the last four years and the extent that increased energy security may be attributable to MLET in accordance with GOAM strategies and policies. The question will also be put to other stakeholders as noted below. KEY INFORMANTS USAID. GOAM, PSRC, possible external stakeholders that might have an interest in the project by actively following it (e.g., NGOs, media), other energy sector donors who may need to coordinate their activities with those of USAID. EFFECTIVENESS • ow effective has MLET been in achieving intended outcomes of developing Armenia’s energy market, promoting energy supply diversification and facilitating cross-border trade? • Which interventions have been most and least effective in achieving the intended outcomes and why? • What were the key factors that enabled and/or hindered the achievement of outcomes? • Which of these factors were within and outside of the MLET’s control and manageable interests? • To what extent was MLET able to steer its strategic approach in response to the unfolding of key geo-economic and sector developments during the implementation period? CONCEPTUAL APPROACH Responses to the above questions will be on a task-by-task basis, as the project comprises many (sub) tasks comprising multiple task leaders within Tetra Tech, as well as several distinct Armenian stakeholders depending on the task. In addition, given the wide range of interventions, some of them were very likely more effective than others. ANNEX III: EVALUATION METHODS AND LIMITATION S 41 The largest task area is related to market development, which comprises several sub-task areas including development of enabling legislation/ regulations, capacity building for the newly created entities as the result of unbundling, and assistance in the implementation of “Transitional Market Rules” from the existing “single buyer” system to the implementation of a competitive electricity market. Key focus areas identified in Tetra Tech’s SOW are provided below, together with interviewees and source documents from which the most relevant information for the evaluation will be drawn: • Third-party access: GOAM (enabling legislation), PSRC (regulations, numbers of customers using) • Ancillary services: GOAM (enabling legislation), PSRC (regulations) • Revised power purchase agreements: Tetra Tech, generating entities, distribution entities. • Retail tariffs: Tetra Tech (reports on recommended tariff policies and methodologies), PSRC (actual tariffs, tariff plans) • Competitive markets: GOAM, Tetra Tech markets specialist, PSRC, SO, MO, utilities, donors, external stakeholders – all with respect to the current status of market development and future plans and timing, as well as views on the matter. • Charts of accounts: Tetra Tech regulatory accounts expert (development), utilities (adoption level) • Grid code: Tetra Tech grid code expert, SO (implementation) • Market rules: Tetra Tech markets expert, MO (implementation) • Social issues (e.g., protection of vulnerable segments of the population): Tatra Tech documentation, GOAM (policies and laws); PSRC (implementation) The second main task, energy supply diversification, includes: i) assistance to develop the Least Cost Energy Development Plan (LCEDP) and ii) development of regulations (also including tariffs) to allow for generation other than that from existing nuclear and thermal sources. To assess effectiveness, the users of the MARKAL/ TIMES15 software used to develop the LCEDP will be interviewed and GOAM will be queried on how it uses (has used) the outputs of the LCEDP for developing energy sector policy. Tetra Tech and PSRC will be queried regarding the development generation tariffs. The third main task, facilitate cross-border trade, involves trading in electricity with Georgia. It covers strengthening Armenia-Georgia dialogue on electricity trading through support to a Joint Working Group, promoting imports/ exports on wholesale markets, supporting regional studies on electricity generation and transmission, assisting in the development and implementation of non-discriminatory cross-border access, and supporting development of cross border trade mechanisms to allow a competitive market to exist that transcends the border. To assess effectiveness of this task, Tetra Tech, GOAM, the SO and MO will be queried. 15 MARKAL is a numerical model used to carry out economic analysis of different energy related systems at the country level to represent its evolution over a period of usually 40 to 50 years. Various parameters such as energy costs, plant costs, plant performances and building data can be input and the software will choose an optimal technology mix to meet that demand at minimum cost. TIMES is a successor model to MARKAL that has many similarities. ANNEX III: EVALUATION METHODS AND LIMITATIONS 42 KEY INFORMANTS Tetra Tech, GOAM, PSRC, SO, MO, generation and distribution entities, media, Joint Working Group on cross-border trade. EFFICIENCY • To what extent did the intervention deliver results in an economic and timely way? CONCEPTUAL APPROACH To assess efficiency as above, the general approach will be to examine Tetra Tech’s original SOW objectives together with corresponding indicators-targets provided in the Monitoring, Evaluation and Learning (MEL) Plan of October 18, 2018, and gauge the progress achieved with respect to those objectives and indicators on a monthly and/or quarterly basis, as provided in the Tetra Tech Quarterly and Annual Reports. The KIIs as outlined in the previous section on effectiveness will be used to ask all interviewees questions regarding originally envisaged targets and respective progress for each (sub)task in meeting those objectives and targets, as well as reasons that targets may have been exceeded or not met. To assess cost efficiency, project reports comparing budgeted costs to actuals will be requested and analyzed for items such as budget overruns and burn rates, by (sub)task. KEY INFORMANTS Tetra Tech, GOAM, PSRC, SO, MO, generation and distribution entities, media, Joint Working Group on cross-border trade. IMPACT • To what extent did the MLET interventions move forward Armenia’s desired energy sector reforms? • To what extent did the intervention generate or will generate in the long run significant positive or negative, intended or unintended, higher-level effects? CONCEPTUAL APPROACH To assess impact with respect to Armenia’s desired energy reforms, all key informants interviewed will be asked this and related questions as to what fundamental changes have occurred in their (sub)task areas as a result of the interventions began and what would have happened had the interventions not taken place. KEY INFORMANTS Tetra Tech, GOAM, PSRC, SO, MO, generation and distribution entities, media, Joint Working Group on cross-border trade. SUSTAINABILITY • Which elements of the intervention approaches and results achieved by MLET are most likely to be taken further and sustained beyond the close of the program? • What are the political, geopolitical and geo-economic risks most likely to impact the sustainability of the results of the MLET program? ANNEX III: EVALUATION METHODS AND LIMITATIONS 43 • Who are the supporters and spoilers of continued market liberalization, increase in the uptake of renewable energy technologies, and increased electricity trade, including regional? CONCEPTUAL APPROACH To assess sustainability, all key informants will be asked the first two questions directly along with follow-up questions as to why and by who. The last question may be answered indirectly by each key informant through questions on their opinions on the liberalization program in general (positive or negative), on how they feel about the interventions in their respective task areas and what actions they would personally recommend in terms of improving the operations of the sector. KEY INFORMANTS Tetra Tech, GOAM, PSRC, SO, MO, generation and distribution entities, media, Joint Working Group on cross-border trade. Evaluation Limitations The MLET evaluation, like all evaluations, is expected to face several limitations. These include: • Timing of the evaluation, which is taking place before the end of the project. This means that: a) not all activities will have been completed, or perhaps even begun (in the case of “going live” with a competitive market); and b) insufficient time will have elapsed to be able to assess the sustainability of the intervention, a common challenge for any “final evaluation” that is conducted before project end. This limitation will be addressed, to the extent possible, by assessing general factors known to contribute to sustainability as they would relate to the electricity supply industry, such as institutionalization of changes, demand for capacity, and change in capacity. • Time and resources may be limited for collecting primary data, particularly if access to key stakeholders cannot be established. This may limit the ability to draw firm conclusions in certain areas, such as representativeness of stakeholder views. This limitation may be addressed by including some quantitative measures in the KIIs such as the 1-10 rating scale described above and by examining available secondary data. • Selection bias, as some key informants may decline to be interviewed. Those respondents who choose to be interviewed might differ from those who do not in terms of their attitudes and perceptions, affiliation with government/non-government structures, and socio-demographic characteristics and experience. • Recall bias, since several questions raised during the interviews will deal with issues that took place in the past. The MLET project together with predecessor projects financed by USAID has a history going back to the mid-2000s, and therefore, some respondents may have difficulty in recalling relevant information and attribution to MLET. • Absence of a counterfactual scenario. Due to the absence of a counterfactual, the evaluation design will not be able to collect evidence that attributes observed results to interventions by MLET. The evaluation design may compensate for several of the above limitations through the following methods: • Triangulate evidence from different qualitative and quantitative data sources, which will increase the credibility of findings via validation by multiple data sources. ANNEX III: EVALUATION METHODS AND LIMITATIONS 44 • A certain degree of selection bias is unavoidable because the evaluation will rely partly on Tetra Tech and USAID for obtaining contacts. To mitigate this bias, the evaluation team will select the key informants to be interviewed. During the KIIs, stakeholders will also be asked to recommend other key informants (snowball method) to verify that data is being obtained from the most relevant persons. • Regarding qualitative findings, the evaluation team will be careful to report only those findings that recur with relative frequency across multiple stakeholders. ANNEX IV: DATA COLLECTION AND ANALYSIS TOOLS 45 ANNEX IV: DATA COLLECTION AND ANALYSIS TOOLS Questionnaires My name is {Evaluation Team Member} and I am working with International Development Group LLC. We are gathering information about the USAID/Armenia Market Liberalization and Electricity Trade (MLET) Program to assess project achievements, implementation challenges, and lessons learned. Our study has been commissioned by USAID and it is part of its commitment to rigorous and high-quality program evaluation – the systematic collection and analysis of information to understand and improve the impact of its development programs. If you agree to participate in this interview, we will talk about your participation in the MLET program. The interview is expected to last up to an hour. Any information you provide that can identify you will be kept strictly confidential by the parties conducting this study, including USAID employees and researchers, to the maximum extent permitted by the laws of the United States of America and the laws of Armenia. These users will use data for qualitative research. Your participation is voluntary, and you may choose not to answer any or all questions for any reason. In other words, you have the alternative to not participate and there will be no consequences for nonparticipation. You may contact Pooja Singh, Project Director at International Development Group LLC at psingh@internationaldevelopmentgroup.com if you have questions, concerns or complaints about the study or your rights as a participant. If you have any questions for me, please feel free to ask at any time. FOR USAID No. Question 1 Overall, talking about the MLET project, do you think it contributed to USAID’s strategic intermediate result “energy security increased”? If yes, to what extent and in what specific ways? 2 To what extent is the MLET project consistent with other USG strategies and objectives in Armenia? 3 How has the activity contributed to wider efforts of GOAM to increase energy security in Armenia? How would you rate (1-10) the effectiveness of that process, Please, elaborate on your assessment. 4 What do you see as the most effective parts of the assistance in meeting project goals? (market, planning, external trade). How effective has general guidance been on compliance with EU directives? 5 How would you rate (1-10) the effectiveness of capacity building in the project to bring about the desired outcomes, globally and by task area? 6 How would you rate (1-10) the effectiveness of direct technical assistance and guidance on technical issues to agencies to achieve objectives, globally and by task area? 7 How would you rate (1-10) the effectiveness of policy initiatives, globally and by task area? 8 How effective has MLET been in facilitating cross-border trade? Please use 1-10 scale and elaborate your answer. 9 In what other specific ways do you think the technical assistance was useful? 10 Do you believe the project is “on schedule” in attaining its objectives? What has helped or hindered the timeline? 11 On reflection, in which subject/ task areas might more/ less effort and resources have been placed? ANNEX IV: DATA COLLECTION AND ANALYSIS TOOLS 46 No. Question 12 What would have happened without the technical assistance? Do you think it might have happened anyway? 13 What other positive or negative effects came about as a result of MLET? 14 Do you think the project has paid enough attention to social issues? 15 Do you think the project outcomes are sustainable (market, planning, external trade)? What conditions are required to ensure sustainability? What are the threats to sustainability? 16 How do you rate (1-10) the coordination with other donors working in the sector such as World Bank, EU, EBRD? Are there specific areas where MLET supports other donors, or vice versa? Are there areas of disagreement or contention? 17 What is your perception of the political issues (if any) that have blocked progress on integration and moving to market? Is this the main constraint to achieving all objectives? 18 What are the political, geopolitical and geo-economic risks most likely to impact the sustainability of the results of the MLET program? 19 Who are the key organizations or people that USAID feels that the evaluation team should meet while conducting the evaluation? 20 How does USAID anticipate using this assessment? What are the special areas of concern or sensitivities that should be considered? 21 What should the assessment be sure to cover that might otherwise be overlooked? 22 Are there plans to continue assistance in some form at the end of the contract period? What elements of the project would be retained or if no further project, folded into other USAID projects with similar objectives? 23 What project activities had an impact in increasing understanding of the issues by the public and by influencing the media in their coverage? Which media /persons should we contact? FOR TETRA TECH AND OTHER IMPLEMENTERS No. Question 1 Can you please summarize your mandate for the assistance? 2 How did the project contribute to the efforts of GOAM to increase energy security in Armenia? How significant and/or relevant were these contributions? 3 Given that the purpose of the project was “improve the electricity market in Armenia and support electricity trade with Georgia”, what constitutes success (by Task area)? 4 How effective has MLET been in facilitating cross-border trade? Please use 1-10 scale and elaborate your answer. 5 How would you define (in terms of the definition) the degree to which Armenia has gone in meeting project objectives? What are reasonable measurements? 6 What interventions by MLET have been most helpful and contributed to Armenia integration into meeting the goal and reaching the desired end point and why 7 In your opinion, how successful has the USAID energy sector assistance been as defined by the objectives and terms of the project? 8 In what specific way do you think the technical assistance was useful? ANNEX IV: DATA COLLECTION AND AN ALYSIS TOOLS 47 No. Question 9 What in your view have been specific obstacles to project implementation? Which of these obstacles were within and outside of the MLET’s control and manageable interests? ow were they overcome? What are the obstacles in Armenia? 10 What was not addressed, that would have been helpful in addressing the issues of concern in liberalizing the markets, sector planning and external trade? 11 Do you believe the project is “on schedule” in attaining its objectives? What has helped or hindered the timeline? How have you dealt with these factors? 12 On reflection, in which subject/ task areas might more/ less effort and resources have been placed? 13 What areas of work will be most affected once USAID assistance ends? Can you specify how progress on the three main Task areas might be affected? What agencies and issues will be most affected at this stage and in what way? 14 What would have happened without the technical assistance? Do you think it might have happened anyway? Under what circumstances? 15 What other positive or negative effects came about as a result of MLET? 16 How sustainable are the reforms brought about by the assistance? Will the changes continue once the assistance has ended? By whom? What are the constraints for the future? 17 Do you see the particular USAID assistance as appropriate and relevant, in light of the current political, regulatory and legal environment in Armenia? How well do you think it fits with the USAID development strategy? 18 In your opinion, did the project pay enough attention to social issues? 19 If new, follow-on assistance is designed, how should it differ from the present assistance? What should it concentrate on? 20 Can you recommend any individuals, groups, or organizations in Armenia the evaluation team should be sure to contact? For example in media, NGO’s, donors? FOR BENEFICIARIES No. Question 1 What is the nature of the assistance that is being provided to your agency? 2 In what specific way was technical assistance useful? How would you rate (1-10) the effectiveness of direct technical assistance and guidance? 3 Given that the purpose of the project was “improve the electricity market in Armenia and support electricity trade with Georgia”, what constitutes success? 4 To what extent has Armenia met the objectives of market liberalization, sector planning and external trade? What are reasonable measurements to evaluate the extent it has met requirements? To what extent has the MLET helped this happen? 5 What assistance from MLET has been most helpful in contributing to Armenian market reform and meeting the goal? 6 What was not addressed? What other assistance would have been helpful to your agency? 7 What has helped your agency in meeting the objectives of the MLET? What has helped or hindered it and how have you dealt with it? 8 Do you believe the project is “on schedule” in attaining its objectives? 9 On reflection, in which subject/ task areas might more/ less effort and resources have been placed? ANNEX IV: DATA COLLECTION AND ANALYSIS TOOLS 48 No. Question 10 Can you evaluate the impact of the assistance on your agency’s work? What specifically has been accomplished or resulted or been impacted in your agency by MLET assistance? 11 What would have happened without the technical assistance (expertise, facilitation)? Would it have happened anyway? 12 What other positive or negative effects came about as a result of MLET? 13 Can you evaluate whether MLET assistance was coordinated with other donor assistance to your agency (or loans)? Are there areas where coordination should have been better? 14 Can you comment on assistance on the economic and governance issues, and the commercial activities of the electricity supply entities? Are there issues that needed more attention and if so, what could have been done? 15 Has MLET activity over the past 4 years had an effect on public support? How? 16 How satisfied are you with the experience with the USAID assistance? How would you rate your level of satisfaction on a scale of 1 – 10 (10 highest)? 17 ow effective do you view the USAID contractor’s work of implementing the USAID assistance? Can you provide comments about the experience, qualifications, and effectiveness of the contractor team? Can you rate effectiveness on a scale of 1 – 10 (10 highest)? 18 What areas of work will be most affected once USAID assistance ends? Can you specify how progress on improving the electricity market will be particularly affected? Where will the cut off in assistance be particularly damaging? 19 What in your view have been obstacles to achieving a fully liberalized electricity market, effective planning function and external trade? 20 What is your opinion regarding further market liberalization, improved planning and external trade? Good for Armenia or not? Why? 21 In your opinion, did the project pay enough attention to social issues? 22 In your opinion, how could the assistance be improved? 23 Do you have a specific recommendation that the evaluation team could provide to USAID to help it improve future assistance? In light of your experience, what advice would you give USAID? FOR DONORS No. Question 1 Given that the purpose of the project was “improve the electricity market [in Armenia] and support electricity trade with Georgia,” do you think MLET was successful? If yes, what constitutes success? 2 How far would you say Armenia has gone over the past 4 years or so regarding electricity market liberalization, energy sector planning and external trade? What have been the accomplishments? 3 In what specific ways has MLET assistance supported and coordinated with your goals and objectives as a donor? 4 In what other specific way do you think the technical assistance was useful? 5 What do you think would have happened without the technical assistance? Would it have happened anyway? 6 Have there been specific areas of disagreement with MLET activities or USAID where you think coordination could have been improved? ANNEX IV: DATA COLLECTION AND ANALYSIS TOOLS 49 No. Question 7 What in your view have been specific obstacles to electricity market liberalization and external electricity trade? What are the obstacles in Armenia? 8 In your view, has public support changed over the last four years or so? Is it less, the same or improved? 9 What has not been addressed, that would have been helpful in meeting the terms in liberalizing the electricity market and external electricity trading? Was sufficient attention paid to social issues? 10 Has enough attention been paid by donors to social issues in Armenia? 11 What areas of work will be most affected once USAID assistance ends? Do you/ can you specify how progress on market liberalization might be affected? What agencies and issues will be most affected at this stage and in what way? 12 How sustainable are the reforms brought about by the assistance? Will the changes continue once the assistance has ended? What are the constraints for the future? FOR MEDIA AND OTHER POSSIBLE EXTERNAL STAKEHOLDERS No. Question 1 What is the nature of your involvement in the energy sector of Armenia? How are you associated with MLET? 2 Did the MLET assistance bring about a change in public understanding of the issues surrounding market liberalization and external electricity trade? 3 How would you evaluate public support for electricity market reform now compared to 4 years ago: worse, the same or better? 4 Has the public supported or resisted the changes that have occurred? Has MLET activity had an effect over the 5 years? 5 Can you evaluate the impact of the assistance on your institution and its work? What specific accomplishments, changes or results in your work have come from MLET assistance? 6 In what other specific way do you think the technical assistance was useful (or not)? 7 What would have happened without the technical assistance? Would it have happened anyway? 8 The goal of the project was to “improve the electricity market [in Armenia] and support electricity trade with Georgia”. What interventions by MLET have been most helpful and contributed to this goal? 9 To what extent do you think the electricity market of Armenia has liberalized over the past 4 years? What are reasonable measurements to evaluate "liberalization"? 10 Did MLET sufficiently address the economic and governance issues? The needs of fragile economic groups? Environmental protection? Commercial activities of the electricity supply entities?? 11 Are there other issues that needed more attention and assistance? 12 How satisfied are you with the experience with USAID assistance? How would you rate your level of satisfaction on a scale of 1 – 10 (10 highest)? 13 ow effective do you view the USAID contractor’s work of implementing the USAID assistance? Can you provide comments about the experience, qualifications, and effectiveness of the contractor team? 14 What in your view have been obstacles to achieving a fully integrated energy sector in the regional and EU market? What are the most contentious and difficult issues? 15 Do you think the results of MLET are sustainable? Why or why not? 16 In your opinion, how could the assistance be improved? ANNEX V: SOURCES OF INFORMATION 50 ANNEX V: SOURCES OF INFORMATION The names of the key informants were deleted from the external version of the report. ANNEX VI: DISCLOSURE OF ANY COMPANY INTEREST 51 ANNEX VI: DISCLOSURE OF ANY COMPANY INTEREST ANNEX VI: DISCLOSURE OF ANY COMPANY INTEREST 52 ANNEX VI: DISCLOSURE OF ANY COMPANY I NTEREST 53 ANNEX VI: DISCLOSURE OF ANY COMPANY INTEREST 54 ANNEX VI: DISCLOSURE OF ANY COMPANY INTEREST 55 ANNEX VI: DISCLOSURE OF ANY COMPANY INTEREST 56 ANNEX VI: DISCLOSURE OF ANY COMPANY INTEREST 57 ANNEX VI: DISCLOSURE OF ANY COMPANY INTEREST 58 ANNEX VI: DISCLOSURE OF ANY C OMPANY INTEREST 59 ANNEX VI: DISCLOSURE OF ANY COMPANY INTEREST 60 ANNEX VI: DISCLOSURE OF ANY COMPANY INTEREST 61 ANNEX VI: DISCLOSURE OF ANY COMPANY INTEREST 62 ANNEX VII: EVALUATION TEAM MEMBERS 63 ANNEX VII: EVALUATION TEAM MEMBERS Evaluation Team Leader (Arvid Kruze). Arvid E. Kruze is an energy market specialist, with over 40 years of expertise in energy project evaluations, monitoring and evaluation, and economic and financial assessments. Mr. Kruze brings both strong technical expertise related to the electricity market reform and USAID evaluation framework. He has served either as a Team Leader or an Energy Expert in more than 35 evaluations for USAID, MCC, ADB, and the World Bank. His international experience includes projects in over 30 countries including in Armenia, Azerbaijan, and Georgia. As an energy expert for the USAID/Georgia Governing for Growth (G4G) Project, Mr. Kruze assessed the energy activities of the project, including the development of a liberalized energy market and trading on day ahead markets with Turkey. As an Energy Economist for a World Bank project, Mr. Kruze worked on the development of the renewable energy sector in Turkey in line with EU energy priorities, strategies, and directives. He conducted an economic analysis on various renewable energy alternatives including the trading of national wholesale day ahead markets. Among numerous relevant prior positions, Mr. Kruze served as the Lead Evaluation Specialist for the final performance evaluation of the USAID Energy Security and Regional Integration Project; and Team Leader of Performance Evaluation of the USAID Energy Links Project in Central Asia. Mr. Kruze is a proven team leader of project evaluations managing a team of two to three experts in conducting evaluations, leading data collection efforts and analysis, integrating inputs from team members, and presenting the results to the Client. Energy and Regional Integration Expert (Gurgen Hakobyan). Dr. Gurgen Hakobyan has over 30 years of experience working as an energy expert. He has supported USAID energy regulatory projects in Armenia, Georgia, Kosovo, and Tanzania. Mr. Hakobyan has a deep understanding of the electric power system operation and management, system synchronization and unbundling generation, transmission, distribution, and regulation activities. He brings an unique understanding of the South Caucasus energy market. As a consultant for the IFC in Armenia between 2018-2019, Dr. Hakobyan provided advisory services to the Electric Networks of Armenia, which included the development of an integrated distribution system plan and assisting in the implementation of the advanced distribution management system. On the USAID/Caucasus-Georgia Hydro Power and Energy Panning Project (HPEP), Mr. Hakobyan supported the conceptual design of an electricity trading mechanism and the development of the Transmission System Operator. This new mechanism liberalized the market and introduced day￾ahead and intraday electricity trades while ensuring system security and reliability. Additionally, Mr. Hakoyan provided capacity building to the Energy Regulator (GNERC), designated Market Operator (MO), designated Transmission System Operator (TSO), Government of Georgia officials and key stakeholders in areas covering power market development and implementation of reforms to support power market sales to Turkey and ENTSO-E (European Network Transmission System Operators for Electricity) markets. Under the USAID Program to Strengthen Reform and Enhance Energy Security in Armenia, he provided strategic assistance to the Government of Armenia’s reform program, focusing on generation planning including nuclear plant decommissioning and replacement capacity, least-cost generation plan, nuclear power plant initial environmental impact assessment and feasibility studies, renewable energy promotion, and regulatory and legal framework development. Energy Market Liberalization Epert (Ms. Milica Vukovljak) Ms. Vukovljak has more than twenty years of experience in the electricity sector - ten years working with the transmission systems market operator in Serbia and another ten as an electricity expert, monitoring market development, and drafting policies and market rules with the Energy Agency of the Republic of Serbia. As a transmission system operator, ANNEX VII: EVALUATION TEAM MEMBERS 64 Ms. Vukovljak was a part of the team that worked on the transition of Serbia to a liberalized energy market. Her work included registering participants in the market, ensuring guarantees of origin, preparing rules for cross-border transmission capacity, processing data related to deviation and billing, and developing the systems to manage the electricity market. She also increased capacity for market participants by organizing training focused on the allocation of cross-border transmission capacity and market code, rules, and procedures. In her current position, Ms. Vukovljak collects, processes, and analyzes electricity market data. She also drafts relevant policy and regulatory guidelines, recommendations, and reports for the Energy Agency to ensure compliance with the current energy market. Monitoring and Evaluation Expert (Naira Vardanyan). Naira Vardanyan is a monitoring and evaluation specialist with expertise in evaluation design, instrument, and protocol design, developing and implementing M&E systems, and supporting data collection efforts. With over 16 years of experience, she has developed evaluation tools to assess program activities and events for different donors including USAID. As the Monitoring, Evaluation and Research Specialist for the USAID My Armenia Project, Ms. Vardanyan updated and implemented the project’s M&E system, used different survey tools to collect data, and analyzed data to input into the system. Currently, Ms. Vardanyan has been working closely with HEKS-EPER Swiss Church Aid to serve as an internal and external specialist to perform evaluations or comprehensive scientific impact assessments of key projects, programs, and topics. She has technical experience in SPSS, Nvivo, Stata, survey data collection, and PMP indicators. Research Assistant/HQ Activity POC (Hillary Chidsey). Hillary Chidsey is an international development professional with 6 years of experience working within project management, global exchange, grassroots development, and communications. Currently, Ms. Chidsey serves as a Project Coordinator for International Development Group Advisory Services, LLC. Within this role, she coordinates financial and technical operations for USAID and MCC funded projects. Alongside her responsibilities working on active IDG projects, she assists with new business efforts and proposal writing. Ms. Chidsey received her MA from the University of Kentucky’s Patterson School of Diplomacy in 2021. TAP EG Manager (Oana Mermeze). Ms. Oana Mermeze is an international development professional specializing in quantitative and qualitative analysis and monitoring and evaluation. Currently, Ms. Mermeze is a Manager at International Development Group LLC, where she provides technical and operational support to USAID and MCC-funded economic growth projects in Sub-Saharan Africa, Southeast Asia, and the Caribbean. ANNEX VII: EVALUATION TEAM MEMBERS 65 Team Roles and Responsibilities Role Responsibilities Evaluation Team Leader (Arvid Kruze) • Provide overall leadership, management, and technical direction of the evaluation. • Serve as the primary point of contact with USAID/Armenia Activity Manager and other counterparts on all technical aspects of the evaluation. • Lead development of evaluation design and development of evaluation protocols and instruments that best address the evaluation questions. • Responsible for preparation and submission of all deliverables working closely with other team members and the TAP EG Manager. • Manage the work of the other members of the evaluation team. Energy and Regional Integration Expert (Gurgen Hakobyan) • Provide regional perspectives and technical insights related to electricity that are pertinent to the design and implementation of the MLET evaluation. • Assist with identifying and connecting with key stakeholders for KIIs, if needed. • Work closely with the Evaluation Team Leader and M&E expert to develop interview questionnaires, survey protocols, and other evaluation instruments. • Produce deliverables, working in close collaboration with the Evaluation Team Leader. • Attend all meetings, presentations, and briefings with USAID counterparts. Monitoring and Evaluation Expert (Naira Vardanyan) • Contribute to the development of the Evaluation Design Plan, working with the rest of the Evaluation team. • Lead the development of survey protocol and instruments, working with Team Leader. • Responsible for quantitative and qualitative data management, cleansing, and analysis. • Responsible for drafting key sections of the deliverables, in collaboration with the Team Leader. Energy Market Liberalization Expert (Milica Vukovljak) • Provide regional perspectives and technical insights related to market liberalization that are pertinent to the design and implementation of the MLET evaluation. • Assist with identifying and interviewing key stakeholders for KIIs, as needed. • Work closely with the Evaluation Team Leader and M&E expert to develop interview questionnaires, survey protocols, and other evaluation instruments. • Produce deliverables, working in close collaboration with the Evaluation Team Leader. • Attend all meetings, presentations, and briefings with USAID counterparts. ANNEX VII: EVALUATION TEAM MEMBERS 66 Role Responsibilities Research Assistant/HQ Activity POC (Hillary Chidsey) • Provide research support as requested by the Evaluation Team Leader. • Work with the Team Leader to develop survey instruments and protocols (as needed). • Assist with data analysis and drafting of certain sections of key deliverables. • Schedule meetings, interviews; assist with travel arrangement; and provide other logistical and administrative support. TAP EG Manager (Oana Mermeze) • Responsible for monitoring the overall management and performance of the evaluation and assuring that the evaluation team receives the resources and support it requires to function effectively. • Conduct internal weekly check-in calls with the Evaluation Team Leader and participate in bi-weekly calls with USAID/Armenia. • Review all deliverables prior to submission to USAID, gathering feedback from key TAP EG resources, as needed. ANNEX VIII: STATEMENTS OF DIFFERENCE 67 ANNEX VIII: STATEMENTS OF DIFFERENCE The Implementing Partner, Tetra Tech, Inc., reviewed the evaluation report and accepted the findings and conclusions. ANNEX IX: DOCUMENTS REVIEWED 68 ANNEX IX: DOCUMENTS REVIEWED Tetra Tech. Year One Work Plan, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, September 2018. Tetra Tech. Year 1 Program Monitoring, Evaluation, and Learning (MEL) Plan, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, October 2018. Tetra Tech. 2nd Quarterly Report: July – September 2018, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, October 2018. Tetra Tech. 3rd Quarterly Report: October – December 2018, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, January 2019. Tetra Tech. 4th Quarterly Report: January – March 2019, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, April 2019. Tetra Tech. 5th Quarterly Report: April – June 2019, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, July 2019. Tetra Tech. Year 2 Work Plan, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, September 2019. Tetra Tech. First Annual Report: October 2018 – September 2019, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, November 2019. Tetra Tech. Armenia Wholesale Electricity Market Rules, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, December 2019. Tetra Tech. Armenia Retail Electricity Market Trading Rules, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, December 2019. Tetra Tech. Reliability and Security Indicators of the Electric Power System of the Republic of Armenia, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, December 2019. Tetra Tech. Power System Transmission Network (Grid) Code of the Republic of Armenia, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, December 2019. Tetra Tech. 6th Quarterly Report: October – December 2019, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, January 2020. Tetra Tech. 7th Quarterly Report: January – March 2020, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, April 2020. Tetra Tech. Review of the Electricity Market Rules, and Market Monitoring Solutions, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, June 2020. Tetra Tech. 8th Quarterly Report: April – June 2020, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, July 2020. ANNEX IX: DOCUMENTS REVIEWED 69 Tetra Tech. Year 3 Work Plan, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, September 2020. Tetra Tech. Second Annual Report: October 2019 – September 2020, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, October 2020. Tetra Tech. Armenia Wholesale Electricity Market Monitoring Guidelines Presentation, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, December 2020. Tetra Tech. 9th Quarterly Report: October – December 2020, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, January 2021. Tetra Tech. Action Plan to Ensure Implementation of the Republic of Armenia Energy Sector Development Strategic Program, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, January 2021. Tetra Tech. Recommendations on Improvement of the Primary Legislation for Autonomous Power Production in Armenia, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, February 2021. Tetra Tech. 10th Quarterly Report: January – March 2021, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, April 2021. Tetra Tech. 11th Quarterly Report: April – June 2021, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, July 2021. Tetra Tech. Year 4 Work Plan, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, September 2021. Tetra Tech. Armenia Electricity Market Gap Analysis Presentation, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, September 2021. Tetra Tech. Third Annual Report: October 2020 – September 2021, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, October 2021. Tetra Tech. 12th Quarterly Report: October – December 2021, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, January 2022. Tetra Tech. 13th Quarterly Report: January – March 2022, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, April 2022. Tetra Tech. 14th Quarterly Report: April – June 2022, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, July 2022. Tetra Tech. Fourth Annual Report: October 2021 – September 2022, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, October 2022. Tetra Tech. Armenia Least Cost Energy Development Plan: 2024-2050, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, December 2022. ANNEX IX: DOCUMENTS REVIEWED 70 Tetra Tech & Government of Armenia, Unofficial Translation of the Republic of Armenia Energy Sector Development Strategic Program to 2040, Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, January 2023. Tetra Tech. Ten-Year Development Plan (2023-2032), Market Liberalization and Electricity Trade (MLET) Program. Yerevan, Armenia: USAID, March 2023 United States Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 usaid.gov