January 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Arvid Kruze and Artashes Sargsyan through Social Impact, Inc. Performance Evaluation of the Energy Security and Regional Integration (ESRI) Project End of Project Evaluation Report Cover Photo: Metsamor Nuclear Power Plant, Metsamor, Armenia. Photographed by Artashes Sargsyan. PERFORMANCE EVALUATION OF THE ENERGY SECURITY AND REGIONAL INTEGRATION PROJECT (ESRI) End of Project Evaluation Report USAID/Armenia January 23, 2013 Task Order # AID-111-TO-12-00002 Evaluation of Five USAID/Armenia Activities DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ACKNOWLEDGMENTS The evaluators with to extend sincere thanks to everyone who provided valuable time and information in support of this evaluation, including all those at the USAID/Armenia Mission, Tetra Tech, Ministry of Energy and Natural Resources, Public Service Regulatory Commission, System Operator, Settlement Center, High Voltage Electricity Networks, Scientific Energy Research Institute, State Engineering University of Armenia, Yerevan State University, State Committee for Nuclear Safety Regulation, and KfW. CONTENTS Acronyms.................................................................................................................................................................................................. 6 Executive Summary................................................................................................................................................................................. 7 Evaluation Purpose & Evaluation Questions..................................................................................................................................13 Project Background ..............................................................................................................................................................................14 Table 1 – Sources of Electricity in Armenia (2010)................................................................................................................15 Figure 1 – Power System of Armenia Map................................................................................................................................17 Evaluation Methods & Limitations ....................................................................................................................................................20 Findings & Conclusions........................................................................................................................................................................22 Evaluation Question 1.....................................................................................................................................................................22 Evaluation Question 2.....................................................................................................................................................................25 Evaluation Question 3.....................................................................................................................................................................27 Evaluation Question 4.....................................................................................................................................................................30 Evaluation Question 5.....................................................................................................................................................................32 Recommendations ................................................................................................................................................................................36 Annexes...................................................................................................................................................................................................39 Annex I: Evaluation Statement of Work....................................................................................................................................40 Annex II: Evaluation Methods and Limitations .........................................................................................................................49 Annex III: Data Collection Instruments.....................................................................................................................................52 Annex IV: Sources of Information ...............................................................................................................................................55 Annex V: Disclosure of any Conflicts of Interest....................................................................................................................59 ACRONYMS ANRA Armenian Nuclear Regulatory Agency CSJS Common Joint Stock Company EIA Environmental Impact Assessment ENA Electric Networks of Armenia ESRI Energy Security and Regional Integration EU European Union GOA Government of Armenia GWh Gigawatt-hour HVEN High voltage Electric Networks IAEA International Atomic Energy Association kV Kilovolt MW Megawatt MOENR Ministry of Energy and Natural Resources MOU Memorandum of Understanding NARUC National Association of Regulatory Utility Commissioners NPP Nuclear Power Plant NRC Nuclear Regulatory Commission PSRC Public Services Regulatory Commission SARG Safety Analysis and Report Guide SC Settlement Center SEUA State Engineering University of Armenia SO System Operator SOW Statement of Work SRIE Scientific Research Institute of Energy US United States USAID U.S. Agency for International Development USEA United States Energy Administration YSU Yerevan State University 6 EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The main purpose of this end-of-project performance evaluation is to provide USAID/Armenia with an objective analysis of USAID assistance to Armenia’s energy sector under the $5.9 million Energy Security and Regional Integration (ESRI) project, undertaken over the period 2009-2012. The ESRI project is one of the larger projects in USAID/Armenia’s Economic Growth Portfolio and, as such, is subject to performance evaluation per USAID Evaluation Policy. The findings herein will be used to make informed decisions on the future directions of USAID/Armenia’s assistance in the Energy Sector. Key evaluation questions focus on whether there is evidence the project contributed to the enhancement of practices and understanding in the Armenian energy sector of the three main task areas of USAID intervention, including nuclear power, renewable energy and regional electricity trade, as well stakeholders’ perceptions of the project’s results and relevance of the various tasks undertaken. The evaluation addresses a set of Key Questions (KQs) that are directly correlated to the evaluation’s purpose. The KQs, as stated in the Evaluation Statement of Work (SOW) are: ˙ Is there evidence that the independent expert services to Government of Armenia for new Nuclear Power Plant contributed to procedures and activities implemented in accordance with the western safety and environmental standards in preparation of documentation for the New Nuclear Power Plant? ˙ Is there evidence that the scenarios of power system integration contributed to increased understanding of renewable energy development tendencies in the region among the main stakeholders? ˙ Is there evidence that the regional integration task contributed to an increase of regional cooperation with Georgia? ˙ What do stakeholders groups see as the results of the project? ˙ How relevant was the intervention in terms of Armenia’s priorities in energy sector? PROJECT BACKGROUND The ESRI project is part of a long history of practically continuous USAID assistance to the Armenian energy sector since the mid-1990s. As part of the USAID mission’s strategy to help Armenia develop an efficient, reliable and cost-effective energy sector to support its emerging market economy, USAID has been supporting Armenia’s efforts to secure diversified energy sources, including nuclear, renewables and international electricity trade. In late 2009, PA Consulting (now Tetra Tech) was recruited by USAID as the contractor to carry out the ESRI project. Tetra Tech’s Statement of Work (SOW) for the project comprised three major tasks: Task 1: New Nuclear Plant Development Under this task, the contractor was to provide services to the GOA through the following activities: 1) Upon the request from the MOENR, provide information and independent expert services with respect to the new nuclear unit development, including its planning, design, tendering and construction; 7 2) In coordination with US Nuclear Regulatory Commission, provide assistance to Armenian Nuclear Regulatory Agency (ANRA) to ensure that important elements needed for regulatory review and plant licensing are developed; and 3) Help implement critical human capacity building actions identified by the International Atomic Energy Agency (IAEA) and encourage donors and other interested parties to finance education and training programs. Expected results were: ˙ MOENR will receive necessary information and advice with respect to the new nuclear plant development; ˙ Assistance will be provided to ANRA to ensure that important elements needed for regulatory review and plant licensing are developed; and ˙ Critical workforce development and technical training requirements will be strengthened, leading to increased government and donor funding for the existing in-Armenia training centers, universities and institutes. Task 2: Renewable Energy Development Strategy and Power System Integration Under this task, the contractor was to undertake the following: 1) Analyze and make recommendations concerning legal, regulatory, and market issues in developing renewable energy investment projects; 2) Provide assistance to PSRC in improving the regulatory framework and tariff policy to support development of wind and solar energy; 3) Conduct static and dynamic stability analysis and consider the implications of renewable energy development for grid investment and operation; 4) Assist the MOENR in conducting research study on development and grid connection of solar energy innovative technology, identified by the MOENR; 5) Provide appropriate software and carry out system operational simulations to help plan for more economic dispatching and effective transmission system management (in coordination with work under task 3; and 6) Facilitate access to information and knowledge transfer about US experience, technologies, and companies active in renewable energy. Expected results were: • Contribute to increasing the number of renewable energy projects and their output of electricity, thereby contributing to diversifying domestic energy supply by at least 15 % and reducing or avoiding greenhouse gas emissions. • Contribute to the creation of an improved investment climate for companies seeking investment opportunities in renewable energy and clarified issues related to grid access and dispatching of these renewable units. • The GOA will promulgate a clearer policy and strategy on renewable energy development and more accurately estimate the potential contribution of these sources and innovative technologies to meeting future energy demands. • As a result of the solar energy research study, the innovative technology will be tested. 8 • Due to project efforts, transmission planning and analysis of grid operational issues – arising from the increasing output of a growing number of renewable projects – will be improved and appropriate transmission investments pursued. Task 3: Regional Electricity System Synchronization and Future Trade and Market Development Under this task, the contractor was to undertake the following1: 1) Regional system synchronous operations, system stability studies and training; 2) Technical assistance and advisory support for harmonization of legal and normative documents governing interregional cooperation; 3) Transmission tariff and investment analysis; and 4) Support of partnership for transmission organizations, exchange visits and training. Expected results were: • Improve transmission planning capabilities of Armenia Transmission Company in order to support Armenia system development and greater integration into the Caucasus system, • Improve Settlement Center’s capacity to calculate power flows and losses in the system in order to perform its critically important role, • Develop plans to integrate Armenia and Georgia power systems into a full interconnection mode. The overall goals of the ESRI project were to ensure the following: (1) That the new Nuclear Unit meets western safety and environmental standards; (2) That Armenia develops the human and institutional capacity to plan and successfully implement the project; and (3) That the contribution of the project to Armenia’s economic growth and development is maximized. The project’s objectives also include improving the enabling environment for the development of renewable energy in Armenia, and supporting Armenia’s integration into regional energy systems and markets. EVALUATION DESIGN, METHODS AND LIMITATIONS Prior to conducting in-country data collection, the evaluation team employed a comprehensive desk review of project work plans, quarterly reports, and numerous technical reports and documents. Key informant interviews with beneficiaries of the USAID assistance and stakeholders within the Armenian energy sector were then conducted in Yerevan. Interviewees included Mission staff, the Chief of Party 1 As per Tetra Tech’s Quarterly Reports. The sub-tasks in the SOW provided to the evaluation team by USAID were: i) support for transmission planning and training with the System Operator, Settlement Center and Scientific Research Institute for Energy, ii) support to a proposed electricity interconnection Working Group between Georgia and Armenia and collaboration on associated technical studies, regulatory issues and cross-border trading issues; and iii) assistance for analysis, benchmarking studies, and training on electricity systems in the region and the potential growth in demand and generation capacity. It is assumed that these sub-tasks were revised prior to commencement of the contractor’s services. 9 for the implementing contractor, a local subcontractor to the US contractor, heads of various organizations that were direct beneficiaries of the assistance and, the only other international lending agency heavily involved in the Armenian energy sector, KfW. Given the nature of the KQs, interviews were deemed the most appropriate qualitative evaluation approach. As with any interview, recall bias and selection bias posed potential threats to internal validity of data. The evaluators determined that recall bias could be effectively mitigated, however, as a result of the project’s recent termination and high recall capability of those interviewed. To correct for potential selection bias, the evaluation team attempted to triangulate qualitative data with data from project reports and other documents, where applicable. Also, in order to put the stakeholders’ views into an appropriate context, achievements were compared with original expectations, as stated in the original project design and Statement of Work. To the extent possible, the analysis offers explanations on how and why particular outcomes occurred. An additional shortcoming is the reliability of the interviewer to respond truthfully, as a respondent may have particular motives for answering in a certain manner. This weakness was addressed through the triangulation of response data with statements made by others, as well as with objective analysis of project results. FINDINGS AND CONCLUSIONS The first evaluation question asks whether there is evidence that the independent expert services to Government of Armenia (GOA) for the new Nuclear Power Plant (NPP) contributed to procedures and activities implemented in accordance with the western safety and environmental standards in preparation of documentation for the new NPP. This question addresses the nuclear component of the three project tasks. Although the process of developing the new NPP has been slower than anticipated, it was found that progress has been significant in: i) support provided to the Ministry of Energy and Natural Resources (MOENR) in holding public hearings and in preparing vital documents, ii) support to Armenian Nuclear Regulatory Agency (ANRA)2 in the development of a safety guide for licensing purposes based on western standards, as well as provision of safety analysis software and training and iii) development of curricula at two State universities in nuclear engineering and nuclear physics that are currently being adopted. Thus, there is strong evidence that the independent expert services to the GOA for the new NPP have contributed to procedures and activities implemented in accordance with western safety and environmental standards in preparation of documentation for the new NPP. The second evaluation question asks whether there is evidence that the scenarios of power system integration contributed to increased understanding of renewable energy development tendencies in the region among the main stakeholders. It should be mentioned that this question addresses a relatively small part of the renewable energy component of the three major project tasks, as it covers only two of the original six sub-tasks. It has been concluded that including renewable energy in the technical and economic simulations of power system integration have contributed somewhat to increased understanding of renewable energy development tendencies in the region among the main stakeholders. This is because the simulations focused on regional integration of power systems. Within this context, the scope for increased 2 Although still known by the acronym ANRA, the organization is now called the “State Committee for Nuclear Safety Regulation under the Government of Armenia” 10 understanding of renewable energy is limited. Also, to most knowledgeable stakeholders (e.g., power system engineers), the results of these simulations with respect to renewable energy were probably a foregone conclusion. Therefore, the simulations probably did not add very much to stakeholders’ renewable energy knowledge base. The third evaluation question asks whether there is evidence that the regional integration task contributed to an increase of regional cooperation with Georgia. This question addresses the interconnection component of the three project tasks. As a result of the project, agreements have been reached on parallel operation of the Armenian and Georgian power systems and Armenian-Georgian power exchanges for emergency and unplanned situations. Also, discussions on provisions for transit power are underway. Thus, there is ample evidence that the regional integration task contributed to an increase of regional cooperation with Georgia. The fourth evaluation question asks what the stakeholder groups see as the results of the project. The project comprised three main tasks with different beneficiaries. Depending how they benefitted from the ESRI project, stakeholders viewed the results of the ESRI project differently. These results included various support to development of the new nuclear power plant, support to the universities in developing the new nuclear curricula, support in carrying out public hearings in connection with the new nuclear plant, development of safety requirements for the new nuclear power plant and staff training on nuclear safety modeling along with the provision of appropriate software, a better understanding of renewable energy policy and regulations, a renewable energy library, power trading agreements with Georgia, and knowledge gained from exchange visits and training. It is sufficient to say that each of the three major tasks produced positive results – but two more than the third. Based on the above, the nuclear and interconnection tasks were perceived as having produced better results by more stakeholders than the renewable energy task. The fifth evaluation question asks how relevant the intervention was in terms of Armenia’s priorities in energy sector. The issue of relevance is best addressed in terms of major task. For Task 1 (Nuclear), all activities undertaken were certainly very relevant to Armenia’s energy sector priorities, as it is presently a cornerstone of Armenia’s energy sector strategy and the support provided was very helpful. However, this is somewhat clouded by the long lead time required for development and escalating costs of additional nuclear power, which creates a degree of uncertainty. For Task 2 (Renewable Energy), activities were relevant to the extent that they kept awareness of renewable energy alive, which is an issue that Armenia will need to address at some point in the near future as part of its obligation to address climate change. Currently, however, given the GOA’s rather inconsistent policy towards renewables, the intervention did not appear to have been so relevant in terms of Armenia’s current energy priorities. For Task 3 (Interconnection), all activities leading up to meeting and negotiating agreements with Georgia regarding interconnection were very relevant. These agreements are viewed by most stakeholders as the ESRI project’s biggest success. Various analyses undertaken intended to support these activities, however, were less relevant. 11 MAIN RECOMMENDATIONS In view of the overall success of the ESRI project and, subject to GOA commitment, work should continue on all three ESRI tasks given their strategic position in Armenia’s energy sector. There is still much to be done in each area; many stakeholders expressed the sentiment that serious gaps would exist if USAID assistance was terminated. USAID has established good rapport with energy stakeholders within the country, in a relationship that dates back nearly 20 years. Continued involvement would also be consistent with the US-Armenia Memorandum of Understanding (MOU) signed October 18, 2012, dealing with cooperation in a number of sectors, among them energy. Specific recommendations include: • Developing a least-cost power sector expansion plan for the country based on least-cost analysis; • Subject to the results of the above analysis, developing and implementing a strategy for movement on the development of the new nuclear plant; • Also subject to the results of the above analysis, support to GOA and ANRA in undertaking the lifetime extension of the existing nuclear plant; • Support in the development of policies/ laws, and then regulations, that are certain to promote the development of renewable energy technologies; and • Support in expanding the foothold established in regional trade with Georgia, such as the development and signing of commercial trading agreements. 12 EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE The main purpose of this end-of-project performance evaluation is to provide USAID/Armenia with an objective analysis of USAID assistance to Armenia’s energy sector under the $5.9 million Energy Security and Regional Integration (ESRI) project, undertaken over the period 2009-2012. The ESRI project is one of the larger projects in USAID/Armenia’s Economic Growth Portfolio and, as such, is subject to performance evaluation per USAID Evaluation Policy. The findings herein will be used to make informed decisions on the future directions of USAID/Armenia’s assistance in the Energy Sector. Additionally, this evaluation is intended to be of interest to a wide array of stakeholders, including Department of State, USAID/Washington, key stakeholders in the Government of Armenia (GOA), other donors and Armenia’s civil society groups. Thus, the evaluation analyzes issues that may not only result in improvements to the delivery of USAID assistance to the sector, but will also add to the knowledge base of all stakeholders in order to move forward with appropriate strategies to meet Armenia’s energy objectives. Evaluation of the ESRI project was conducted from October 1-12, 2012. EVALUATION QUESTIONS Key evaluation questions focus on whether there is evidence the project contributed to the enhancement of practices and understanding in the Armenian energy sector of the three main task areas of USAID intervention, including nuclear power, renewable energy and regional electricity trade, as well stakeholders’ perceptions of the project’s results and relevance of the various tasks undertaken. The evaluation questions to be addressed in this report include: • Is there evidence that the independent expert services to Government of Armenia for the new Nuclear Power Plant contributed to procedures and activities implemented in accordance with the western safety and environmental standards in preparation of documentation for the New Nuclear Power Plant? • Is there evidence that the scenarios of power system integration contributed to increased understanding of renewable energy development tendencies in the region among the main stakeholders? • Is there evidence that the regional integration task contributed to an increase of regional cooperation with Georgia? • What do stakeholders groups see as the results of the project? • How relevant was the intervention in terms of Armenia’s priorities in energy sector? A complete Statement of Work for the evaluation is provided in Annex I. 13 PROJECT BACKGROUND INTRODUCTION The ESRI project is part of a long history of practically continuous USAID assistance to the Armenian energy sector since the mid-1990s. As part of the USAID mission’s strategy to help Armenia develop an efficient, reliable and cost-effective energy sector to support its emerging market economy, USAID has been supporting Armenia’s efforts to secure diversified energy sources, including nuclear, renewables and international electricity trade. In line with this support, the ESRI project comprises three components: ˙ New Nuclear Plant Development ˙ Renewable Energy Development Strategy and Power System Integration ˙ Regional Electricity System Synchronization and Future Trade and Market Development The evaluation team spent two weeks in Yerevan, Armenia, between October 1 and October 12, 2012, interviewing key beneficiaries of the USAID assistance, reviewing available documentation and discussing/ analyzing findings. A preliminary presentation of findings was made to USAID Mission staff in Yerevan on October 12, 2012. POWER SYSTEM OVERVIEW The power system of Armenia has total installed electricity generating capacity of about 3,500 Megawatts (MW) and total annual production of about 5,800 Gigawatt-hours (GWh). The resulting load factor on existing capacity is about 20% (quite low), which can be explained by the seasonality of hydropower and the relatively low usage of the existing thermal plant capacity. The power system’s actual peak demand is in the 1,200 MW range. Thus, there is much excess capacity. Sources of generation and 2010 production levels are summarized in Table 1. 14 Table 1 – Sources of Electricity in Armenia (2010)3 Type of generation Installed capacity (MW) Energy production (GWh) % of production Nuclear 400 2,400 41.4% Thermal 2,082 1,059 18.3% Large hydro 960 2,020 34.9% Small hydro 103 308 5.3% Wind 3 5 0.1% Total 3,548 5,792 100.0% Armenia’s 400 MW nuclear capacity comprises one unit of a 2x400 MW plant built during the 1970s about thirty kilometres west of Yerevan, in the town of Metsamor. This plant is one of just a few remaining nuclear power reactors that were built without primary containment structures. However, the Armenian government decided to reopen the plant in 1993, shortly after independence, at a time that the country faced great economic challenges. Consequently, one of the reactor units was brought back into operation in 1995. The power plant currently produces about 40% of Armenia's electricity. . The GOA formally agreed in 2007 to close the Metsamor plant after several years of pressure from the United States (US) and the European Union (EU). The plant would be replaced with a new nuclear power plant of about 1,000 MW, with a projected cost in the $5 billion to $7 billion range. It had been hoped that construction of the new nuclear power plant would start in 2011, with commissioning in 2017 to correspond with the closing of the old plant. However, delays have pushed these dates back. It is currently estimated that if detailed design of the new nuclear plant was to begin immediately, it would not become operational until 2021. Hydropower accounts for about as much electricity production as nuclear. Of the 40% share, 35% is classified as “large” hydropower, produced from nine stations located in two cascades – Sevan-Hrazdan and Voratan. “Small” hydropower makes up the remaining 5% share and comprises more than 80 plants producing electricity for commercial purposes. About 60 of these have been developed and constructed over the past 10 years. There are also numerous small (micro) units that are operated by individuals to satisfy their own electricity needs. The development of small hydropower is the cornerstone of Armenia’s renewable energy policy. It is hoped that the share of total production from small hydropower will increase from 5% to 10% within a few years. Wind energy in Armenia comprises a four turbine, 2.6 MW wind farm in the northern region of Lori. The facilities were commissioned in 2005 with Iranian support and are owned by the GOA. The balance of Armenia’s electricity production (about 20%) is from three Soviet-era thermal power plants at Hrazdan, Vanadzor and Yerevan, and a new plant in Yerevan (inaugurated April 2010), running on imported natural gas from Russia and Iran. Because of its relative efficiency, the new Yerevan plant currently provides the bulk of this energy. 3 Source: Small Hydro Power Plants in Armenia, from The Armenian Weekly, June 11, 2012. http://www.armenianweekly.com/2012/06/11/small-hydro-power-plants-in-armenia 15 Imports and exports of electricity to neighboring countries, specifically Georgia and Iran, have historically been in the hundreds of GWh annually. In the last few years, however, annual energy exported to Iran has risen to over 1,000 GWh. Transmission links with other neighboring countries do exist, but these lines are out of service. Armenia wishes to increase electricity trade with its two current trading partners, as evidenced by 400 kV lines to each of these countries currently under construction. With completion of these projects, transfer capabilities will increase well into the hundreds of MW. The rationale for these projects is: ˙ In the case of Georgia, over the long-term, access will be provided to countries of the Black Sea regional energy market. In the near-term, Armenia will have access to low cost Georgian hydropower during summer months, while being able to supply base load nuclear power to Georgia during the winter. ˙ In the case of Iran, the current arrangement of acquiring relatively low cost Iranian natural gas in exchange for electricity can be expanded. A map of the Armenian power system, showing all major generating plants and lines, as described above, is shown in Figure 1. ORGANIZATION OF THE ELECTRICITY SUPPLY INDUSTRY Power sector reform, begun in the mid-1990’s, has resulted in an organization of the electricity supply industry comprising the following main players: • The Ministry of Energy and Natural Resources (MOENR), which is responsible for sustainable electric energy supply to consumers. It also has responsibility for defining the overall policy for energy sector development. • Separate-entity generating stations. Some of these entities are owned by the GOA, while some, notably the hydropower generators, are owned by independent investors. • Electric Networks of Armenia (ENA), a privately-owned distribution and electricity retail company, which is the single buyer for all electricity destined for consumption in Armenia. • High Voltage Electric Networks (HVEN) Closed Joint Stock Company (CJSC), a GOA entity which owns, operates and maintains the country’s high voltage (110 kV and above) network. • System Operator (SO), a GOA entity that essentially dispatches electricity through the country’s power system. • Settlement Center (SC), the GOA entity that calculates the amounts the ENA should pay to all market participants in accordance with the Energy Law. • The independent Public Services Regulatory Commission (PSRC), which, in line with MOENR policy, issues licenses for energy companies, sets and approves tariffs in regulated businesses and draws up regulations to define relationships between stakeholders in the energy sector. The PSRC sets the feed-in tariffs for all generators, the transmission tariff for high voltage networks, and the service fees for the operator and the settlements center. • The Armenian Nuclear Regulatory Agency (ANRA), whose task is the regulation of atomic energy utilization, aimed at ensuring the safety of personnel and the population, the environment, and the Republic of Armenia. 16 Figure 1 – Power System of Armenia Map Source: Global Energy Network Institute (http://www.geni.org/globalenergy/library/national_energy_grid/armenia/armeniannationalelectricitygrid.shtml) 17 THE ESRI PROJECT In late 2009, PA Consulting (now Tetra Tech) was recruited by USAID as the contractor to carry out the ESRI project. Tetra Tech’s Statement of Work (SOW) for the project comprised three major tasks. Task 1: New Nuclear Plant Development Under this task, the contractor was to provide services to the GOA through the following activities: 4) Upon the request from the MOENR, provide information and independent expert services with respect to the new nuclear unit development, including its planning, design, tendering and construction; 5) In coordination with US Nuclear Regulatory Commission, provide assistance to Armenian Nuclear Regulatory Agency (ANRA) to ensure that important elements needed for regulatory review and plant licensing are developed; and 6) Help implement critical human capacity building actions identified by the International Atomic Energy Agency (IAEA) and encourage donors and other interested parties to finance education and training programs. Expected results were: ˙ MOENR will receive necessary information and advice with respect to the new nuclear plant development; ˙ Assistance will be provided to ANRA to ensure that important elements needed for regulatory review and plant licensing are developed; and ˙ Critical workforce development and technical training requirements will be strengthened, leading to increased government and donor funding for the existing in-Armenia training centers, universities and institutes. Task 2: Renewable Energy Development Strategy and Power System Integration Under this task, the contractor was to undertake the following: 7) Analyze and make recommendations concerning legal, regulatory, and market issues in developing renewable energy investment projects; 8) Provide assistance to PSRC in improving the regulatory framework and tariff policy to support development of wind and solar energy; 9) Conduct static and dynamic stability analysis and consider the implications of renewable energy development for grid investment and operation; 10) Assist the MOENR in conducting research study on development and grid connection of solar energy innovative technology, identified by the MOENR; 11) Provide appropriate software and carry out system operational simulations to help plan for more economic dispatching and effective transmission system management (in coordination with work under task 3; and 12) Facilitate access to information and knowledge transfer about US experience, technologies, and companies active in renewable energy. 18 Expected results were: • Contribute to increasing the number of renewable energy projects and their output of electricity, thereby contributing to diversifying domestic energy supply by at least 15 % and reducing or avoiding greenhouse gas emissions. • Contribute to the creation of an improved investment climate for companies seeking investment opportunities in renewable energy and clarified issues related to grid access and dispatching of these renewable units. • The GOA will promulgate a clearer policy and strategy on renewable energy development and more accurately estimate the potential contribution of these sources and innovative technologies to meeting future energy demands. • As a result of the solar energy research study, the innovative technology will be tested. • Due to project efforts, transmission planning and analysis of grid operational issues – arising from the increasing output of a growing number of renewable projects – will be improved and appropriate transmission investments pursued. Task 3: Regional Electricity System Synchronization and Future Trade and Market Development Under this task, the contractor was to undertake the following4: 5) Regional system synchronous operations, system stability studies and training; 6) Technical assistance and advisory support for harmonization of legal and normative documents governing interregional cooperation; 7) Transmission tariff and investment analysis; and 8) Support of partnership for transmission organizations, exchange visits and training. Expected results were: • Improve transmission planning capabilities of Armenia Transmission Company in order to support Armenia system development and greater integration into the Caucasus system, • Improve Settlement Center’s capacity to calculate power flows and losses in the system in order to perform its critically important role, • Develop plans to integrate Armenia and Georgia power systems into a full interconnection mode. 4 As per Tetra Tech’s Quarterly Reports. The sub-tasks in the SOW provided to the evaluation team by USAID were: i) support for transmission planning and training with the System Operator, Settlement Center and Scientific Research Institute for Energy, ii) support to a proposed electricity interconnection Working Group between Georgia and Armenia and collaboration on associated technical studies, regulatory issues and cross-border trading issues; and iii) assistance for analysis, benchmarking studies, and training on electricity systems in the region and the potential growth in demand and generation capacity. It is assumed that these sub-tasks were revised prior to commencement of the contractor’s services. 19 EVALUATION METHODS & LIMITATIONS Prior to conducting in-country data collection, the evaluation team employed a comprehensive desk review of project work plans, quarterly reports, and numerous technical reports and documents. The project produced many reports and, indeed, many of the project’s results were the reports themselves (as opposed to a reporting of the results). The reports can be found on the ESRI project website http://www.armesri.am/Public_Docs/Reports.html. The usefulness of these reports, together with other results of the project, was assessed after face-to-face interviews with the beneficiaries of the assistance. This method was selected because direct beneficiaries were relatively few and easily identifiable. Details of the rationale used for selecting the particular beneficiaries and interviewees are provided in Annex II. Key informant interviews with beneficiaries of the USAID assistance and stakeholders within the Armenian energy sector were then conducted in Yerevan. Interviewees included Mission staff, the Chief of Party for the implementing contractor, a local subcontractor to the US contractor, heads of various organizations that were direct beneficiaries of the assistance and, the only other international lending agency heavily involved in the Armenian energy sector, KfW. For the interviews, questionnaires were prepared in advance and varied to some extent depending on the stakeholder. These were generally used as a guide in approaching the interviews and were never read verbatim to the interviewee. The questionnaires are provided in Annex III. A full list of documents reviewed and interviewees is provided in Annex IV. The ESRI project had many beneficiary organizations, each of which benefited from different components of the project. The particular people interviewed were suggested by either Mission staff or the USAID contractor (or both). Presumably, these suggestions were based on their perceptions of who would have the best knowledge of the assistance provided within each organization. Although a full list of interviewees is provided in Annex IV, the key beneficiaries interviewed included: • Deputy Minister of Energy, MOENR • Head, Atomic Energy Department,, MOENR • General Director, CJSC Settlement Center, MOENR • General Director, Electro Power Systems Operator, MOENR • General Director, High Voltage Electric Networks CJSC, MOENR • General Director, Scientific Research Institute of Energy CJSC • Head and Chair of Heat and Power Engineering, State Engineering University of Armenia • Dean of Faculty of Physics, Yerevan State University • Chairman, Armenian Nuclear Regulatory Authority • Head, Licensed Activities Monitoring and Investment Program Department, Public Services Regulatory Commission (the Commissioner was not available) Given the nature of the KQs, interviews were deemed the most appropriate qualitative evaluation approach. As with any interview, recall bias and selection bias posed potential threats to internal validity of data. The evaluators determined that recall bias could be effectively mitigated, however, as a result of 20 the project’s recent termination and high recall capability of those interviewed. Interviewees were heavily involved in the ESRI tasks, which allowed for high recall capability, even for events dating from the beginning of the project (about three years). Selection bias posed a larger issue, as all interviewed beneficiaries were direct participants in the project and were recommended by either USAID or the contractor. As expected, all of these interviewees held generally positive views of the project. Identifying respondents who could be classified simply as observers was challenging, mainly because the nature of the product (e.g., nuclear safety regulations) made it difficult to find a neutral, detached party with insufficient interest in the subject matter to have formed a strong opinion. The most informed expert on the assistance provided consistently remained the beneficiary. The only true unbiased informed interviewee was the KfW representative in Armenia, who found the USAID assistance valuable and complementary to that provided by KfW (as opposed to in competition with). Selection bias in this case was mitigated partly by the tangible nature of the results themselves - e.g., the fact that breakthrough agreements were made with Georgia, the actual introduction of more appropriate nuclear engineering/ physics curricula at local universities, the existence of a particular product-document (which, admittedly, would then have been judged by the beneficiary as useful or not, but still, it is a tangible result). Another strong mitigative factor is the technical knowledge base of the evaluation team, which enabled credible professional judgments on the work to be carried out. For example, as will be seen further on, a question arises on the suitability of some of the software used to undertake certain technical analyses. None of the beneficiaries questioned this; only the evaluation team did. To correct for potential selection bias, the evaluation team attempted to triangulate qualitative data with data from project reports and other documents, where applicable. Also, in order to put the stakeholders’ views into an appropriate context, achievements were compared with original expectations, as stated in the original project design and Statement of Work. A final check on potential selection bias is the comparison of the very real, tangible results with original expectations. Although the interviews may provide certain evidence that the project generated certain results, the question then arises of how significant the results actually were; i.e., the results, though evident, may, perhaps, have fallen far short of expectations. Therefore, it was considered useful to compare achievements with original expectations, as may have been stated in the original project design (these expected results have been summarized in the previous section describing each task of the ESRI project). Of course, changing circumstances over the course of the project and other factors certainly did influence many of the project outcomes – both positively and negatively. Therefore, to the extent possible, the analysis that follows also offers explanations on how and why particular outcomes occurred. An additional shortcoming is the reliability of the interviewer to respond truthfully, as a respondent may have particular motives for answering in a certain manner. For example, an interviewee may find great value in a particular component of the assistance (e.g., new equipment), while not finding anything useful in another type of assistance provided. In such a case, he or she may feel compelled not to jeopardize his or her chances of receiving similar help in the future. This shortcoming was mitigated in two ways. First, to the extent possible, revealing statements made by any particular interviewee were cross￾checked and verified through what was said in other interviews. Second, technical aspects of the assistance provided were reviewed to some extent (and at a relatively high level) and some limited professional judgments were made by the evaluators on the merits of the work itself. 21 FINDINGS & CONCLUSIONS EVALUATION QUESTION 1 Is there evidence that the independent expert services to government of Armenia for the new nuclear power plant contributed to procedures and activities implemented in accordance with the western safety and environmental standards in preparation of documentation for the new nuclear power plant? FINDINGS The above question addresses the nuclear component of the three project tasks. This component, in turn, included assistance, mainly in the form of various reports, to: i) MOENR, ii) ANRA and iii) the two major universities in Armenia, i.e., the State Engineering University of Armenia (SEUA) and Yerevan State University (YSU). In the case of the MOENR support, accomplishments noted by stakeholders included: • Support in the preparation of four public hearings held over 2010-11 on development of a new nuclear power plant (NPP) at Metsamor. According to the Head of the Atomic Energy Department at MOENR, this activity was particularly helpful, as the MOENR had no previous experience in conducting public hearings and required guidance to undertake them. • Preparation of an environmental assessment report for the new NPP. It should be noted that this is not the Environmental Impact Assessment (EIA) that would typically be conducted for most large infrastructure projects, but an initial environmental assessment for potential investors to acquire an appreciation/ understanding of the NPP’s potential environmental effects and the mitigative steps envisaged to be undertaken by the project’s sponsors. The report can be found on the GOA website http://www.minenergy.am. Both the Head of the Atomic Energy Department and the Deputy Minister cited this document as a major contribution by the ESRI project, as ESRI efforts in this area span the three years of the ESRI project, beginning with notifications, first public hearing, input for the initial environmental report, two sessions of the second hearing, and expertise review.. • Documents and procedures on spent fuel and nuclear damage liability. These documents cannot be found on the ESRI website, but their existence was verbally communicated to the evaluation team by the Head of the Atomic energy Department. The MOENR appreciated these documents because previously they had very little knowledge of the subjects from an international perspective. In the project’s SOW, the contractor was tasked with providing information and independent expert services with respect to the new NPP development, including its planning, design, tendering and construction. Relatively little progress has been made by the GOA in reaching the NPP’s design stage. Thus, this particular subtask became somewhat dated as the project progressed. However, the above results may be regarded as contributing significantly to the NPP’s development. It is worth mentioning that the lack of progress in NPP development is a concern. With 22 decommissioning of the existing plant set for September 2016 and, given the lead time required for NPP development (about seven years if detailed design was to start immediately), there is evidently a time gap during which no nuclear power will be available, unless other options such as life extension of the existing plant are explored (and, as soon as possible). In the case of ANRA support, accomplishments noted by the Chairman of ANRA and the Director of the Nuclear and Radiation Safety Center, were: • Development of the Safety Analysis and Report Guide (SARG). This 300 + page report provides details of NPP safety requirements for licensing purposes, based on guidelines of the International Atomic Energy Association (IAEA) and the US Nuclear Regulatory Commission (NRC). ANRA will now compare the contents of this report to similar Russian requirements, as the new NPP will likely be Russian made. • Capacity building at ANRA on the use of software to evaluate safe operation of NPPs. This included provision of specialized ANSYS CFX software, which simulates the operation of a NPP under a variety of scenarios to assess safety, as well training for ANRA staff to run the model. In the project’s SOW, the contractor was tasked with coordinating with the NRC to provide assistance to ANRA to ensure that important elements needed for regulatory review and plant licensing are developed. The SARG appears to have done this, as evidenced by over 300 pages of details of NPP safety requirements based on standards of the IAEA and NRC,. In addition, appropriate NPP simulation software and training has been provided to ANRA. In the case of institutional capacity building to Armenia’s main State universities, the most significant accomplishment, as ascertained from interviews with the Head of Thermal Power Engineering at State Engineering University of Armenia and the Dean of the faculty physics at Yerevan State University, was the development of curricula in nuclear plant engineering and nuclear physics at two universities in Yerevan. This need rose to the surface in 2010, when the GOA decided that the two universities in Armenia teaching nuclear engineering/ nuclear physics should change their curricula to improve the quality of education. This was driven by the decision to construct a new NPP. Thus, the USAID decision to help the process was very timely. The SEUA moved quickly to study various reports issued by the contractor on nuclear engineering curricula at other universities in the world, which ultimately led to the introduction of a new Bachelor level program in nuclear engineering at SEUA in the fall of 2012. In the case of YSU, the university has closely studied and has largely agreed with the suggested curricula; however, it has been slow to implement the recommended changes. Changes, however, are being made. In the project’s SOW, the contractor was tasked with helping to implement critical human capacity building actions identified by the IAEA and encourage donors and other interested parties to finance education and training programs. The desired result was that critical workforce development and technical training requirements will be strengthened, leading to increased government and donor funding for the existing in-Armenia training centers, universities and institutes. On its part, the GOA has increased its support to this capacity building effort through increasing the number of “free” (i.e., no tuition) places in the nuclear engineering program at SEUA and in nuclear physics at YSU. CONCLUSIONS There is strong evidence that the independent expert services to the GOA for the new NPP have contributed to procedures and activities implemented in accordance with Western safety and environmental standards in preparation of documentation for the new NPP. 23 Although the process of developing the new NPP has been slower than anticipated, progress has been significant in: i) support provided to the MOENR in holding public hearings and in preparing vital documents and ii) support to ANRA in the development of a safety guide for licensing purposes based on Western standards. Without ESRI’s contribution, NPP development would not have proceeded to the degree that it has, but seemingly in the face of lack of GOA action. While MOENR cites a lack of funding from its Russian partner as the main obstacle, there may be a lack of political will on the part of the GOA to act decisively. For example, an IAEA study in 2007-2008 recommended a staff of 20 or more at MOENR for management of this project, which has yet to materialize - and this would not cost the GOA very much. The adoption of the new curricula has definitely strengthened the teaching of nuclear education, which has, and will, encourage increased donor support, as the IAEA is evidently the only other foreign donor to Armenia’s nuclear programs, so the IAEA is likely to continue its funding, given current developments; and iii) development of curricula in nuclear engineering and nuclear physics that are currently being adopted. 24 EVALUATION QUESTION 2 Is there evidence that the scenarios of power system integration contributed to increased understanding of renewable energy development tendencies in the region among the main stakeholders? FINDINGS The above question addresses a relatively small part of the renewable energy component of the three major project tasks, as it covers only two of the original six sub-tasks. Accordingly, this section directly answers the question and ignores the other four sub-tasks. These other sub-tasks are addressed under the two more general evaluation questions dealing with stakeholder perceptions of the project results and relevance. The ESRI project examined scenarios of power system integration under the following two sub-tasks: • Conduct static and dynamic stability analysis and consider the implications of renewable energy development for grid investment and operation. • Provide appropriate software and carry out system operational simulations to help plan for more economic dispatching and effective transmission system management (in coordination with work under task 3). For this purpose, Tetra Tech sub-contracted the analysis work to the Scientific Research Institute of Energy (SRIE), a state company under the purview of the MOENR. Personnel within this organization were trained to run power system analysis software, including, among others, PSS/E and GTMax. PSS/E is used to assess power system stability by simulating load flows under various conditions, while GTMax undertakes certain economic analyses of given power system generation configurations. Three scenarios of power system integration were examined from a stability standpoint (using PSS/E) in tandem with a renewable energy development scenario that was stated as being consistent with GOA renewable energy targets. The two forms of renewable energy used in the PSS/E simulations, hydropower and wind, have unique production characteristics in terms of output and location that might affect power system stability. If stability were affected, then the system would require additional investment to mitigate the effects. Having renewable energy data as inputs to the integration analyses can be interpreted as contributing to understanding renewable energy tendencies in the region in that the specific effects of adding renewable energy to the system, vis-à-vis their production characteristics, are being fully considered. The conclusion with respect to renewable energy in each of the “Phase I” and “Phase II” reports on system stability can be summarized as follows (from the Phase II report): “The program for development of RE [Renewable energy] of Armenia in 2015 and 2020 approved by the Ministry of Energy and Natural Resources does not need changes, since the calculations demonstrated that RE integration into the system did not meet obstacles, i.e. the recommended secondary and tertiary power reserves of the Armenian power system would ensure compensation of non-controllable changes of wind capacities. Besides, to transfer controllable capacities there is no need to construct new lines” In other words, adding renewable energy to the extent envisaged has no material effect on the system. This conclusion might not have been surprising to a power system engineer. In fact, a read through the reports suggests a pre-occupation with stability issues under three different scenarios of synchronous 25 operation with neighboring countries, with the renewable energy issue relegated to being quite secondary in nature. This leads one to question whether the inclusion of renewable energy in the scenarios was really a knowledge building exercise as far as renewable energy is concerned. Perhaps the finding that additional renewable energy sources would not materially affect power system stability has led to some sort of increased understanding of renewable energy. Another consideration is that the interconnection with Georgia draws on low-cost Georgian hydropower during summer months, thus contributing to increased awareness of regional renewable energy sources. This is more of an interconnection issue, however. With respect to economic issues, the results of the GTMax software showed a slight increase in production cost with renewable energy included in the production mix, which should also not be surprising. The foregoing discussion is not meant to question the wider application of the technical and economic analyses undertaken with the PSS/E and GTMax software – only that related to renewable energy possibilities, which, at this point in time, do not appear to require system simulation software to understand. One might argue that it is a matter of being able to model the variable and relatively unpredictable nature of renewable energy production such as wind and solar in PSS/E, however small. But it seems this is relatively easy to simulate. When asked about increased understanding of renewable energy, the beneficiaries involved in analyzing the results of the simulations, including the System Operator (SO), the Settlement Center (SC) and High Voltage Electric Networks (HVEN, which is the company that owns and maintains the country’s high voltage network), all said their understanding of renewable energy had increased. However, their elaboration on this question (regarding evidence) then focussed on the other renewable energy components of the project, and not about how the integration simulations increased understanding. Thus, the only evidence of increased understanding of renewable energy tendencies in the region arising from the integration scenarios are the unanimous stakeholder conclusions from the exercise that: i) there are no adverse effects on system stability as a result of adding renewables and ii) there is a small accompanying increase in overall cost. This could be interpreted as evidence of increased understanding of renewable energy tendencies in the region. In any case, using such tools for any purpose, even for a not so particularly useful one, increases the capacity of Armenian power system specialists to undertake the analyses for more useful exercises - such as examining the effects of regional integration. CONCLUSIONS Including renewable energy in the system technical and economic simulations of power system integration can be said to have contributed somewhat to increased understanding of renewable energy development tendencies in the region among the main stakeholders. However, to most knowledgeable stakeholders (e.g., power system engineers), the results of these simulations with respect to renewable energy may have been a foregone conclusion. The simulations were not found to have added substantially to stakeholders’ renewable energy knowledge base. 26 EVALUATION QUESTION 3 Is there evidence that the regional integration task contributed to an increase of regional cooperation with Georgia? FINDINGS The above question addresses the interconnection component of the three project tasks. This component, in turn, included: i) regional system synchronous operations, system stability studies and training, ii) technical assistance and advisory support for harmonization of legal and normative documents governing interregional cooperation, iii) transmission tariff and investment analysis, and iv) support of partnership for transmission organizations, exchange visits and training In the case of regional system synchronous operations, system stability studies and training, the system stability analyses discussed in the previous section analyzed system stability under a number of synchronous operation scenarios. The analyses were undertaken by SRIE under sub-contract to Tetra Tech and the report was reviewed by the SO, the SC and HVEN. Related training was carried out for these participants in both the US and Serbia. In the project’s SOW, the contractor was tasked with conducting the above analyses and with providing associated training, as well as developing regional capacity and load balances for 2015 and 2020 – which are, in any case, inputs to the system stability studies. The SOW also required that five PSS/E educational licenses were to be provided to the SEUA, which also included training of staff. Although not verified during the interviews, the PA Consulting Quarterly Report dated April 2010 stated that training was indeed provided on Load Flow and Short Circuit Analysis using PSS/E to the HVEN, SEUA, SC, SO and SRIE and that five educational licenses were provided to the SEUA. Thus, the work carried out matches the original requirements. The results of the simulations were shared, through joint working groups, with Georgian counterparts and served as a basis for discussion during the interconnection negotiations. As all those interviewed agreed that the interconnection agreements with Georgia were a major accomplishment, the sharing of simulation results constituted a big part of the increase in regional cooperation with Georgia that resulted from undertaking the task. Technical assistance and advisory support for harmonization of legal and normative documents governing interregional cooperation basically addresses support provided to MOENR in reaching an agreement with the Ministry of Energy of Georgia and signing a Memorandum of Understanding (MOU) between the two ministries. In 2010, Armenia and Georgia did sign such an MOU, which laid the groundwork for subsequent joint activities aimed at integration of the power systems. The joint Working Group formed as a result of the MOU subsequently developed agreements on parallel operation of the two systems and provisions for emergency power. The Working Group is currently in the process of negotiating a transit agreement, mainly for the purpose (from Armenia’s perspective) of selling electricity to Turkey via Georgia. In the contractor’s SOW, the specific objective associated with this sub-task was to “develop plans to integrate Armenia and Georgia power systems into a full interconnection mode”. Thus, it appears that this objective was not only met, but far surpassed. Most stakeholders view the particular accomplishments under this subtask as being the biggest success of the ESRI project. Some stakeholders 27 went further, to say that without Tetra Tech’s intervention as a well trusted intermediary between the parties during often difficult negotiations, the countries would probably still be quite far apart today in reaching any agreements on electricity trade. In the case of the transmission tariff and investment analysis, the most evident products appear to have been the development of the report “Proposed Methodology for Transmission Line Cost Benefit Analysis” and the reports on stability analysis where Armenian power system transmission requirements have been physically defined to the year 2020. However, capital investments associated with these needs have not been estimated. And, the methodology for the cost-benefit analysis remains only a methodology. Needless to say, with no investment requirements, there cannot be any transmission tariff developed, although principles for the determination of a transmission tariff for exports are alluded to in some of the technical reports. In Tetra Tech’s third year work plan, it was mentioned that Tetra Tech had developed a draft report “Evaluation of the Economics, Financial Viability, and Environmental Consequences of the Proposed Georgian Transmission Interconnection”, describing the potential economic benefits of investment projects that could benefit regional (and Armenian) energy systems and analyzing the existing transmission tariffs to determine the extent to which these tariffs are economically justifiable and do not present undue obstacles to cross-border trading. This report is not available on the ESRI website. In any case, given the title of this activity and the lack of any further work in this area, there seem to be some “loose ends” in that: i) neither the methodology for the cost-benefit analysis is developed any further in terms of a real cost-benefit analysis (subject to the contents of the above report), ii) a breakdown of transmission investment needs has not been estimated and iii) little has been done on development of transmission tariffs. On the other hand, the contractor’s SOW does not set out any specific requirement to further address these issues. An expected outcome of task 3 that matches this activity is that the transmission planning capabilities of HVEN would be improved. It is not clear that this happened because no formal transmission planning was undertaken by HVEN in conducting the task. On the other hand, the fact that HVEN has been a participant in negotiations with Georgia, where very technical transmission issues were probably discussed, indicates that HVEN did benefit in terms of its transmission planning capabilities. However, the unresolved loose ends of this sub-task suggests that, by itself, the task probably did not contribute much in terms of increasing regional cooperation with Georgia. In the case support of partnership for transmission organizations, exchange visits and training, this activity is about support provided in the formation and interaction of joint working groups for the purpose of developing electricity trading with Georgia. The two working groups (one of them “technical” and the other one more general) have met on a number of occasions to this end. This subtask is closely related to the advisory services provided to MOENR discussed above and the distinction between the two is not clear. Whether they are indeed the same or whether there is a subtle difference, the activity has been carried out extensively, which has led to a few agreements having been made with Georgia on electricity trading. Finally, it should be mentioned that one of the expected outcomes of the interconnection task was to improve the SC’s capacity to calculate power flows and losses in the system in order to perform its role. Aside from the question as to why the SC should be undertaking these functions (in other places, addressing such issues would normally be the role of the SO), there is no subtask that matches this expected outcome. In any case, the SC did cite a losses calculation program provided by Tetra Tech as being useful for its purposes, as well the economic dispatch/ production modelling program GTMax. 28 CONCLUSIONS There is ample evidence that the regional integration task contributed to an increase of regional cooperation with Georgia. An agreement has been reached on parallel operation of the Armenian and Georgian power systems. An agreement has been reached on Armenian-Georgian power exchanges for emergency and unplanned situations. Finally, discussions on provisions for transit power are underway. These major accomplishments far outweigh the noted shortcomings of some of the subtasks and expected outcomes. 29 EVALUATION QUESTION 4 What do stakeholders groups see as the results of the project? FINDINGS The three major task areas had, as their beneficiaries, different organizations, including electricity market participants, regulatory authorities and educational institutions. Their perceived results therefore differ considerably. These results are summarized below by task. Task 1: New Nuclear Plant Development Various support provided to development of the new NPP was cited by PSRC and MOENR as being important. While PSRC’s observation was general in nature, the Deputy Minister specifically pointed to the environmental assessment as being a very good and “complicated” document. The Head of the Atomic Energy Department praised the work carried out by Tetra Tech under this activity and specifically mentioned: i) the environmental assessment, ii) the support to the universities in developing the new nuclear curricula, iii) reports on spent fuel and nuclear damage liability, iv) review of the Worley Parsons feasibility study of the new NPP and v) the support provided for the new NPP public hearings. From interviews with ANRA senior staff, the results for ANRA were the development of safety requirements for the new NPP and staff training on nuclear safety modeling along with the provision of appropriate software. For the State University of Armenia, the result was the adoption (this year) of new curricula in nuclear engineering based on the work carried out by Tetra Tech, while for Yerevan State University, it was the development of nuclear physics courses by Tetra Tech that are presently slowly being incorporated. Task 2: Renewable Energy Development Strategy and Power System Integration The only stakeholder to mention an accomplishment under this task was PSRC, which gained a better understanding of renewable energy policy and regulations. Also, this task financed a renewable energy library at the PSRC premises, of which it is very proud. Task 3: Regional Electricity System Synchronization and Future Trade and Market Development MOENR, PSRC and all GOA market participants including the SO, SC and HVEN cited the agreements with Georgia and the promise of increased regional trade in electricity as being an important (and probably the most important) result of the ESRI project. Other results PSRC noted better understanding of electricity regulation through ESRI’s exchange visits. In particular, PSRC is involved in an exchange program with the regulatory authority of Iowa through a program partially financed by the ESRI program and the National Association of Regulatory Utility Commissioners (NARUC) and the United States Energy Administration (USEA). Also, PSRC has participated in six meetings over the last two years of the Black Sea Regional Regulatory Partnership. PSRC, MOENR, SC, SO and HVEN all mentioned short-term training conducted over the period of the ESRI project in Armenia, the US and Serbia. 30 CONCLUSION As can be seen above, depending on how they benefitted from the ESRI project, stakeholders viewed the results of the ESRI project differently. It is sufficient to say, however, that each of the three major tasks produced positive results – but some more than others. The nuclear and interconnection tasks were perceived as having produced better results by more stakeholders than the renewable energy task. 31 EVALUATION QUESTION 5 How relevant was the intervention in terms of Armenia’s priorities in energy sector? FINDINGS The issue of relevance is addressed below by major task: Task 1: New Nuclear Plant Development For the nuclear task, all stakeholders agreed that the work produced by the contractor was valuable and, thus, relevant to their ongoing work. Since 2007, the GOA has been committed to building a new nuclear plant as a replacement to the existing Metsamor nuclear plant, which had been targeted for retirement in 2016. This commitment has remained in place for five years, even after the Fukushima disaster prompted some countries to abandon their nuclear power programs. However, it has been estimated that the current lead time to new NPP commissioning is nine years – and that is if detailed design begins immediately. Many possible developments could intercede during this time period that can result in abandonment of the project. Further, nuclear power plant capital costs have escalated rapidly – to roughly $7 billion – since the original $5 billion estimate for a 1,000 MW plant. Moreover, the last least-cost power sector expansion plan for Armenia is now six years old. Task 2: Renewable Energy Development Strategy and Power System Integration This task produced reports on: i) barriers to small hydropower and wind development in Armenia and actions required to promote renewable energy development, ii) system stability analysis under various scenarios, iii) solar and wind connection standards, iii) wind power purchase agreements and iv) a very general report on solar power. Also, a one-week solar study tour of the US was conducted and a renewable energy library was established at PSRC. The stakeholders’ perceived results of the ESRI project, provided in the previous section, are very telling, as only PSRC initially referred to any of the above work when asked about the ESRI’s results. Other stakeholders seemed happy with the renewable energy work, when prompted. As previously noted, the original renewable energy subtasks were: i) an analysis of legal, regulatory and market constraints, ii) provision of assistance to PSRC to support development of wind and solar energy, iii) conduct the system stability studies, iv) assistance in a research study on solar technology, v) conduct economic dispatch studies and vi ) a US study tour. Aside from the system stability and load flow studies (addressed further below), stakeholders, including PSRC, SO, SC and HVEN generally found the reports produced on the other activities “interesting” and “useful for comparison purposes”. A US study tour on renewable energy did take place. However, a 10 kW solar pilot project developed out of the solar research study was cancelled due to tendering difficulties. It would be useful to compare the accomplishments noted above with the originally expected results: • Contribute to increasing the number of renewable energy projects and their output of electricity, thereby contributing to diversifying domestic energy supply by at least 15 % and reducing or avoiding greenhouse gas emissions. - Not realized • Contribute to the creation of an improved investment climate for companies seeking investment opportunities in renewable energy and clarified issues related to grid access and dispatching of these renewable units. 32 - Not realized, unless one considers an exemption from regulations for projects up to 150 kW. • The GOA will promulgate a clearer policy and strategy on renewable energy development and more accurately estimate the potential contribution of these sources and innovative technologies to meeting future energy demands. - Not realized • As a result of the solar energy research study, the innovative technology will be tested. - Not realized • Due to project efforts, transmission planning and analysis of grid operational issues – arising from the increasing output of a growing number of renewable projects – will be improved and appropriate transmission investments pursued. - As seen in a previous section, it is not clear whether transmission planning capabilities increased. Certainly, there was no large increase in renewable energy projects that warranted extra transmission requirements. When asked about renewable energy, the Deputy Minister of MOENR said “it is not so important” and that he is “not so involved”, indicating a lack of GOA support and leadership in this area. While there appears to be some interest in developing small hydropower (and one project in the 50 MW range), a foray into other renewable energy technologies does not appear to be on the horizon, as evidenced by the lacklustre performance of the renewable energy task. It appears that the contractor tried to deliver on the renewable energy subtasks, but was effectively stymied by a lack of interest. It should, however, be mentioned that the Deputy Minister, in the same interview, also noted the imminent signing of a MOU with the US Ambassador about future energy cooperation, which among other areas of interest, includes renewable energy5. These mixed signals by the GOA were also noted by the KfW representative in Yerevan, who mentioned that, although the German government had been ready to support the development of renewable energy projects in Armenia, the GOA’s approach on the matter has been inconsistent and, therefore, they have, for the time being, decided not to provide any funding for renewable energy projects. Task 3: Regional Electricity System Synchronization and Future Trade and Market Development Armenia has, in general, been relatively isolated since its independence in 1991, surrounded by not-so￾friendly and/ or politically motivated neighbours. Its electricity trading opportunities have, as a result, been restricted for the last 20 years. It has therefore been a significant accomplishment for the ESRI project to have opened a door to electricity trading with a country other than Iran, not only because of the diversity of supply from a second source, Georgia, but also because of the opportunities it presents in terms of trading within the larger Black Sea market. This accomplishment far outweighs some of the shortcomings noted below in the analyses undertaken, by both the PSS/E and GTMax programs. Also, the fact that training has been carried out on using this software, notably PSS/E, is also important. Armenian engineers can now carry out load system stability analyses independently and on a regular basis. However, the PSS/E analyses suffer from a couple of weaknesses. First, they do not include Iran and are therefore unrealistic. Second, all scenarios examined assumed synchronous operation with neighbours, 5 This MOU was actually signed October 18, 2012. 33 which is plausible, but it is not a comprehensive analysis. Asynchronous operation is a distinct possibility (“Armenia in isolation” scenarios might possibility address this issue, but this is not mentioned in the reports). The GTMax shortcomings are more fundamental. GTMax is essentially a production modelling program that simulates the operation of a given power system using the principle of least-cost dispatch. It can therefore calculate both the average and short-run marginal costs of production by hour using a fixed combination of generating sources and their incremental costs of production. It can have a number of applications; for example, its results might provide useful information to system dispatchers or, it may be used for tariff purposes in calculating prices and revenues under a real-time pricing regime. Under the ESRI project, GTMax no doubt provided useful and relevant information for the interconnection scenarios. Likely power flows established under a regional least-cost dispatch scenario probably served to better interpret the results of the stability analyses under PSS/E. Also, GTMax provided incremental production costs under different operating scenarios. The results of this latter facility were often quoted by stakeholders with respect to the significant difference in future production costs – by time-of-day even – between interconnected and non-interconnected scenarios. GTMax does not consider capital costs, not only of generation, but also of associated main transmission. For the year 2020 – eight years from now – the scenarios had to be based on a guess (or guesses) of the likely mix of generating sources. But more important, the capital costs of the associated facilities were not included. It is understood from Mission staff that a reason for the contractor not using appropriate planning software for economic analysis is that the GOA did not want a least-cost planning study. This is not a good reason. The goal of the analyses would have been to examine various interconnection (and non￾interconnection) scenarios – not to develop a least-cost plan for the country (at least, not directly) – and results would have been presented in reports in that manner. As per the GOA’s desire, they would not have had a least-cost planning proposal on the table for their consideration (although the results of the least-cost analysis would have been readily available). That would have been a good thing, as decision-makers in Armenia’s electricity supply industry would at least have been up-to-date on the directions they should probably be taking in developing the industry – to be formalized by an actual least-cost planning study at some point in the future when the GOA wants it. As it stands, the optimum course of action remains generally unknown, bringing about the need to rely on a least-cost expansion program that is six years old. CONCLUSIONS For task 1 (Nuclear), all activities undertaken, including assistance to MOENR, ANRA and the universities, have been very relevant in terms of Armenia’s priorities in the energy sector. They will continue to be relevant if and when a new NPP is constructed, depending on how events unfold over the coming decade. For task 2 (Renewable Energy), activities were relevant to the extent that they kept awareness of renewable energy alive, which is an issue that Armenia will need to address at some point in the near future, as part of its obligation to address climate change. Currently, however, given the GOA’s rather inconsistent policy towards renewables, the intervention did not appear to be so relevant in terms of Armenia’s current energy priorities. Regarding the analytical work carried out under the PSS/E and GTMax simulations, it has already been noted that the impact of renewable energy developments in the near-term really do not require any computer based simulations regarding power system effects. Thus, their relevance is questionable. 34 For task 3 (Interconnection), all activities leading up to meeting and negotiating agreements with Georgia on interconnection were very relevant to Armenia’s energy sector priorities of obtaining secure and reliable supplies of electricity – and doing so cost-effectively. On the other hand, analyses intended to support these activities, such as GTMax, were less relevant. The analyses would have been a lot better served with least-cost planning software such as WASP (the most well-known), or, as has previously been used in Armenia, MARKAL. In any case, there are many such programs available. Such a program could have established a least-cost planning sequence for power facilities – comprising generation and main transmission – not only for Armenia, but also for the region. The cost comparison between an isolated and interconnected Armenia would then have been much more meaningful. Instead of running through a few interconnection scenarios based solely on economic dispatch of generation with GTMax, and then concluding that “yes, our variable costs of generation will be much lower in the case of interconnection than in an isolated scenario”, a least-cost planning exercise would have better pinpointed a likely best choice scenario (and maybe also answered questions such as whether a new NPP was necessary in the face of the availability of less expensive electricity imported from neighbouring countries). GENDER The ESRI project has no gender component. Also, the project has addressed relatively technical topics that are gender-neutral. As such, there have been no discernible gender issues to address. 35 RECOMMENDATIONS In view of the overall success of the ESRI project and, subject to GOA commitment, work should continue on all three ESRI tasks given their strategic value in Armenia’s energy sector. There is still much to be done in each area; many stakeholders expressed the sentiment that serious gaps would exist if USAID assistance was terminated. USAID has established good rapport with energy stakeholders within the country, in a relationship that dates back nearly 20 years. Continued involvement would also be consistent with the US-Armenia MOU signed October 18, 2012, dealing with cooperation in a number of sectors, among them energy - and specifically, renewable energy, energy efficiency, nuclear energy and related education. More specific areas of involvement in the future could be: 1) Strategy for new NPP development. There is a perception that the development process is currently at a standstill. With the GOA’s support, efforts could be made to help “jump-start” the process. A plan to do this should be developed. Crucial elements of the plan could be: - A reconfirmation of the country’s least-cost energy sector development plan and the place of the new NPP within that plan, as well as other technical questions such as appropriate size. This is a critical step in the process, as its outcome will mark a critical “go, no-go” decision point with respect to new NPP development and also life extension of the existing NPP. With increased regional trade and possible access to lower cost electricity, the issue to resolve at this point might well be how much the security of an expensive NPP will cost Armenia (compared to imports). And, even if imports turn out to be more expensive than any nuclear option, any decision to pursue nuclear might need to be weighed against the public’s appetite for nuclear power. - Appointment of a “transaction advisor”, first to identify and gauge the likely appetite of potential investors and then to actively set out to promote the NPP as a viable investment through, for example, travelling road shows or other means. This advisor would then be engaged right through to a possible financial close of the project, providing pertinent strategic, financial, legal and technical advice to the MOENR (or a GOA agency that might be established to develop the NPP). - The above would have to be coordinated with the current Russian partner, whose involvement thus far has been perceived as not very proactive. The plan should therefore include an element of how to either better engage this partner or perhaps even terminate the relationship. - Needless to say, the above strategy requires a strong “buy-in” from the GOA. As a prerequisite to any further USAID assistance, the GOA should not only agree to the assistance (which it must do in any case), but it should also be required to take actions as may be specified in the strategy to demonstrate its on-going commitment (e.g., staffing at least part way up to levels originally recommended by IAEA, taking action on the recommended surveys and meteorological monitoring, doing its part to promote the investment to potential investors, developing possible financing strategies). 2) Life extension of the existing plant. Subject to decisions made as the result of the least￾cost planning study, the existing NPP will probably need to remain in service considerably longer than its projected retirement date of September 2016. The GOA has recently issued a decree on lifetime extension of the currently functioning unit. If not already undertaken, studies may 36 need to be performed on possible further work regarding life extension. USAID may want to investigate possibilities for assistance in this area. ANRA in particular has expressed a desire for help in the development of licensing regulations for aging NPPs. Such support may be provided in cooperation with the NRC and IAEA. 3) Renewable energy. There is no sense proceeding much further in the development of renewable energy possibilities without the firm support of the GOA. Hopefully, the upcoming US-Armenia MOU will act as a catalyst to help promote renewable energy development by the private sector. For this, the GOA needs to develop a policy beyond general statements, one in which specific goals are set, along with enabling conditions (e.g., 100 MW wind power and 50 MW solar power by 2020, supported by appropriate feed-in tariffs that will provide adequate returns to investors). In this light, and, with strong GOA willingness, support could be provided to the MOENR in drafting an appropriate energy policy and/or law. This, in turn, would then lead to the development of regulations in support of the policy (and the more prescriptive the policy or law, the easier it will for PSRC to develop the regulations). Without such a working framework in place, renewable energy in Armenia will be a non-starter. Undertaking any other work related to renewable energy in the meantime might be a useless endeavour, as enough studies on renewable energy potential in Armenia have already been done. 4) Regional trade. More work is required to expand on the foothold established in regional trade with Georgia. Possibilities for further trade within the Black Sea region need to be investigated, processes need to be initiated and commercial agreements should be negotiated. Some thought may be required on how to explicitly define the assistance required in a SOW for this task, as the process in arriving at agreements has been, and will probably continue to be, quite fluid. In the ESRI project, the contractor’s role as a mediator and facilitator in the negotiations with Georgia was extremely valuable, but it was a role that was not foreseen in the original SOW. Although loosely-worded SOWs are generally not desirable, in this case, they might be appropriate, under a general umbrella of “support to increase regional trade”. 5) Other possible USAID interventions, as suggested by stakeholders: - Continuing general strategic advice to the MOENR - Support to PSRC in terms of helping to become more customer oriented, development of quality of supply and service standards, transparency and related capacity building - Help in developing EU power system standards for potential EU integration - Laboratory for testing meters at the SC - English language training for senior managers - New software for the SO - Coordination with IAEA and US Department of Energy in procuring new lab equipment at the universities - Funding to publish course text books at the universities - Funding for university graduates and lecturers to gain short-term (3 months) experience at US schools - Teleconferencing equipment for lectures - Training on western high voltage equipment 37 6) GOA commitment. Some of the above recommendations make reference to GOA commitment with respect to nuclear development and renewable energy. Since this seems to be a recurring theme, future agreements on USAID technical assistance might, in general, prescribe more specific obligations by the GOA so it may be better perceived as “pulling its own weight”, 7) Analytical work. Although helpful, some of the analytical work carried out under the ESRI project had questionable value – e.g., i) the relevance of examining renewable energy effects on system stability; ii) not including Iran in the simulations; and iii) the appropriateness of using GTMax when other least-development software could have produced more meaningful results. Perhaps more attention can be given during the project design stage on what is to be accomplished by the analytical work and what software will be used to accomplish this task. A review of the detailed tasks by an experienced power system planner, either at the project design stage or even on submission of the contractor’s first work plan, could help to optimize the work and ensure meaningful results. 8) Performance metrics. For the purpose of good monitoring and evaluation, objectives of the tasks and expected outcomes should be clearly stated. For task 3 (Regional Interconnection), there appears to have been a disconnect between the subtasks and the expected results, probably because the subtasks changed at some point between the time the SOW was first drafted and the date of the contractor’s agreement. It should be ensured that the subtasks and expected results remain consistent throughout. 38 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK ANNEX II: EVALUATION METHODS AND LIMITATIONS ANNEX III: DATA COLLECTION INSTRUMENTS ANNEX IV: SOURCES OF INFORMATION ANNEX V: SUMMARY NOTES FROM INTERVIEWS ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST 39 ANNEX I: EVALUATION STATEMENT OF WORK Statement of Work Evaluation of USAID/Armenia Health Systems Strengthening Project, Civil Society and Local Government Support Program, Alternative Resources in Media Project, Assistance to the Energy Sector to Support Energy Security and Regional Integration Program, Pension and Labor Market Reform Project Summary: USAID/Armenia requires performance evaluations of the following activities: Health Systems Strengthening Project (HS-STAR), Civil Society and Local Government Support Program (CSLGSP), Alternative Resources in Media Project (ARM), Assistance to the Energy Sector to Support Energy Security and Regional Integration Program (ESRI), and Pension and Labor Market Reform Project (PALM). The purposes of this Task Order are to evaluate the success of these projects in their relevant technical areas and to assess the overall effectiveness of the projects in achieving set programmatic goals and USAID/Armenia’s strategic objectives. Three of the five planned evaluations are designed as mid￾term performance evaluations (HS-STAR, CSLGSP, and ARM), while the remaining two are designed as end-of-project performance evaluations (ESRI and PALM). The purposes of the mid-term evaluations are to inform USAID’s determination on whether the set programmatic goals and targets are being achieved, and whether the initial designs of the projects are still valid in leading to the achievement of the original objectives. Findings from the mid-term evaluations must inform future work plans of the relevant projects, as well as designs of future similar activities. The purpose of the end-of-project evaluations is to assess the effectiveness of resources spent and to inform design and development of future strategies and projects. The evaluation findings must be used primarily by USAID/Armenia, the respective implementing partners, and by interested government entities where applicable. The respective project AORs/CORs will develop plans for incorporation of relevant recommendations from the evaluations in their future work plans. Contractor Responsibilities and Projects: The evaluations should measure and analyze the accomplishments or the progress toward achievement of the results of the activities, guided by the evaluation questions formulated for each individual activity. Each evaluation question must be answered empirically, relying on factual evidence, and must be addressed distinctly in the final reports. HS-STAR Jan 2011-Jan 2014, $9.6m This is a three-year activity in the second year of its implementation. The project aims to address key constraints in health financing, leadership and governance, human resources, and information systems that impede access to and delivery of quality health services. The project relies on an approach that simultaneously aims to strengthen the health system while improving the quality of care and increasing population knowledge in priority service areas, including maternal and child health, reproductive health and family planning, tuberculosis, non-communicable diseases and emergency medicine. HS-STAR aims to significantly enhance local capacity to design, implement and monitor reform interventions to foster sustainability. To this end, HS-STAR provides technical assistance to the Armenian government through program components of health financing and governance, quality improvement (including MCH, TB, RH/FP), and the adoption of proactive health care seeking behaviors through civil society engagement. 40 CSLGSP Sep 2010-Sep 2014, $15.5m This is a four-year activity in the second year of its implementation. The project aims to increase the level of informed and organized civic activism at the local and national levels, along with more participatory, decentralized, accountable governance that leads to a more democratic society. The project will reach this objective through three programmatic components: (1) local government and civil society collaboration, (2) fostering civic participation, advocacy and activism, and (3) facilitating decentralization and local fiscal autonomy. The project expands opportunities for the public to organize and advocate on behalf of their needs and concerns; it also increases local government accountability and capacity to be responsive to citizen interests. The program takes advantage of opportunities for reform at the local level by creating models of democratic governance through a highly integrated community development approach to local self-governance, fueled by increased civic participation from Civil Society Organizations (CSOs) and businesses. ARM July 2010-June 2014, $4m. ARM is a four-year activity in its second year of implementation. The project is aimed at (1) supporting the production and dissemination of alternative content though multiple high- and low-tech platforms; (2) building consumer demand for news and information at local and regional levels by promoting media literacy, critical analysis, and use of the latest and most useful web and mobile-based media applications and (3) encouraging timely, concerted public advocacy by NGOs, think tanks and citizens on specific key media-related issues. Alternative Resources in Media supports the diversification of media content, utilizing both high- and low-tech means to produce and distribute news and information. It assists citizens to transform themselves from passive users to active information producers in an otherwise controlled media environment. ESRI Sep 2009-Sep 2012, $5.9 million The project provides expert services to the Government of Armenia to ensure (1) that the new Nuclear Unit meets western safety and environmental standards; (2) that Armenia develops the human and institutional capacity to plan and successfully implement the project; and (3) that the contribution of the project to Armenia’s economic growth and development is maximized. The project’s objectives also include improving the enabling environment for the development of renewable energy in Armenia, and supporting Armenia’s integration into regional energy systems and markets. PALM Oct 2009-Sep 2012, $5.4 million PALM is USAID’s primary activity aimed to assist individuals, households, and communities to better manage social risks or needs, provide support to the vulnerable employed and jobless individuals (particularly the low-skilled, women, and the disabled), and give greater attention to countering the effects of the global economic crisis. The program is focused on (1) building institutional and human resource capacity to implement a multi-pillar pension system that will provide an adequate income for its elderly population in an environment that is fiscally sound and sustainable, with reformed current pay-as-you-go (PAYG) scheme and the newly created private retirement savings system; and (2) countering the effects of the economic downturn, through catalyzing new linkages at the local level between the private sector, local authorities, and vocational training institutions to address labor market gaps, and through improving the country’s workforce competitiveness. Evaluation Questions: 41 The evaluation questions for mid-term evaluations are designed to help the projects adjust and/or modify future implementation and work plans, as well as designs of future similar projects. The questions for end-of project evaluations will help assess the effectiveness of the resources spent and will help inform the design of future projects and strategies. The contractor must review and summarize the data collected from document review and from field work to answer the following evaluation questions: HS-STAR • To what extent is the project on track in achieving its expected results of a) establishing transparent and accountable health financing and governance; b) institutionalizing a system of continuous improvement of the quality of provided services; c) building the capacity of the national TB program; and d)enabling civil society to exercise their health rights and responsibilities? • Are the project implementation approaches relevant to the current state of health reforms in Armenia? • How did the implementer perform in terms of project management? o Were the project leadership and the management structure appropriate for its implementation? • What internal and external factors slow down organization of procurement and training processes? • Is there evidence that project interventions will be sustainable beyond the project lifetime? CSLGSP • How relevant was the Civil Society and Local Government Support project intervention to the current civil society and local governance situation in Armenia? • To what extent has the grants component contributed to the diversity of sustainable watchdog, advocacy and policy development mechanisms? • To what extent has the program been successful in promoting decentralization? • How effective is the project implementation? o Is the project management structure and staffing appropriate for the effective implementation of the project? o How successful has the project been in leveraging resources for Public-Private Partnerships in partner communities? o Are the project components on track for achieving results/benchmarks (Please use data from comparison and intervention communities to assess the achievements in citizen engagement in decision making and in community development projects)? • Do local governments/community councils, and citizens feel that the outcomes of the projects (greater citizen participation, new decision-making processes, etc.) serve their interests? ARM • How relevant has the ARM project intervention been to the media field challenges in Armenia? • To what extent has the project contributed to creating opportunities for (a) a diversity of alternative sources of information and opinion, (b) increased access to alternative sources of information and opinion, and (c) advocacy for improved enabling environment for media independence? 42 • What has the project done or mechanisms has it created to ensure that the project products will be sustained? • What have been the added value and/or drawbacks of the project management structure (consortium of three local organizations with an unaffiliated Chief of Party)? • How successful has the project been in leveraging resources of (a) the skills and organizational strengths of the consortium members, and (b) private and other donor partnerships? ESRI • Is there evidence that the independent expert services to Government of Armenia for new Nuclear Power Plant contributed to procedures and activities implemented in accordance with the western safety and environmental standards in preparation of documentation for the New Nuclear Power Plant? • Is there evidence that the scenarios of power system integration contributed to increased understanding of renewable energy development tendencies in the region among the main stakeholders? • Is there evidence that the regional integration task contributed to an increase of regional cooperation with Georgia? • What do stakeholders groups see as the results of the project? • How relevant was the intervention in terms of Armenia’s priorities in energy sector? PALM • What do stakeholder groups - the Ministry of Labor and Social Affairs, the Central Bank of Armenia, the Ministry of Finance of Armenia, State Revenue Service, State Employment Service Agency, “Pension System Awareness Center” foundation, “Youth Orientation Center” NGO – see as the achievements of the program? • To what extent has the project contributed to the development of significant pension reform legislation and to the identification and addressing of contradictory elements? • To what extent has the support given to the GOAM to build the foundation of Multi Pillar Pension System contributed to the preparedness of the System to be implemented? • To what extent did the project contribute to the advancement of the ability of the State Employment Service Agency to match labor market demand and to develop/implement active labor market policies? • How adequate was the project PMP in reporting on project progress? USAID’ S Role in the Evaluation The USAID Mission in Armenia will: • provide relevant programmatic and budgetary information to the Contractor; • provide project documents and evaluations to the Contractor; • facilitate obtaining USAID/Mission input; and • arrange USAID/Armenia meetings. In some instances (although the Contractor should not depend on this), an additional USAID staff person may join the Contractor during the field visits/stakeholder interviews in Armenia. USAID Mission staff 43 and/or the USAID team members will be available to assist the Contractor in providing in-depth knowledge of the various projects and activities that are being evaluated. Methodology The Contractor must: 1. Conduct a comprehensive review of performance reports and other materials and identify data gaps. 2. Identify data collection methodology to provide the best possible evidence to answer the evaluation questions, also considering feasibility issues. 3. Identify informants and stakeholders, samples and/or other relevant data sources. 4. Prepare a field work plan. 5. Conduct field research in Armenia. 6. Analyze data and compile key findings, conclusions and recommendations. 7. Revise the draft reports addressing comments by USAID and submit final reports to USAID/Armenia for acceptance. 8. Address implementing partner comments within one week as necessary after USAID/Armenia shares the final reports with implementing partners, and if partners choose to submit “Statements of Differences”. The proposed methodology should address the need for data collection from qualitative and quantitative sources, and provide the best possible combination of methods, given the evaluation questions and the available resources and timeline. All evaluation questions need to be answered empirically; therefore the data collection methods should be tailored to ensure that adequate evidence is collected to answer each of the questions in a definitive manner. There is no preference for any particular method. The ability of particular method(s) to properly answer the evaluation questions is important. Data should come from facts, rather than be based on anecdotal evidence. Conclusions should be based on findings received from multiple sources, and strengths and limitations of the methodology should be explicitly communicated. All people-level data should be disaggregated by sex to allow analysis of findings by sex. Baseline data for all projects is available from their monitoring data. A sample of indicators used for monitoring of each of the projects is provided in the Annex. Some of the baseline data sources include surveys, official statistics, automated information systems, and project records. In the case of CSLGSP baseline data was collected not only from participating 43 communities, but also from 15 control communities. Although not randomly assigned to treatment and control groups, communities nonetheless are comparable, and can therefore serve as comparisons for the achievements in citizen engagement in decision making and in community development projects. Deliverables The Contractor’s deliverables must include: 1. Written methodology plans (evaluation designs and work plans). The evaluation designs must include detailed evaluation design matrices (including the key questions, the methods and data sources used to address each question), draft questionnaires and other data collection instruments, and known limitations to the evaluation design. The final designs require COR approval. The work plans must include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation teams. The methodologies will be shared with relevant stakeholders, including the implementing partners for their comments before finalizing. 44 2. Verbal debriefing The evaluation teams must meet with USAID/Armenia upon arrival. The teams must also provide oral briefings of findings and recommendations to the USAID/Armenia senior management and relevant technical teams prior to departure. 3. Draft Evaluation Reports Prepare draft evaluation reports which must analyze data and summarize key findings, conclusions and recommendations. The Findings section must list all facts and evidence received from desk research and from field work in relation to each evaluation question. The Conclusions section must analyze the facts specified in the Findings section, and must discuss what worked, what did not work and why. The Recommendations section must address issues of what can be improved for future programming, based on Findings and the analysis provided in the Conclusions section. More specifically the Recommendations section must address the following: For HS-STAR – a) what can be changed for future implementation of the project, b) what recommendations can be made to ensure sustainability of project results; for CSLGSP - what can be improved for the remainder of the project given the resources available; for ARM a) what changes should be made in the approaches and directions of the current program, and b) what recommendations can be made for the design of future media programs; for ESRI - what recommendations can be made for future energy sector programming; for PALM - what recommendations can be made for design of future pension reform projects. The Evaluation Reports must at a minimum contain: 1) a 3-5 page Executive Summary summarizing key points (purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations); 2) a brief description of the project; 3) a section on the purpose and the methodology of the evaluation; 4) a section on clearly defined findings, conclusions and action-oriented recommendations. This section should be organized around the evaluation questions defined for each project. 5) Annexes, including the Scope of Work; all evaluation tools; all sources of information properly identified and listed; any “statements of differences” regarding significant unresolved differences of opinion by funders, implementers and/or members of the evaluation team; disclosure of conflicts of interest forms for all evaluation team members, either attesting to a lack of conflict of interest or describing existing conflict of interest. Draft reports must be submitted to USAID/Armenia within three weeks after completing the fieldwork for each evaluation. USAID will be responsible for compiling Mission comments for inclusion and submission to the Contractor. USAID/Armenia will provide the Contractor with a summary of such written comments within three weeks of having received the draft reports. 4. Final Reports The Contractor must submit final reports to USAID/Armenia within two weeks after USAID's comments are provided. The reports shall follow USAID branding procedures. The reports must include an executive summary and not exceed 30 pages (excluding appendices). 45 The final reports must meet the following quality standards (Please see the USAID Evaluation Policy): a) The reports must represent a thoughtful, well-researched and well￾organized effort to objectively evaluate what worked in the project, what did not and why; b) The reports must address all evaluation questions included in the scope of work; c) The report shall include the scope of work as an annex; d) Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report; e) Evaluation findings must assess outcomes and impact on males and females; f) Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology; g) Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, or the compilation of opinions; h) Sources of information need to be properly identified and listed in an annex; i) Recommendations need to be supported by a specific set of findings; j) Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. 5. Evaluation Data The Contractor must submit Evaluation data to USAID/Armenia along with the final reports for record keeping and future use by the Mission. The data must be in an easily accessible format, such as MS Word documents for qualitative data, and SPSS or Excel files for quantitative data. Schedule: The anticipated duration of all evaluations is not to exceed nine months. Staffing and Level of Effort: It is expected that the Contractor will provide five different teams of consultants to carry out the evaluation of the five activities described above. Consultants with diverse or cross-cutting skills may serve on more than one evaluation team. The mix of skills necessary for each evaluation team is specified below. Each evaluation team should consist of two experts, although other team compositions might also be considered. Inclusion of qualified local Armenian experts in the evaluation teams is highly encouraged. All evaluation team members must be required to provide a written disclosure of conflict of interest. Required Qualifications of the Personal For all evaluation teams: • Excellent English writing skills • Experience with evaluation designs, quantitative and qualitative methods and data analysis • Substantial experience with program evaluations. Prior experience with USAID or other donor project evaluations is a plus. For HS-STAR: • Substantial experience in health systems management, public health, international health or a related field • Substantial experience with program management and/or organizational development • Experience working with health projects, health systems project evaluations, or with program evaluations in the region is a plus 46 For ARM: • Substantial experience in communications, media, journalism or related field • Experience working with media projects, media project evaluations, or with program evaluations in the region is a plus For CSLGSP: • Substantial experience in public administration, international relations or related field with emphasis on civil society and local government • Experience working with civil society and/or local government projects, experience with civil society and local government project evaluations, or with program evaluations in the region is a plus For PALM: • Substantial experience in public administration, international relations or related field with emphasis on social services and pension reform • Experience working with social services or pension reform projects, experience with social services project evaluations, or with program evaluations in the region is a plus For ESRI: • Substantial experience in engineering with emphasis on energy systems, nuclear energy or renewable energy • Experience working with energy system projects, experience with energy project evaluations, or with program evaluations in the region is a plus Instructions on Preparation of Branding Implementation Plan and Marking Plan As part of the proposal submission, the Contractor will develop a Branding Implementation Plan (BIP) and a Marking Plan in accordance with the policies found at Automated Directive System (ADS) Chapter 320, revised on May 5, 2009, or any successor branding policy, and with the “USAID Graphics Standard Manual” that is available at www.usaid.gov/branding. Among other provisions, ADS 320 states that: 1. Contractors and subcontractors' corporate identities or logos must not be used on USAID￾funded program materials. 2. Marking is not required on contractor vehicles, offices, office supplies or other commodities used solely for administration of the USAID-funded program. 3. Marking is not permitted on any communications that are strictly administrative, rather than programmatic, in nature. USAID identity is also prohibited on contractor and recipient communications related to award administration, such as hiring/firing of staff or renting office space and/or equipment. The Contractor shall also develop a Marking Plan for public communications, commodities, program materials, deliverables, and other items that visibly bear or will be marked with the USAID identity. The marking plan may include requests for exceptions to marking requirements, to be approved by the Contracting Officer. Contract deliverables to be marked with the USAID identity must follow design guidance for color, type, and layout in the Graphic Standards Manual (available at www.usaid.gov/branding) or any successor branding policy. 47 With respect to this Task Order, the Contractor should develop a BIP and Marking Plan bearing in mind the following branding strategy: 1. Program Name: Evaluation of Five USAID/Armenia Projects. 2. Positioning: This task order is funded through the USAID/Armenia Mission. Materials and communications must be positioned as from the American People, using the USAID Identity. 3. Outreach to Beneficiaries and Host-Country Citizens: No special outreach efforts to beneficiaries and host-country citizens are planned under this Task Order. 4. Level of Visibility: The findings of the final evaluation report will be used by USAID in its implementation and further planning its activities. The report will be submitted to USAID’s Development Experience Clearinghouse for wider access. 5. Other Organizations to be Acknowledged: No other organizations are required to be acknowledged. 6. Specific branding issues: The only branding issue expected to arise as a result of implementing this Task Order is the proper use of graphics standards for all reports and other printed or electronically distributed information. Environmental Considerations The Evaluation Services IQC has an Initial Environmental Examination (IEE) approved on April 8, 2010. The IEE has determined that the activities described under the Evaluation Services IQCs qualify for a categorical exclusion per 22 CFR 216. Any activities found to be outside the scope of the approved Regulation 216 environmental documentation shall be halted until an amendment to the documentation is submitted and written approval is received from USAID. 48 ANNEX II: EVALUATION METHODS AND LIMITATIONS Technical approach In its original proposal, SI proposed that the performance evaluation be undertaken as a variant of a multi-level Mixed Methods approach. Mixed Methods offer a high level of rigor and empirical validity through multiple, reinforcing data collection activities that together synergistically enhance the reliability and validity of evaluation analysis and conclusions that can be drawn. Mixed Methods designs can include a variety of quantitative evaluation data collection methods, including, where appropriate sources of data can be identified and obtained, QED comparison groups, representative sampling, non-parametric mini￾surveys, or quantitative time series analysis based upon secondary statistical data sets, as well as intensive qualitative data collection techniques such as case studies, structured key informant interviews, focus groups, social network mapping, structured site observations, and qualitative review of documents. A major feature of mixed method approaches is that multiple forms of data obtained from multiple sources or multiple levels within social systems and hierarchies can be used to triangulate and verify the accuracy of data collected elsewhere by other means ensuring a rigorous empirical basis for all findings and the conclusions drawn from these. Additionally, evaluators can be alert to unanticipated data collection and data mining opportunities that may emerge during the planning and/or implementation stage, and thus can continuously refine their hypotheses and seek additional confirmation that their understanding of project implementation or success is accurate and based upon sound data. When confronted with limited time, resources and data, extensive use of mixed method designs employing a variety of data collection methodologies should generally be used. SI’s initial analysis of the USAID/Armenia TO suggested that this general design was the appropriate approach for the ESRI evaluation, with minor design and methodological variation required as the project structure and research questions required. After contract award and further analysis, it was determined that the approach would comprise a mix of document review and stakeholder interviews. Documentation was a major product of the assistance and the direct beneficiary stakeholders would know the most about this documentation and its relative usefulness and relevance. Other stakeholders having knowledge of the assistance who were not direct beneficiaries would also be sought out. These outputs would then be compared with original expectations of project, as outlined in the contractor’s original SOW. Document review It was intended that the evaluations would begin with a thorough and comprehensive desk review of the documents associated with the project. This would include a review and analysis of project management plans (PMPs), annual and quarterly progress reports, project design documents and contracts or cooperative agreements, and any other relevant external documents. Accordingly, the USAID Armenia Mission office provided SI with a series of annual and quarterly reports and work plans a few weeks before the evaluators began their field work. These were thoroughly reviewed. Also, all USAID issued technical reports produced by the ESRI project, as well as additional material in the way of notes, illustrations, etc., were available on the ESRI project website at http://www.armesri.am/Public_Docs/Reports.html. Finally, various background material on the Armenian energy sector was obtained through internet search. 49 Stakeholder interviews The project produced many reports and, indeed, many of the project’s results were the reports themselves (as opposed to a reporting of the results), which included technical information that could be used as a basis for further work. The usefulness of these reports, together with other results of the project, could be judged after face-to-face interviews with the beneficiaries of the assistance. This method was selected because direct beneficiaries were relatively few and easily identifiable. For the interviews, questionnaires were prepared in advance and varied to some extent depending on the stakeholder. These were generally used as a guide in approaching the interviews and were never read verbatim to the interviewee. The questionnaires are provided in Annex III. A full list of documents reviewed and interviewees is provided in Annex IV. Interviewees included Mission staff, the Chief of Party for the implementing contractor, a local subcontractor to the US contractor, heads of various organizations that were direct beneficiaries of the assistance and, the only other international lending agency heavily involved in the Armenian energy sector, KfW. The ESRI project had many beneficiary organizations, each of which benefited from different components of the project. The particular people interviewed were suggested by either Mission staff or the USAID contractor (or both). Presumably, these suggestions were based on their perceptions of who would have the best knowledge of the assistance provided within each organization. Key beneficiaries interviewed included: • Deputy Minister of Energy, MOENR • Head, Atomic Energy Department,, MOENR • General Director, CJSC Settlement Center, MOENR • General Director, Electro Power Systems Operator, MOENR • General Director, High Voltage Electric Networks CJSC, MOENR • General Director, Scientific Research Institute of Energy CJSC • Head and Chair of Heat and Power Engineering, State Engineering University of Armenia • Dean of Faculty of Physics, Yerevan State University • Chairman, Armenian Nuclear Regulatory Authority • Head, Licensed Activities Monitoring and Investment Program Department, Public Services Regulatory Commission (the Commissioner was not available) As a significant amount of the evidence collected was from interviews, it is appropriate to discuss recall and selection biases. Recall bias is quite minimal as the project was actually still being carried out at the time of the interviews, although in its very final stages. Also, all interviewees were very much involved in the ESRI tasks, so even if the recall of events going back to the beginning of the project (about three years) was an issue, the recall capability was high. Selection bias, however, was a somewhat bigger issue, as all interviewed beneficiaries were direct participants in the project and were recommended by either USAID or the contractor. All of them, as expected, said great things about the project. There was a problem finding people who could be simply classified as observers, mainly because the nature of the product (e.g., nuclear safety regulations) was such that it is hard to find a neutral, detached party who would have enough interest in the subject matter to have formed a strong opinion. The most informed expert on the assistance provided was always the beneficiary, by far. The only real unbiased informed interviewee was the KfW representative in Armenia, who found the USAID assistance valuable and complementary to that provided by KfW (as opposed to in competition with). 50 Selection bias in this case was mitigated partly by the tangible nature of the results themselves - e.g., the fact that breakthrough agreements were made with Georgia, the actual introduction of more appropriate nuclear engineering/ physics curricula at local universities, the existence of a particular product-document (which, admittedly, would then have been judged by the beneficiary as useful or not, but still, it is a tangible result). Another strong mitigative factor is the technical knowledge base of the evaluation team, which enabled it to make credible professional judgments on the work carried out. For example, on one issue, a question arose on the suitability of some of the software used to undertake certain technical analyses. None of the beneficiaries questioned this; only the evaluation team did. A final check on potential selection bias is the comparison of the very real, tangible results with original expectations. Although the interviews may provide certain evidence that the project generated certain results, the question then arises of how significant the results actually were; i.e., the results, though evident, may, perhaps, have fallen far short of expectations. Therefore, it was considered useful to compare achievements with original expectations, as may have been stated in the original project design. Of course, changing circumstances over the course of the project and other factors certainly did influence many of the project outcomes – both positively and negatively. Therefore, to the extent possible, the analysis also offered explanations on how and why particular outcomes occurred. A final shortcoming associated with any interview is that the interviewee may not always be completely truthful in answering a question. He/ she might have other motives for answering in a particular way. For example, an interviewee might find great value in particular component of the assistance (e.g., new equipment), while not finding anything useful in some of the other assistance provided. In such a case, he/ she might feel compelled to not say anything detrimental to not jeopardize the chances of receiving similar help in the future. The evaluators found that this existed to some extent in the case of ESRI project. For example, although one glaring weakness of the simulated integration scenarios was that they did not include Iran (Armenia’s largest electricity trading partner at present), only one of the four involved beneficiaries mentioned this. Such a shortcoming was mitigated in two ways, basically through triangulation. First, to the extent possible, revealing statements made by any particular interviewee were cross-checked and verified through what was said in other interviews. Second, technical aspects of the assistance provided were reviewed to some extent (and at a relatively high level) and some limited professional judgments were made by the evaluators on the merits of the work itself. In the case of the Iran-less simulation scenarios, the shortcoming was mentioned by the contractor (and duly noted by the evaluation team), which was then reinforced by review of the relevant technical reports. Summary of approach The approach to the evaluation, as with all evaluation work performed by SI, was intensely “evidence based.” Through a rigorous application of the best methods suitable, given time and resources available, the highest quality of objective and subjective evidence available was collected. Every effort was made to review quantitative data to the fullest possible extent, i.e., in this case, the technical reports produced by the ESRI project. This was followed up through stakeholder interviews. Qualitative evidence gathered through interviews was the verified through other sources – e.g., other interviews, the documents. Then, the project’s results, as ascertained from all these sources, was compared with original expectations. Thus, the evaluation team used triangulation across various sources, levels, methods, and data sets to ensure both qualitative and quantitative information was consistent and, more importantly, verified the findings. 51 ANNEX III: DATA COLLECTION INSTRUMENTS Below are illustrative interview questions for: i) the beneficiaries of the assistance - in this case, relevant officials of the concerned Government of Armenia ministries or departments, ii) other stakeholders (e.g., Tetra Tech, KfW) and iii) USAID itself. Interview questions for GOA beneficiaries: 1. What is the nature of your organization? What functions does your organization undertake? By what mandate? 2. What is your relationship with USAID? 3. What has been the nature of the assistance being provided? 4. What have been the results to date? 5. Have the results been useful? 6. How satisfied are you with the experience with the USAID assistance? 7. Had you not been able to obtain this assistance, what other alternatives would have been available to you? 8. How effective do you view the USAID contractor’s work of implementing the USAID assistance? Can you provide comments about the experience, qualifications, and effectiveness of the contractor team? 9. What evidence can you cite that the services provided in connection with the new Nuclear Power Plant contributed to its continuing development? 10. What evidence can you cite that the scenarios of power system integration contributed to increased understanding of renewable energy development tendencies in the region among the main stakeholders? 11. What evidence can you cite that the regional integration task contributed to an increase of regional cooperation with Georgia? 12. Can you provide the evaluation team with information to help us understand the impact of the assistance on your activities? 13. How sustainable are the impacts of the assistance? Do you anticipate that your organization will continue with the same practices after the USAID assistance has finished? 14. Do you see the assistance as being relevant in light of the current regulatory and legal environment in Armenia? 15. What, in your opinion, is the main constraint acting on USAID assistance in Armenia? 16. What, if any, kind of training have you received? What kind of training? Should this training be augmented? How? 17. In your view, how effective is the USAID assistance? Do you believe that it has achieved what it set out to do? 18. In your opinion, how could the assistance be improved? 19. What other types of assistance could be offered by USAID to the energy sector in Armenia? 52 20. Do you have a specific recommendation that the evaluation team could provide to USAID to help it improve future assistance? In light of your experience, what advice would you give USAID? Interview questions for other stakeholders (e.g., Tetra Tech): 1. Can you please summarize your mandate for assistance to the Armenian energy sector? 2. Are you involved in energy sector assistance projects in Armenia other than those with USAID? What are they? 3. What are the respective roles of you, USAID/Armenia and USAID/Washington with regard to your involvement in the Armenian energy sector? 4. In your view, have the results of USAID assistance been useful? 5. In your opinion, how appropriate is the USAID assistance? What are the strongest points of USAID energy sector assistance? Are any improvements needed? 6. In your opinion, how successful is USAID energy sector assistance? How well has it met GOA objectives? 7. What evidence can you cite that the services provided in connection with the new Nuclear Power Plant contributed to its continuing development? 8. What evidence can you cite that the scenarios of power system integration contributed to increased understanding of renewable energy development tendencies in the region among the main stakeholders? 9. What evidence can you cite that the regional integration task contributed to an increase of regional cooperation with Georgia? 10. How effective do you consider the work to be of implementing the assistance provided? 11. How sustainable are the improvements brought about by the assistance? Will the changes continue once the assistance has ended? 12. Are there any areas of the Armenian energy sector assistance in which provision of assistance might be more effective? 13. What, in your opinion, is the main constraint acting on USAID assistance in Armenia? 14. Do you see the particular USAID assistance highly appropriate and relevant, in light of the current regulatory and legal environment in Armenia? Do you believe it fits well within the desired development strategy for Armenia? 15. Are the objectives of the assistance being accomplished? Please elaborate. 16. Should any new, follow-on assistance be designed any differently than the present one? Please explain in detail. 17. Does your component of the project(s) have a human capital development component or an institutional strengthening component? If not, in your opinion, should there be? 18. Can you recommend any individuals, groups, or organizations in Armenia the evaluation team should be sure to contact? 19. On a scale of 1 – 10 (10 highest), how would you rate USAID assistance to the Armenian energy sector? 53 20. Do you have a specific recommendation that the evaluation team could provide to USAID to help develop similar assistance in the future? In light of your experience, what advice would you give USAID? Interview questions for USAID: 1. How does USAID/Armenia anticipate using this assessment? What are the special areas of concern? What should the assessment be sure to cover? 2. Can you briefly summarize USAID’s past involvement in Armenia’s economic development and its specific plans for the future? 3. What is a rough estimate of the amount of annual funding that might possibly be available for USAID’s future energy sector development programs in Armenia? 4. Are there policy or other issues that presently cause disagreement between USAID and GOA? 5. What are the respective roles and responsibilities of USAID/Armenia, USAID/ Washington, and the Armenia implementing agencies in terms of project implementation? What were the respective roles in the design of each of the projects? 6. Can the evaluation team obtain a timeline of the cost summary of the project-to-date? 7. How do the individual components fit with USAID’s development strategy for Armenia? 8. Can you please provide the evaluation team with the performance monitoring plan for each of the project’s components with the targets for these indicators, and the latest reports on how well the targets have been achieved? 9. Is it possible to obtain the original SOW for Tetra Tech, as well as a timeline of modifications to the work over the life of the project? 10. Can you please provide the evaluation team with copies of the technical reports produced by the project? 11. In your view, how sustainable are the changes brought about by the projects? 12. How effective are the projects’ contractors? How effective are the targeted recipients of the assistance? 13. How important is energy sector assistance to USAID’s overall portfolio in Armenia? How important is the project to USAID support to the electricity sub-sector? 14. What, in your opinion, are the main constraints in effectively carrying out the assistance? 15. Has USAID considered other assistance alternatives to the energy sector? If so, what alternatives were considered? 16. In your opinion, does the project have a sufficient human capital development component or an institutional strengthening component? If not, should these support activities be augmented? 17. Who are the key organizations or people in Armenia that USAID feels that the evaluation team should meet while conducting the evaluation? 54 ANNEX IV: SOURCES OF INFORMATION Documents It should be noted that a few of the technical documents listed below – i.e., most of those dated September 2012 or later – were available only after submission of the Draft Report to USAID. Although these reports were reviewed, this review is not reflected in the main body of the report, which represents an interpretation of the information available as of the beginning of October 2012 and which was subject to extensive review by both Social Impact and USAID. For this reason, it has been decided not to make any changes in the report that might arise in light of the new information, especially since such changes would have had no material effect on the findings, conclusions and recommendations – and, in any event, would have probably reinforced them. Status reports: ESRI First Year Work Plan, December 2009 ESRI Second Year Work Plan, November 2010 ESRI Third Year Work Plan, November 2011 Joint Work Plan of the RoA Ministry of Energy and Natural Resources and PA Government Services for the US Agency for International Development Armenia Assistance to the Energy Sector to Strengthen Energy Security and Regional Integration (ESRI) FY 2010 – Task 1, Nuclear Plant Development Quarterly Report, October - December 2009 Quarterly Report, January – March 2010 Quarterly Report, April – June 2010 Quarterly Report, July – September 2010 Quarterly Report, October – December 2011 Quarterly Report, January – March 2012 Quarterly Report, April – June 2012 Technical reports and documents – New Nuclear Plant Development (Task 1): Various documents related to public hearings, including: notifications, presentations, handouts, lists of participants, transcripts, press releases, summary comments and recommendations, pictures Nuclear Curricula Development at Armenian Universities to Support New Nuclear Plant Development, December 2011 Nuclear and Radiation Safety Center Comments on Draft Safety Analysis Report Guide and ESRI Project Responses, June 2012 Safety Analysis Report Format and Content Guidance Based on IAEA and Nuclear Regulatory Commission Documents, July 2012 Curricula Development at State Engineering University of Armenia for Automation Specialists, August 2012 Curricula Development for New Specialty “Automation of Technological Processes and Production (in the Nuclear Energy Industry)” (Code 220300) at State Engineering University of Armenia (presentation) 55 Improvement of Analytic Capabilities for Nuclear Safety Regulation in Armenia with ANSYS-CFX, November 2012 Safety Analysis Report Format and Content Guidance Including Normative Documents of the Russian Federation, November 2012 New Nuclear Generation, November 2012 Technical reports and documents - Renewable Energy Development Strategy and Power System Integration (Task 2): Small Hydro Power (SHPP) Sector Framework, Status, Development Barriers and Future Development, July 2010 Small Hydro Power (SHPP) Sector Framework, Status, Development Barriers and Future Development 2012 Update, March 2012 Wind Energy Development in Armenia: Legal, Regulatory, Tax and Customs Regulations, April 2010 Wind Energy in Armenia: Overview of Potential and Development Perspectives, March 2010 Connection of Photovoltaic Power Plants (with up to 5 MW of Capacity) to the Common Grid of Electrical Power System, General Provisions, July 2010 Developing an Improved Wind Power Purchase Agreement for Armenia, July 2011 Contract for the Sale and Purchase of Electricity Generated by Wind Power, August 2011 Overview on Solar Electric Power in Buildings with Applications in Armenia, July 2011 Integrating Solar Energy into the High Voltage Network, Study Tour Report, July 11-17, 2011 Connection of Wind Turbines to the Power System’s Grid, September 2011 Technical reports and documents - Regional Electricity System Synchronization and Trade and Market Development (Task 3): Armenia Energy Security and Regional Integration Summary of Year 1 Technical Results, December 2010 GTMax and PSS/E Conversion Methodology, April 2011 Methodology for Evaluating the Economics, Financial Viability, and Environmental Consequences of Proposed Georgian Interconnection and Transmission Line Options, April 2011 Congestion Management in European Grid and Applicability of Methodologies for Armenia, August 2011 Armenia Energy Security and Regional Integration Summary of Year One and Two Technical Results, May 2012 Electricity Transit Principles, September 2012 GTMax Simulation of the Armenian Power System and Isolated Regime Calculations with Assessment of Impacts of Renewables, December 2011 Analysis of Opportunities for the Armenian Power System’s Regional Integration with Georgia, Turkey and Russia, October 2012 Armenia Power Market Harmonization Issues and Recommendations, August 2012 Armenia Electricity Demand Forecast, November 2012 Report on Armenia Markal-to-Times Model Migration and Training, November 2012 56 Overview of the Natural Gas Sector of the Southern Caucasus and Neighboring Countries, November 2012 Customs Issues and Regulations for Cross-Border Trade of Electricity, November 2012 Armenia Energy Security and Regional Integration Summary of Year Three Technical Results, November 2012 Transmission Tariff and Investment Analysis, November 2012 Technical reports common to Tasks 2 and 3: Technical Report – Phase I. System Stability Analysis (Scenario N3 with Renewables Impact Assessment), December 2010 Technical Report – Phase II. System Stability Analysis (Scenarios N1 and N2 with Renewables Impact Assessment), November 2011 Economic Efficiency of the Armenian Power System Integration and Analysis of Impacts of New Renewable Development in Armenia, November 2012 People interviewed Mariam Gevorgyan, Monitoring and Evaluation Specialist, USAID Armenia Dr. Marina Vardanyan, Energy and Water Adviser, Environmental Officer, Office of Economic Growth, USAID Armenia Conan E. Peisen, Director, Office of Economic Growth, USAID Armenia Sally Patton, Acting Program Officer, USAID Armenia Armen Arzumanyan, Chief of Party, ESRI Project, Tetra Tech Abgar Budaghyan, Head, Licensed Activities Monitoring and Investment Program Department, Public Services Regulatory Commission Garegin Baghramyan, Head, Tariff Department, Public Services Regulatory Commission Areg Galstyan, Deputy Minister, Ministry of Energy and Natural Resources Hrachya Tsughunyan, Head, Department of Development, Ministry of Energy and Natural Resources Tigran Melkonyan, Head, Department of International Relations, Ministry of Energy and Natural Resources Aram Gevorgyan, Head, Atomic Energy Department, Ministry of Energy and Natural Resources Torgom Madoyan, General Director, CJSC Settlement Center, Ministry of Energy and Natural Resources Dr. Levon Aghekyan, General Director, Scientific Research Institute of Energy CJSC Mnatsakan Mnatsakanyan, General Director, Electro Power Systems Operator, Ministry of Energy and Natural Resources Levon Vardanyan, Country Expert, INOGATE Technical Secretariat and Adviser to the Electro Power Systems Operator Vostanik Marukhyan, Head and Chair of Heat and Power Engineering, State Engineering University of Armenia 57 Hambardzum Hovsepyan, Dean of Faculty of Power Engineering, State Engineering University of Armenia Sahak Abramyan, General Director, High Voltage Electric Networks CJSC, Ministry of Energy and Natural Resources Simon Igitbashyan, Chief Engineer, High Voltage Electric Networks CJSC, Ministry of Energy and Natural Resources Roland Avagyan, Dean of Faculty of Physics, Yerevan State University Ashot Martirosyan, Chairman, Armenian Nuclear Regulatory Authority Armen Amirjanyan, Director, Nuclear and Radiation Safety Center, Armenian Nuclear Regulatory Authority Anna Melkonyan, Head, Department of International Relations, Armenian Nuclear Regulatory Authority Bella Andriassyan, Senior Project Coordinator, KfW Yerevan 58 ANNEX V: DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Arvid Kruze Evaluation Position Lead Evaluation Specialist Organization Social Impact Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Energy Security and Regional Integration (ESRI) Project I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 12.12.2012 59 Name Artashes Sargsyan Evaluation Position Evaluation Specialist Organization Social Impact Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Energy Security and Regional Integration (ESRI) Project I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 12.12.2012 60 USAID ARMENIA 1 AMERICAN AVENUE YEREVAN 0082, ARMENIA TELEPHONE: +37410 46 47 00 FAX: +37410 46 47 28 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 61