This report was produced for review by the United States Agency for International Development. It was prepared by Integra Government Services International LLC. The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development (USAID) or the United States Government. INDONESIA CLEAN ENERGY DEVELOPMENT PROJECT (ICED) II MID TERM EVALUATION FINAL EVALUATION REPORT 1 INDONESIA CLEAN ENERGY DEVELOPMENT PROJECT (ICED) II MID TERM EVALUATION FINAL EVALUATION REPORT Contract Title: ICED II Mid Term Performance Evaluation Prime Contract No.: AID-497-C-16-00006 Subcontract No.: 610900.01.17.160SUB Prime Contractor: Management Systems International, Inc. Subcontractor: Integra Government Services International LLC 1100 Vermont Avenue NW Suite 750 Washington, DC 20005 USAID Office: USAID/Indonesia COR: Amita Kulkarni 2 ABSTRACT The purpose of the Mid Term Evaluation of the United States Agency for International Development (USAID) Indonesia Clean Energy Development (ICED) II Project, was to provide the Mission with information and an analysis of which activities have been most effective, as well as how a reallocation of resources moving forward might benefit the future of the project. The MTE evaluated ICED II’s performance from May 2015 through December 2017, and was conducted by Integra Government Services under subcontract to Management Systems International. The MTE sought to answer the following two main questions: 1) To what extent is ICED II on track to meet its expected high-level and technical component-level (“Minimum”) results, and in what manner could either implementation or monitoring efforts be enhanced? 2) To what extent can evidence be collected that demonstrates the program’s contribution to either domestic US job creation or expanded market opportunities for US firms? Additional sub-questions are included and discussed in the body of the report. The Evaluation Team conducted a total of 36 key informant interviews and four site visits, and received nine responses to a supplemental online survey. Interviews were conducted in East Java, Jakarta, North Sumatra and South Sulawesi. A desk review was also performed which included a review of ICED II monitoring and evaluation plans and data, work plans, as well as quarterly and annual reports, among other documentation. Conclusions from the MTE state that ICED II has already exceeded or is on track to meet the majority of its high-level targets and performance indicators, with the exception of one indicator for which the information was not available to determine or validate the level of ICED II’s direct contribution (GHG emissions reduced). The MTE also found that ICED II supported activities are appropriately tracked, and are making significant contributions to support the Government of Indonesia in improving its renewable energy sector. 3 CONTENTS ACRONYMS I EXECUTIVE SUMMARY 1 INTRODUCTION 1 BACKGROUND 1 EVALUATION PURPOSE 2 QUESTIONS, METHODOLOGY AND LIMITATIONS 2 PROJECT BACKGROUND 8 LOCAL CONTEXT 8 OVERVIEW OF THE ICED II PROJECT 8 DESCRIPTION OF ACTIVITIES 9 EVALUATION PURPOSE AND QUESTIONS 12 EVALUATION PURPOSE 12 EVALUATION QUESTIONS 12 EVALUATION METHODS AND LIMITATIONS 13 DATA COLLECTION METHODS 13 DATA ANALYSIS METHODS 15 STRENGTHS AND LIMITATIONS 15 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 16 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS FOR QUESTION 1 16 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS FOR QUESTION 1A 23 FINDINGS AND CONCLUSIONS FOR QUESTION 1B 25 FINDINGS AND CONCLUSIONS FOR QUESTION 1C 26 FINDINGS AND CONCLUSIONS FOR QUESTION 1D 26 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS FOR QUESTION 1E 28 FINDING AND CONCLUSIONS FOR QUESTION 1F 31 FINDINGS AND CONCLUSIONS FOR QUESTION 1G 32 FINDINGS AND CONCLUSIONS FOR QUESTION 2 34 ANNEX 1 EVALUATION STATEMENT OF WORK 36 ANNEX 2 REFERENCES 50 ANNEX 3 KII PROTOCOLS 51 STAKEHOLDER GROUP: USAID/INDONESIA 51 STAKEHOLDER GROUP: ICED II COP, DCOP AND COMPONENT LEADS 53 STAKEHOLDER GROUP: ICED II M&E MANAGER 55 STAKEHOLDER GROUP: ICED II NATIONAL AND PROVINCIAL STAKEHOLDERS 57 STAKEHOLDER GROUP: ICED II FINANCIAL STAKEHOLDERS 59 STAKEHOLDER GROUP: ICED II PROJECT STAKEHOLDERS 64 ANNEX 4 ONLINE SURVEY 69 ANNEX 5 EVALUATION DESIGN MATRIX 73 4 ANNEX 6 LIST OF ENTITIES INTERVIEWED 1 ANNEX 7 DISCLOSURE OF CONFLICT OF INTEREST FORMS 3 I ACRONYMS ADB Asian Development Bank AFD French Development Agency ASEAN Association of Southeast Asian Nations ASHRAE American Society of Heating, Refrigerating, and Air Conditioning Engineers BAPPEDA Provincial Development Planning Agency Bappenas National Development Planning Agency BI Bank Indonesia BKF Indonesia’s Fiscal Policy Agency (under Ministry of Finance) BPSDM Human Resources Development Agency BRI Bank Rakyat Indonesia CDCS Country Development Cooperation Strategy CEGC Clean Energy Generation Capacity CETIF Clean Energy Technology and Innovation Fund COP Chief of Party COR Contracting Officer’s Representative DCA Development Credit Authority (of USAID) DCOP Deputy Chief of Party DGNREEC Directorate General of New Renewable Energy and Energy Conservation DO Development Objective EBT Energi Baru Terbarukan / PLN’s Renewable Energy Division EBTKE New and Renewable Energy and Energy Conservation EDR Evaluation Design Report EE Energy Efficiency EOP End of Project ESDM Ministry of Energy and Mineral Resources, provincial counterpart ESRA Environmental and social risk analysis FIT Feed-in tariff GDA Global Development Alliance GHG Greenhouse Gas GOI Government of Indonesia GW Gigawatts HLR High Level Results ICED I USAID’s Indonesia Clean Energy Development Project I ICED II USAID’s Indonesia Clean Energy Development Project II IFC International Finance Corporation IIF Indonesia Infrastructure Finance IPP Independent Power Producer IR Intermediate Result KII Key Informant Interview LEAP Long-Range Energy Alternatives Planning System LOP Life of Project M&E Monitoring and Evaluation MER Monitoring, Evaluation, and Reporting MEMR Ministry of Energy and Mineral Resources MoEF Ministry of Environment and Forestry MOF Ministry of Finance MoU Memorandum of understanding MR Minimum Results II MTPE Mid-Term Performance Evaluation MW Megawatt NEC National Energy Council NTB Nusa Tenggara Barat/West Nusa Tenggara OJK Indonesia’s Financial Services Authority/Otoritas Jasa Keuangan OPIC US Overseas Private Investment Corporation PFAN Private Financing Advisory Network PIRS Performance Indicator Reference Sheet PLEA Planning for Energy Access PLN State Electricity Company/Perusahaan Listrik Negara PMU Project Management Unit PPA Power Purchase Agreement PV Photovoltaic RAD-GRK Local Action Plan for Greenhouse Gas Emission Reduction/ Rencana Aksi Daerah Gas Rumah Kaca RAN-GRK National Action Plan for Greenhouse Gas Emission Reduction/ Rencana Aksi Nasional Gas Rumah Kaca RDMA USAID Regional Office RE Renewable Energy RPJMD Regional Medium-Term Development Plans RTRWP/K Provincial/District/City Regional Spatial Management Plan RUED Rencana Umum Energy Daerah/Regional Energy Planning RUKD Regional General Plans for Electricity RUEN Rencana Umum Energy Nasional /National Energy Planning RUPTL Rencana Umum Penyediaan Tenaga Listrik /PLN’s Power Supply Business Plan SOW Statement of Work TCMR Technical Component Minimum Results USAID United States Agency forInternational USG United States Government 1 EXECUTIVE SUMMARY INTRODUCTION Integra Government Services International (Integra) supplied a team of four experts to conduct the Mid Term Evaluation (MTE) of USAID’s Indonesia Clean Energy Development (ICED) II Project, implemented by Tetra Tech ES. The purpose of this evaluation was to provide insight to USAID on ICED II’s progress to date, and on how the Agency might reallocate resources to improve or capitalize upon the most impactful activities to date. The evaluation was conducted from January to May 2018, and included five weeks of work in country in Jakarta, North Sumatra, South Sulawesi and East Java. The Evaluation Team was comprised of a Team Leader, an Institutional and Human Capacity Development Specialist, an Energy Policy Specialist and a Renewable Energy Project Development and Financing Specialist. In total, and in consultation with USAID/Indonesia, the team interviewed 37 key informants and conducted four site visits during the five￾week period. Findings, conclusions and recommendations provided this report are a result of the compilation and analysis of data gathered during that time. BACKGROUND ICED II is a $17.2 million project implemented by Tetra Tech ES, which began in May 2015 and will run through May 2020. The project builds on the achievements made during the Mission’s first iteration of ICED, which ran from February 2011 – February 2015 (also implemented by Tetra Tech). Its goal is to strengthen the foundation for a low-carbon energy system in Indonesia by providing assistance to the Government of Indonesia (GOI) in policy, regulatory and incentive frameworks for low emissions growth. It also works with banks, regional government entities, project developers and other stakeholders to enhance the environment for low energy investments, and attract public and private sector investment in clean energy development. ICED II’s programmatic activities fall into three interrelated technical components: 1) Improve the enabling environment for rapid co-investment in clean energy. 2) Accelerate the mobilization of private and public sector co-investment in clean energy. 3) Established funding and implementation frameworks for mitigation actions in the clean energy sector at the subnational level. ICED II also operates a Clean Energy Technology and Innovation Fund (CETIF) that is used to augment project interventions, and is intended to encourage the participation and/or collaboration of the private sector and local entities in the achievement of ICED II components and associated tasks. The development hypothesis of ICED II is that by reinforcing assistance to national and local government institutions, the project improves clean energy and climate change mitigation programs. The establishment, or refinements of law, policy, strategy or regulation is the foundation of the project. ICED II also trains individuals in critical functional areas associated with implementing the project. When coupled with guidelines, tools and standard operating procedures, this leads to improved institutional capacity, resulting in the internal training o new or transferred managers and key staff, that should result in the design and implementation of more effective programs in the long term. 2 Finally, ICED II recognizes that the implementation of clean energy and climate change mitigation programs requires public and private sector investment in renewable energy and energy efficiency projects. ICED II assistance to project developers, banks and financial institutions, and consultants and engineering companies, is designed to improve the viability of these clean energy projects. Once operating, the electricity generated or energy saved results in a reduction of GHG emissions. The additional generation also results in improved energy services to new or existing consumers. EVALUATION PURPOSE The purpose of this evaluation was to provide USAID/Indonesia with an independent review of progress made by ICED II during its implementation from May 2015 through December 2017. As discussed during the in brief with USAID/Indonesia on February 2, 2018, the findings, conclusions and recommendations provided within this report are intended to assist the Mission in identifying which aspects of ICED II have been most effective, as well as how a reallocation of resources might benefit the project in the future to achieve higher levels of success. QUESTIONS, METHODOLOGY AND LIMITATIONS EVALUATION QUESTIONS The ICED II MTE was centered on the following two evaluation questions: 3) To what extent is ICED II on track to meet its expected high-level and technical component-level (“Minimum”) results, and in what manner could either implementation or monitoring efforts be enhanced? a) Do the goal, results, indicators, and targets selected for ICED II appropriately and completely track all program activities, and accurate tools for monitoring and reporting on implementation? Is there a need to refine, add, omit and/or more clearly define the selected results, indicators, and targets? b) Are there any activities in which ICED II is engaged that are making limited contribution to the expected results and/or overarching goal of the program? If so, to what extent and what is the rationale for continued engagement? c) Are there any activities in which ICED II is engaged that show proportionally high contributions to the expected results in consideration of resources allocated? If so, to what extent have efforts been made to build on these activities to intensify their contribution to ICED II’s goal? d) How do the implementation approaches and management of the ICED II program affect the ability to achieve expected results (outputs, outcomes and strategic impacts) effectively, and what constraints are limiting its effectiveness? In addressing this question, the following aspects must be considered: project management approach, relationship with stakeholders, human resources availability, geographic focus, and local versus national stakeholder engagement. e) What are emerging external factors (locally, nationally, regionally and globally) that are affecting/have the potential to affect the implementation or effectiveness of ICED II’s 3 interventions? What mitigating measures are recommended for ICED II address and adapt to these factors? f) How has monitoring, learning and research been used to inform adaptive management in terms of adjusting ICED II’s technical approaches and intervention designs? g) Has ICED II identified key factors and strategies to ensure sustainability and future scaling of program results? If yes, has ICED II incorporated these aspects in the work plans? 4) To what extent can evidence be collected that demonstrates the program’s contribution to either domestic US job creation or expanded market opportunities for US firms? METHODOLOGY The ICED II MTE methodology consisted of a desk review, key informant interviews (KIIs) and site visits. Prior to arriving in country, the Evaluation Team reviewed all ICED II documentation made available to them, including quarterly and annual reports, monitoring and evaluation plan and reports, as well as work plans. A full list of the documentation is included in Annex 2. Following an inbrief with USAID/Indonesia on February 2, 2018 in Jakarta, the team split up to conduct 37 key informant interviews across Jakarta, East Java and South Sulawesi, and two site visits to ICED II supported projects. Over the course of four weeks, 37 interviews were conducted with ICED II staff, government counterparts (national and provincial), project developers and financial partners. No KII lasted longer than 90 minutes, and questions asked were guided by the protocols included in Annex 3. Furthermore, an online survey was administered to all financial and project developer participants to gather additional data. All survey instruments link directly to the evaluation questions, and data gathered was coded where appropriate. For a full list of the evaluation methods used during this MTE please see Annex 5. LIMITATIONS The amount of time allotted for the MTE and key informant availability were the two major constraints of this MTE. While the team was able to meet with the majority of entities identified by USAID/Indonesia and ICED II as relevant stakeholders, several were either unavailable or became unavailable during the course of the evaluation. It should also be noted that the provincial stakeholders were chosen by USAID based on the availability of staff, and their progress with ICED II. Ultimately North Sumatra and East Java were selected to represent the various lengths of ICED II engagement. EVALUATION QUESTIONS Table 1 below summarizes the findings, conclusions and recommendations (FCRs) of the ICED II MTE, organized by evaluation question. For a full explanation of each FCR and its supporting data, please see the FCR section in the main body of this report. TABLE 1. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS BY QUESTION 4 QUESTION FINDINGS CONCLUSIONS RECOMMENDATIONS 1. To what extent is ICED II on track to meet its expected high￾level (HLR) and technical component￾level (“Minimum”) results (TCMR), and in what manner could either implementation or monitoring efforts be enhanced? HLR: ICED II has exceeded 2 HLR targets (HLRs 1 and 3) and is on track to meet targets for HLR 3 and 5, despite a lag in results stemming from investment mobilized (HLR 5). HLR 4: The activity that ICED II supports extensively, RAD-RAN GRK, records large reductions in GHG emissions but it is unclear how much of this is directly attributable to ICED II. TCMR: ICED II’s ability to meet the targets for Component 1 TCMRs 1 and 2 are reliant upon provinces’ abilities to fully engage, constrained by budgetary and other issues. This is partly due to the timing of the Presidential Decree requiring the completion of RUED￾RUEN by the end of March 2018. ICED II is on track to meet or exceed all of its HLR, with the exception of GHG Emissions Reduced. It is clear that ICED II has provided important, critical assistance to the GOI to be able to develop national, regional and local development plans that focus on GHG emission reductions and that the GOI credits large amounts of emission reductions to this planning. More information is needed to be able to attribute these to ICED II . ICED II is on track to meet or exceed all of its TCMRs except for Component 1 TCMRs 1 and 2. USAID should consider refining the Component 1 TCMR 1 to something that better reflects strengthening the capacity of government entities to develop improved RUEDs and RUKDs, as opposed to counting the number of those developed. For HLR 4, ICED II should work with Bappenas to gather and report information that will allow clear attribution of the GHG emission reductions. 1A. Do the goal, results, indicators, and targets selected for ICED II appropriately and completely track all program activities, and accurate tools for monitoring and reporting on implementation? (2) Is there a need to refine, add, omit and/or more clearly define the selected results, indicators, and targets? The first five out of 11 performance indicators track against the same five HLRs discussed above. ICED II has exceeded, or is on track to meet or exceed, the remaining six indicators, except for performance indicator 11 (expected energy savings). The Evaluation Team found that the delay on meeting ICED II’s target for expected energy savings was not due to nonperformance, but rather resulting from a decision made by the Ministry of Energy and Mineral Resources (MEMR). The goal, results, indicators and targets appropriately and completely track all program activities, and tools used are accurate. USAID should consider supporting ICED II’s removal of the energy efficiency indicator, acknowledging that the ICED II is no longer working in this area because of MEMR’s change in direction. 1B. Are there any activities in which ICED II engaged that are making limited contribution to the expected results and/or overarching goal of the program? (2) If so, to what extent and what is the rationale for continued engagement? Energy efficiency had potential to be important in ICED II. However, MEMR wanted ICED II to move in the direction of working on energy efficiency standards. There appear to be two problems with this move: 1) another GOI entity is legally charged with creating standards, and 2) the results on which ICED II is judged would not likely appear within the LOP. ICED I and ICED II have made significant contributions in the area of RAD/RAN-GRK, but now the GOI has placed increased importance on the energy plans, RUED/RUEN. There were two activities in ICED II that are now making a limited contribution: Energy Efficiency and RAD/RAN￾GRK efforts. ICED II has dropped the energy efficiency work. RAD￾GRK has been scaled down to a minimum necessary to maintain ICED II’s already substantial efforts. The scaled back assistance to RAD-GRK is justified because RAD-GRK is still an important part of the GOI’s integrated energy program. No recommendations. The rationale to drop or scale back work in these areas is justified. 5 1 This is based on questions asked during the KII and through the online survey and is based on a total of 18 KIIs with project developers and financial entities and 8 surveys from project developers (out of the twenty invited) or 60% of the ICED II stakeholder list in these two areas. QUESTION FINDINGS CONCLUSIONS RECOMMENDATIONS 1C. Are there any activities in which ICED II is engaged that show proportionally high contributions to the expected results in consideration of resources allocated? (2) If so, to what extent have efforts been made to build on these activities to intensify their contribution to ICED II’s goal? Two ICED II activities exhibited proportionately higher contributions relative to the resources used: 1) Support to PLN including feasibility reviews, guidelines and grid integration 2) Use of geospatial data and tools ICED II has introduced GIS into the planning process, which allowed national and provincial energy authorities as well as PLN to get a more accurate picture of electrification and develop more optimal routes. ICED II also developed datasets and GIS maps of village electrification in 34 provinces, which was then used for the RUED starter kit for all provinces to begin the planning process. Working with PLN in a specific area yields greater results than working with a project developer using relatively the same amount of resources. Not only does it move and improve projects that PLN is developing, but it also helps to remove roadblocks outside project developers. None. 1D. How do the implementation approaches and management of ICED program affect the ability to achieve expected results (outputs, outcomes and strategic impacts) effectively, and what constraints are limiting its effectiveness? In addressing this question, the following aspects must be considered: project management approach, relationship with stakeholders, human resources availability, geographic focus, and local versus national stakeholderengagement. Feedback from respondents on overall program management and relationship with stakeholders was positive. Some respondents cited communication issues that occurred early on in their engagement (program management and relationship with stakeholders) ICED II stakeholders do not communicate well among themselves, but ICED II has been able to play a coordinating role. The experience with ICED II consultants has been positive regarding human resources. ICED II’s geographic focus has not limited its effectiveness. Work performed at the local level to develop guidelines and procedures is replicated at the national level and used to train other stakeholders. ICED II has been effectively managed during the evaluation period. No systemic issues surfaced in the evaluation and KIIs noted the importance of ICED II’s assistance and the quality of that assistance provided to financial entities and project developers was rated a 4 on a scale of 1 to 5, with 5 being the highest quality1. ICED has been able to work with several national and provincial entities that often see themselves as competing and yet been seen as a neutral third party. None. 6 2 Subsequently and partly in reaction to IPP concerns, the MEMR promulgated Regulations 49/2017 to amend 10/2017 on principles of PPA calculation and 50/2017 to revise and replace 12/2017. These two additional regulations are minor in comparison to what they amended or replaced in the sense that either clarified positions, relaxed some conditions on IPPs or extended the restrictions to all IPPs. The net impact on the industry is unclear. Some provisions are good for IPPs but overall for smaller renewable energy IPPs it appears these do not improve on 10/2017 or 12/2017. 3 ICED II has put in place a number of measures that contribute to possible scaling in Component 2 but these rely on the Government not disadvantaging renewable energy (RE). Since 2017, the GOI has created an unfavorable investment climate for most RE, as discussed above, and it is beyond the capability of ICED II assistance to obviate these measures. ICED II has put the measures in place and prior to 2017, these measures would have ensured scaling. 4 By direct effort, it is meant that ICED II has not made any efforts directly with the project developer community. 1E. What are emerging external factors (locally and nationally) that are affecting/have the potential to affect the implementation or effectiveness of ICED II’s interventions? (2) What mitigating measures are recommended for ICED II address and adapt to these factors? The team found two2 major external factors emerging during the MTE: 1) Regulation 10/2017 to address Power Purchase Agreements (PPAs) 2) Regulation 12/2017 addressing Feed-in Tariffs (FITs) While this does not impact any of the projects being support by ICED II, it does influence the decision of developers and financial entities to continue future IPP activity. Most that have not signed PPAs indicate that future projects are on hold except for those associated with palm oil and geothermal energy. These two major factors have not impacted projects that ICED II is assisting since ICED II made the decision to only support projects that had a signed PPA. However, it negatively impacts the entry of new projects into the ICED II pipeline, since both financial entities and most developers indicate that they will not invest in RE until there is a more favorable economic environment. It should also be noted that there are still a large number of projects that can benefit from ICED II assistance in the pipeline. The team recommends that ICED II continue working with those late state projects that needs its help, and increase its efforts with the PLN in addressing standards, guidelines and business practices that remove barriers and reduce costs for renewable energy. The team also recommends that USAID continue its dialogue with the GOI to stress the incompatibility of these recent emerging factors. 1F. How has monitoring, learning and research been used to inform adaptive management in terms of adjusting ICED II’s technical approaches and intervention designs? In detailed discussions with ICED II staff, there is a conscious effort at work planning time to incorporate the lessons learned from the past, including ICED I, and input from stakeholders about their needs and the future of their respective spheres of influence and activity. ICED II’s project management remains flexible and proactive. Monitoring of the project’s performance together with a good understanding of stakeholders’ needs and directions is used both to measure past performance and inform future assistance. None. 1G. Has ICED II identified key factors and strategies to ensure sustainability and future scaling of program results? (2) If yes, has ICED II incorporated these aspects in the workplans? ICED II has put in place measures that will ensure the sustainability and future scaling3 of ICED II assistance in all but one area, the technical advice given to project developers. ICED II provides technical assistance to project developers to facilitate their transaction and the KII’s indicated that this assistance was critical, well-received and high quality. Yet, the team found no evidence that ICED II had made any direct4 effort to ensure sustainability. ICED II staff explained that it has not found a mechanism to create the direct demand for these services or advice. ICED II has made concrete efforts to sustain and scale its efforts and they appear to have taken hold with two exceptions: 1) Scaling with the financial and project developer community. 2) Efforts to develop sustainability mechanisms for the work it has with project developers. ICED II should continues its efforts to sustain and scale efforts, including but not limited to developing guidelines and tools such as feasibility studies to ensure they become standards for the project development community. 2. To what extent can evidence be collected that demonstrates the program’s contribution to either domestic US job creation or expanded market opportunities for US None. The evaluation team found no evidence that ICED II has directly or indirectly created opportunities in terms of jobs or exports for US firms. However, ICED II has been supporting US firms in their attempts to enter and None. 7 firms? understand the market. 8 PROJECT BACKGROUND LOCAL CONTEXT During the design phase of ICED II, the GOI estimated that national electricity demand would increase six times over the next two decades. This translates into 230 GW of additional generating capacity that would need to be integrated into the national energy system -- a substantial increase of 415% from the June 2016 generating capacity of 55.4 GW. USAID/Indonesia recognized that in the absence of a concerted effort to improve the foundation for a low emission energy system, it would be likely that a heavier reliance on coal-fueled power generation would meet the electricity demand of the rapidly growing economy, while remote, underserved populations would continue to rely on expensive and dirty diesel-fueled power. As such, the Mission developed the ICED II technical assistance program to engage with a rangeof public and private sector partners to implement policies, programs, and projects that would make measurable contributions to increasing the portion of Indonesia’s energy mix that is derived from renewable sources. 5 Indonesia has a wealth of renewable energy resources, including hydro, solar, wind, biomass, and geothermal power. However, as of June 2016, only 6.3 percent of Indonesia’s 55.4 GW installed generation was from renewables (this figure had increased to 11 percent as of February 2018, per PLN). Since the inception of ICED II, a number of new measures and commitments have been made by the GOI to increase the use of renewable energy (RE), expanding energy access, and reducing greenhouse gas (GHG) emissions. In accordance with the 2014 National Energy Policy (Kebijakan Energi Nastional or KEN), Indonesia aims to have 23 percent of its energy demand supplied by new and renewable sources by 2025. The country also aims to increase energy access from 85 percent to 100 percent by 2020; and in line with its Nationally Determined Contributions (NDC) reaffirmed at the 2015 United Nations Climate Change Conference of Parties (COP 21), aims to reduce its GHG emissions by 29 percent by 2030 (when compared to a business as usual scenario). Indonesia has stated that that this reduction could be as high as 41 percent with international donor support. However, in spite of these targets, in recent years Indonesia has not made notable steps in bringing large amounts of renewable energy generation facilities online. This is particularly evident when compared to many other nations in the region (i.e. Malaysia, Thailand, and the Philippines). This has largely been attributed to the persistence of an under-supportive policy environment, exacerbated by government administration changes, shifts in planning priorities and incentive regimes, and differences between the Ministry of Energy and Mineral Resources (MEMR) policies and State Electricity Company (Perusahaan Listrik Negara or PLN) implementation realities. Consequently, a number of renewable energy projects of various sizes and technologies have stalled in their planning stages and have been unable to secure power purchase agreements (PPAs) with PLN. OVERVIEW OF THE ICED II PROJECT ICED II is a $17.2 million project implemented by Tetra Tech ES over a five-year period (May 2015 – May 2020). “The overall goal of ICED II is to strengthen the foundation for a low-carbon energy system in Indonesia. This goal is the same as the Mission’s Sub-Intermediate Result (Sub-IR) 3.3.1 of the 5 ICED II MTE RFTOP 9 USAID/Indonesia 2014-2018 Country Development Cooperation Strategy (CDCS), and it contributes to the CDCS Intermediate Result (IR) 3.3: ‘Climate Change Mitigation and Resilience to Support a Green Economy Strengthened,’ and the corresponding CDCS Development Objective (DO) 3: ‘Global Development Priorities of Mutual Interest Advanced.’ In recognition of the potential for program interventions to support technological and human capacity advancements in the clean energy space, ICED II also complements the work of the DO 4: ‘Collaborative Achievement in Science, Technology and Innovation Increased. In light of Indonesia’s current and aspired energy and policy landscape, ICED II provides assistance to the GOI in policy, regulatory and incentive frameworks for low-emission growth. At the same time, ICED II works with banks, regional government entities, project developers and other stakeholders to enhance the environment for low emission energy investments and attract public and private sector investment in clean energy development. ICED II’s programmatic activities fall into three interrelated technical components: (i) Improve the enabling environment for rapid co-investment in clean energy; (ii) Accelerate the mobilization of private and public sector co-investment in clean energy; and (iii) Outreach on USAID and US Government (USG) inter-agency activities in Indonesia’s energy sector. ICED II also operates a Clean Energy Technology and Innovation Fund (CETIF) that is used to augment project interventions. The purpose of the Innovation Fund is to be a flexible means, where necessary and/or useful to the project’s objectives, to encourage participation and/or collaboration of private sector and local entities toward the achievement of ICED II’s objectives. The development hypothesis of ICED II is that by reinforcing assistance to national and local governmental institutions, the project improves clean energy and climate change mitigation programs. The establishment, or refinements, of a law, policy, strategy or regulation is the foundation of the project. ICED II trains individuals in critical functional areas associated with implementing the project. When coupled with guidelines, tools, and standard operating procedures, this leads to improved institutional capacity. Furthermore, improved institutional capacity will result in the internal training of new or transferred managers and key staff, that should, over the longer term, result in the design and implementation of more effective programs. Finally, ICED II recognizes that the implementation of clean energy and climate change mitigation programs requires public and private sector investment in renewable energy and energy efficiency projects. ICED II assistance to project developers, banks and financial institutions, and consultants and engineering companies is designed to improve the viability of these clean energy projects. Once operating, the electricity generated or energy saved results in a reduction of GHG emissions. The additional generation also results in improved energy services to new or existing consumers. DESCRIPTION OF ACTIVITIES COMPONENT 1: IMPROVE THE ENABLING ENVIRONMENT FOR RAPID CO-INVESTMENT IN CLEAN ENERGY Component 1 was designed, primarily at the government level, to increase capacity for low emissions energy sector planning and implementation, and lend support for policy and regulatory reform for clean energy project development. It works with various GOI entities such as Bappenas (the National Planning Agency), Ministry of Finance, MEMR, Regional Government entities (e.g., Bappeda and ESDM), 10 Perusahaan Listrik Negara (PLN. the vertically integrated national utility), and, OJK (Otoritas Jasa Keuangan/ Financial Services Authority) at a minimum the activities must include6: 1) Enhanced integration of Regional Action Plan for Greenhouse Gas Emissions Reduction (RAD-GRK) with the National Action Plan for Greenhouse Gas Emissions Reduction (RAN-GRK), National/Regional General Plans for Energy (RUEN/RUED), Regional General Plans for Electricity (RUKD), the Regional Medium-Term Development Plans (RPJMD), Provincial/District/City Regional Spatial Management Plan (RTRWP/K), and associated strategicplanning. 2) Preliminary RUEDs and RUKDs are developed that reflect integrated resource planning, responsive to energy demands for the region, and aligned with the respective RAD-GRK. 3) Improved capacity of institutions at the local level to operationalize and refine RAD-GRK plans based on sound quantitative data and analysis and synchronization of methodologies at national and local level; and to develop, operationalize and align RUEDs and RUKDs with local development priorities. 4) Enhanced participation of district and municipal institutions in RAD-GRK, RUED and RUKD planning and implementation process. 5) Streamlined reform of existing policies and regulations relevant to the clean energy growth. 6) Public outreach strategies developed at the national/subnational level to effectively communicate the impact of policy/regulatory reform. 7) Facilitated installation of at least 400MW-equivalent of clean energy generation capacity. 8) Adoption of standard business practices for clean energy development, integration and related procedures with PLN. 9) Piloting of at least five renewable energy/energy conservation innovations with a roadmap for scaling-up and/or rapid replication. COMPONENT 2: ACCELERATE THE MOBILIZATION OF PRIVATE AND PUBLIC SECTOR CO-INVESTMENT IN CLEAN ENERGY Component 2 was designed to address challenges at the project level by working with project developers and financial entities. At a minimum the activities must include: 1) Build a database of viable clean energy project pipeline with respective emission reduction contribution. The pipeline of projects should include and align activities identified in the RAD-GRK and/or RUED carried out by local governments and/or state-owned enterprise in select provinces, and projects that are being pursued by private sector project developers as attractive investments with measurable returns. 6 Component level activity descriptions are taken verbatim from the ICED II statement of work. 11 2) Provide advisory services, technical assistance and capacity building to the key players in clean energy market – including but not limited to project developers, investors, domestic banking and non-banking financial institutions, and equity companies – with the main objectives to build their internal capacity in conducting proper assessment of clean energy projects, increase their clean energy portfolio and endorse diverse project financing mechanisms for clean energy projects. The assistance could be in the form of technical evaluations of project designs and business plans, linking projects in pipeline to a range of financing mechanisms that may be available for public/private project developers to access beyond the conventional banking /financial services industry, and facilitating development of pilot domestic financing mechanisms. Examples of financial resource options are national/ international/multi-donor clean energy/climate change funds, insurance providers, and USAID financial mechanisms – such as those provided through the Development Credit Authority (DCA) or Global Development Alliance (GDA). 3) Provide technical assistance to PLN in establishing improved and standardized business practices for engagement with renewable IPPs, for example best practices in PPA structure for small to medium scale projects, and to PLN’s regional offices on relevant policies/policy reforms and their implication on local IPP project development. 4) Conduct initial landscape assessments covering government agencies, industry associations, and academic institutions to identify the capacity building needs of these entities to promote science and technology advancements in the relevant area of clean energy, such as energy planning, modeling and integration. 5) Work with key relevant national/subnational government entities, PLN and industry associations to identify specific renewable energy and energy conservation technology innovations that will be crucial for rapid deployment of domestic clean energy capacity and develop roadmaps and/or pilots for deployment and scaling-up, as well as to assess their individual needs for assistance in strengthening human capacity to support such innovations. 6) Conduct an inventory of the existing educational/training programs that provide training in specific clean energy technology production and maintenance. The Contractor may engage with existing schools and associations to further develop their clean energy curriculum and delivery of associated training. 7) Provide technical assistance to stakeholders in data verification/analysis by developing their capabilities in GHG mitigation/clean energy data collection and modelling tools in order to increase their capacity for evidence-based decision making. COMPONENT 3: OUTREACH ON USAID AND USG INTER-AGENCY ACTIVITIES IN INDONESIA’S ENERGY SECTOR USAID Indonesia and other USG agencies such as USAID’s Regional Development Mission for Asia (RDMA), the Millennium Challenge Corporation, the Department of Commerce, Department of Energy, and the Trade and Development Agency had ongoing energy activities in Indonesia. The purpose of this component was to assist coordination and provide outreach and awareness of USG energy activities and USG-Indonesian collaborative efforts in energy. Activities at a minimum must include: 12 1) Assist an annual coordination forum comprising USG agencies engaged in Indonesia’s climate change mitigation and clean energy activities. 2) Develop a system to document, track and monitor progress of projects assisted under the previous USAID programs and ICED II’s project pipeline. 3) Manage, improve and expand the clean energy website developed by the ICED project. 4) Identify potential areas for collaboration with other USAID environment programs, in particular water and sanitation, sustainable land-use, engagement with sustainable palm oil, climate change resilience, and disaster response, and a develop work plan for joint interventions. 5) Identify potential activities for collaboration with other donors/lenders/initiatives or other gaps in interventions and develop work plan for feasible join-activities. 6) Support to USAID and GOI’s clean energy working group, to be established under the CDCS’s bilateral agreement. EVALUATION PURPOSE AND QUESTIONS EVALUATION PURPOSE The purpose of this midterm performance evaluation “is to provide USAID/Indonesia with an independent review of progress made by the Mission’s largest energy sector activity after its first two years of implementation.”7 The results of this effectiveness evaluation may be used by the USAID/Indonesia Mission to “make informed decisions regarding which aspects of the program have been most effective in advancing the goal of ICED II”8 and if and how a reallocation of project resources might impact effectiveness. EVALUATION QUESTIONS The ICED II mid-term performance evaluation is framed around two primary evaluation questions. The evaluators will thoroughly integrate each of these questions and associated sub-questions within the design of the evaluation and their methodology. 1) To what extent is ICED II on track to meet its expected high-level and technical component-level (“Minimum”) results, and in what manner could either implementation or monitoring efforts be enhanced? a) Do the goal, results, indicators, and targets selected for ICED II appropriately and completely track all program activities, and accurate tools for monitoring and reporting on implementation? Is there a need to refine, add, omit and/or more clearly define the selected results, indicators, and targets? The first part of this question is quantitative, and the second part is qualitative. 7 ICED II MTE RFTOP, page 8. 8 Ibid, page 4. 13 b) Are there any activities in which ICED II is engaged that are making limited contribution to the expected results and/or overarching goal of the program? If so, to what extent and what is the rationale for continued engagement? This requires a combination of quantitative and qualitative data to answer. c) Are there any activities in which ICED II is engaged that show proportionally high contributions to the expected results in consideration of resources allocated? If so, to what extent have efforts been made to build on these activities to intensify their contribution to ICED II’s goal? This requires a combination of quantitative and qualitative data to answer. d) How do the implementation approaches and management of the ICED II program affect the ability to achieve expected results (outputs, outcomes and strategic impacts) effectively, and what constraints are limiting its effectiveness? In addressing this question, the following aspects must be considered: project management approach, relationship with stakeholders, human resources availability, geographic focus, and local versus national stakeholder engagement. This is primarily a qualitative question. e) What are emerging external factors (locally, nationally, regionally and globally) that are affecting/have the potential to affect the implementation or effectiveness of ICED II’s interventions? What mitigating measures are recommended for ICED II address and adapt to these factors? The first part of this question is quantitative, and the second part is qualitative. f) How has monitoring, learning and research been used to inform adaptive management in terms of adjusting ICED II’s technical approaches and intervention designs? This is primarily a qualitative question. g) Has ICED II identified key factors and strategies to ensure sustainability and future scaling of program results? If yes, has ICED II incorporated these aspects in the work plans? This is primarily a qualitative question. 2) To what extent can evidence be collected that demonstrates the program’s contribution to either domestic US job creation or expanded market opportunities for US firms? Evidence can be found through a combination of quantitative and qualitative data. EVALUATION METHODS AND LIMITATIONS DATA COLLECTION METHODS DESK REVIEW Integra reviewed as many ICED II documents as were made available (including but not limited to annual and quarterly reports, work plans, monitoring and evaluation (M&E) plans, ICED II work products, fact sheets, and news updates) prior to arrival in Jakarta for data collection in February. Additionally, the team conducted a literature review of other project-related documents and GOI 14 policies, guidance documents related to ICED II technical areas, as well as relevant USAID policies and guidance documents. Primary documents included: • ICED II Work Plans for Fiscal Years 2015, 2016 and, 2017. • Annual Reports for Fiscal Years 2015, 2016 and, 2017. • Monitoring and Evaluation Plan and annual M&E reports. • Success stories, fact sheets and ICED II updates from the ICED II website. • Various background documents (concept notes, business cases and assessment reports) for activities in Components 1, 2, and 3. Annex 2 contains a complete list of the documents reviewed. The evaluation team used baseline data from internal sources, such as ICED II project appraisal documents and the Activity Monitoring & Evaluation Plan, and external sources, such as published literature, third party reporting and open databases, to establish objective baseline conditions of the ICED II operating environment. The evaluation team reviewed relevant documentation, literature and expert sources to develop the types and sources of information to be used for quantitative and qualitative analysis to answer the evaluation questions. The desk review was used to gather both qualitative and quantitative data, the latter consisting largely of monitoring data. Site visits performed during the MTE were used to verify this data, supplemented by key informant interviews (KIIs), to identify the key results and achievements of ICED II, as well as any areas where the project may have failed to achieve its targets. KEY INFORMANT INTERVIEWS Integra conducted key informant interviews with a range of stakeholders (37 in all) that provided insight and perspective to the ICED II management, implementation and performance. The interviews also explored critical success factors, challenges or barriers to success, results at both the project and policy levels, as well as gender. The KIIs were semi-structured in nature, enabling the team to gather data related to the evaluation questions, while also allowing flexibility to add probing questions based on respondents’ answers. Each KII lasted no longer than 90 minutes (including time required for translation) in order to respect respondents’ other daily obligations. Prior to each interview, the team identified the highest priority questions to cover with that respondent to ensure that it collected the most pertinent data (in light of data already collected). Furthermore, resulting from discussions with USAID/Indonesia in Jakarta, the evaluation team split up in an effort to interview government stakeholders in East Java and South Sulawesi during the same field visit timeframe. See Annex 3 for the KII interview protocols. ONLINE SURVEY 15 Concurrent with the MTE fieldwork period, Integra administered an online survey to all 20 project developers that were not part of the KIIs to collect structured responses related to the evaluation questions. Only question 2 of the survey gathered quantitative data. The online surveys were anonymous but required respondents to identify the type of entity and areas of participation with ICED II to enable the research team to triangulate the findings from KIIs and the desk review. Annex 4 presents the online survey instrument, for which the Evaluation Team received 8 responses. DATA ANALYSIS METHODS Integra took detailed written notes of all interviews (with consent) so that the team could generate transcriptions and translations (if necessary) for coding. The coded responses allowed us to transform qualitative data into quantitative tabulations where possible and appropriate. Each question in each KII protocol had a direct link to an evaluation question (or component of an evaluation question) and was categorized according to those linkages during data analysis. The findings generated through these methods were interpreted in the context of those generated through other qualitative and quantitative methods described above and triangulated accordingly. In addition to the KII responses, the Evaluation Team reviewed reports, administered the online survey and visited sites for verification. Thus, we used a combination of content, quantitative, descriptive and contribution analysis as appropriate. These different analytical techniques are presented alongside the evaluation questions in the Evaluation Design Matrix in Annex 5. Responses from the KIIs were also used as illustrative support. Conclusions were drawn on the preponderance of these sources for each evaluation question and are clearly stated to reflect the logic that supports them. Similarly, recommendations were drawn from those conclusions, along with the expert counsel of the Evaluation Team. The underlying logic supporting the conclusions and recommendations is presented and organized by question, later in the report, so that readers can trace these to the evaluation findings and expert inputs. The Team Leader will also present the findings and recommendations to USAID/Indonesia for further discussion, prior to finalizing the MTE report. STRENGTHS AND LIMITATIONS Integra interviewed as many of the ICED II stakeholders as time and resources would permit based on the list of stakeholders provided by USAID. We interviewed seven of the nine National Government stakeholders, four of the 13 Provincial Government stakeholders, six of the 12 financial entities, and four of the eight PLN regional and corporate stakeholders, six out of six ICED staff, and interviewed or surveyed 18 of the 31 project developer stakeholders. Several individuals identified for the interviews agreed to meet but then at the last minute cancelled while others could not meet the evaluation team during the time available. The individuals the team ultimately interviewed were representative of the project’s stakeholder community with the possible exception of the Provincial Government Stakeholders. The Provincial Stakeholders were chosen by USAID based on the availability of staff, their progress with ICED II and 16 the timeframe within which the evaluation took place. The two provinces represent extremes in regards to their length of engagement: ICED II had been working with North Sumatra since its inception, and had recently begun direct involvement with East Java. Annex 6 contains a list of Entities interviewed and which may have had multiple persons participating in any given interview. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS FINDINGS, CONCLUSIONS AND RECOMMENDATIONS FOR QUESTION 1 Question 1: To what extent is ICED II on track to meet its expected high-level and technical component-level (“Minimum”) results, and in what manner could either implementation or monitoring efforts be enhanced? FINDINGS High Level Results (HLR) ICED II has exceeded two of its five HLR targets, 1) Institutions with improved capacity and 2) Laws, policies, strategies, plans or regulations, is close to meeting the target for HRL 3) investment mobilized, and appears to be significantly behind in the target for HLR 5) Persons with improved energy services, as seen in Table 2. The Evaluation Team found that in the case of HLR 4) GHG emissions reduced, the available information did not establish a direct link of causation from the activities that ICED II supported to the emission reduction. Thus, while the reported progress is very close to meeting the target, it is unclear how much of the progress can be attributed to ICED II. These findings are based on a review of results reported from May 2015 through December 2017, a review of the Monitoring and Evaluation Plan, and discussions with ICED II staff and stakeholders. TABLE 2. HIGH LEVEL RESULT TARGETS AND PROGRESS HIGH LEVEL RESULTS LOP TARGET RESULTS REPORT RESULTS AS OF DECEMBER. 31, 2017 PERFORMANCE TO DATE (%) HIGH LEVEL RESULTS 1. Institutions with improved capacity 20 31 155.0 1. Institutions with improved capacity 2. Laws, policies, strategies, plans, or regulations At least 20 25 125.0 2. Laws, policies, strategies, plans, or regulations 3. Investment mobilized $800 million $730.3 million 91.3 3. Investment mobilized 4. GHG emissions reduced (tons CO2e) 4.5 million 4.03 million 89.5 4. GHG emissions reduced (tons CO2e) 5. Improved energy services (persons) 5 million 725,772 14.5 5. Improved energy services (persons) 17 HLR 4) GHG emissions reduced. The Integra team found two issues of concern with this HLR meeting its end of project (EOP) target. First, ICED II bases this HLR on results from activities that only ICED I worked on, as well as those that both ICED I and II worked on. Second, and an important issue in MTE HLR targets is that ICED II counts GHG emissions reductions as those claimed by the GOI by using the RAD-GRK process. RAD-GRK has and is being supported by ICED II. It is managed and operated by Bappenas, the national planning ministry. It is a planning tool that consolidates the planned activities of implementing entities so that a national level aggregate estimate of GHG Emissions exists. It can also be used by these implementing entities to evaluate the GHG emission reductions of different investments. While the GOI uses this to track the GHG emission reductions of its combined activities (the number reported by ICED II), it is unclear how much of this can be attributed to ICED II’s assistance. The information was missing that would establish the link of causation that these implementing entity activities included in the plan could be attributed to ICED II. HLR 5) Improved energy services. This target is linked to investment mobilized with a time lag. First, the investment is mobilized and financial closing is achieved. Second, construction takes place, is completed and commissioned, and only then, with the plant becoming operational (and generation capacity is counted in the targets), are improved energy services counted (by persons). This lag can be more than two years. The team reviewed the projects counted in the investment mobilized indicator that had not yet been counted in the clean energy generation capacity (CEGC) indicator and this together with KIIs indicate that most of these projects will reach operational status during the remainder of ICED II’s time. ICED II is expected to meet its target. Technical Component Minimum Results (TCMR) ICED II has 18 minimum results spread over three components. ICED II has met or can be expected to meet most of these minimum results with two possible exceptions. Table 3 presents the TCMR by component and the finding for each. TABLE 3. TCMR BY COMPONENT AND FINDING COMPONENT 1 MINIMUM RESULT (MR) FINDING 1. Enhanced integration of RAD-GRK with RAN￾GRK, RUEN/RUED, RUKD, RPJMD, Provincial/District/City Regional Spatial Management Plan (RTRWP/K), and associated strategic planning. Finding: Not yet met. Possible but outside ICED II control.* Explanation: ICED II has contributed to this but the MR has not yet been met because the integration has not taken place; budget & other issues have prevented provinces from fully engaging. The Presidential Decree came out in early 2017 requiring the completion of the RUED￾RUEN by the end of March 2018. This was already into the fiscal year for some provinces and they could not budget for this activity until well after the Decree, putting them well behind the time frame for completion. MEMR also did not issue operational guidance until late in 2017. While this was a Presidential Decree, some provinces had more pressing and urgent issues and so did not engage early on. So, while ICED II has provided data, training and tools, it can’t make the multiple participants in the process integrate the tools. Examples of ICED II Contribution: ICED II completed data encoding of Local Action Plan for Low Emission Development (RAD-GRK) implementation measures and achievements in 34 provinces during 2010-2015 period. The database is part of an online system of monitoring, evaluation, and reporting (MER) of RAD GRK that RAN/RAD-GRK Secretariat prepared. ICED II also developed a template of the narrative section on the energy sector for the revised Provincial RAD-GRK and is collecting relevant provincial planning documents. The template is part of a starter kit prepared by the Secretariat to facilitate the RAD-GRK revision process. 18 9 Provinces engaged in the activities that led to this TCMR because it was (1) associated with training and (2) the RAD-GRK process was fully budgeted and an ongoing activity. RUED-RUEN is new, having been introduced in 2017, and many provinces did not have the funding in place to carry out implementation activities. 2. Preliminary RUEDs and RUKDs are developed that reflect integrated resource planning, are responsive to modeled energy demands for the region, and are aligned with the respective RAD- GRK. Finding: Not yet met. Possible but outside ICED II control.* Explanation: Same as above. ICED II has contributed to this MR, but it has not yet been met because the RUEDs have not been completed by the provinces. Examples of ICED II Contribution: MEMR and the National Energy Council (NEC) General Secretary adopted ICED II's proposals on: 1) an outline of the RUED general and technical guidelines, 2) a draft of RUED general guidelines, 3) incorporating electrification planning as part of RUED, and 4) establishing a RUED technical team as part of the Energy Planning Joint Secretariat comprised MEMR, NEC and Bappenas. MEMR and the NEC General Secretary formally introduced the RUED Provincial Starter Kit in the National Workshop on RUEN Dissemination and Implementation, March 13-15, 2017, attended by the Head of Provincial Energy Offices. ICED II contributed in developing the data set, energy model and four guidelines in the kit. ICED II provided input to the RUEN finalization. ICED II always emphasizes the need to align the finalization with RUED preparation. It includes how to structure and collect data, run the model, and conduct energy planning as well as training for Provinces. * Comment on TCMR 1 and 2. Both of these TCMR are well beyond the control of ICED II. Budget issues and the lack of guidelines from the GOI are two of the major reasons that these two MRs have not been met. ICED II has made major contributions to facilitate these results including the technical assistance to help overcome human resource, data, and tool constraints. 3. Improved capacity of institutions at local level to operationalize and refine RAD-GRK plans based on sound quantitative data and analysis and synchronization of methodologies at national and local level; and to develop, operationalize and align RUEDs and RUKDs with local development priorities. Finding: Met. Examples of ICED II Contribution: Finalized the draft narrative template of the Local Action Plan for GHG Emission Reduction (RAD-GRK) for the energy sector. Indonesia’s 34 provinces will use the template to revise their RAD-GRK documents. ICED II also held a workshop in which 20 provinces completed and submitted the initial drafts of their RAD-GRK energy sector documents. Twenty-seven institutions received improved capacity, especially for development of Monitoring, Evaluation, and Reporting (MER) of RAD-GRK in Energy Sector. ICED II has been providing support for National Action Plan for Low Emission Development Secretariat (RAN Secretariat) by developing General Guidelines, Technical Guidelines, and Manual of the MER and methodology to calculate Greenhouse Gas emission reduction in the energy sector that becomes part of the MER document. The guidelines and methodology became main references for provincial governments in developing MER of RAD-GRK for further submission to National Action Plan for Low Emission Development Secretariat. 4. Enhanced participation of district and municipal institutions in RAD-GRK, RUED and RUKD planning and implementation process9. Finding: Met. Examples of ICED II Contribution: Proposed the Draft MER RAN GRK, which improved the previous practice of using the template of MER RAD GRK (sub-national level) for national￾level reporting and this was adopted. 30 provinces attended the training event. In energy sector class, 22 provinces attended the class and 20 provinces updated their RAD-GRK Review Draft. ICED II co-sponsored a second training session on MER assistance and evaluation in Lombok. The training was designed for all 34 provinces’ RAD-GRK working groups, in an effort to facilitate their completion of MER calculations and reporting. ICED II supported the finalization of RAD-GRK Monitoring, Evaluation and Reporting (MER) guidelines. The project also co-funded the National Workshop on the Finalization of Monitoring, Evaluation and Reporting (MER) of RAD-GRK. Twenty-one provinces participated in the energy and transportation sectors, which resulted in 64 percent submission of the reports. ICED II conducted two workshops on Planning for Energy Access (PLEA) to local stakeholders of North Sumatra. 5. Streamlined reform of existing policies and regulations relevant to the Finding: Met. Examples of ICED II Contribution: Presidential Regulation 22/2017 on the National Energy Plan (RUEN) was issued in early March 2017. ICED II previously participated in updating the 19 clean energy growth. database, reviewing the RUEN energy model and developing the narrative section, which all became part of the attachment of the Presidential Regulation. MEMR and the NEC General Secretary adopted ICED II’s proposals on: 1) an outline of the RUED’s general and technical guidelines, 2) a draft of RUED general guidelines, 3) incorporating electrification planning as part of RUED, and 4) establishing a RUED technical team as part of the Energy Planning Joint Secretariat comprised MEMR, NEC and Bappenas. Proposed the Draft MER RAN GRK, which improved the previous practice of using the template of MER RAD GRK (sub-national level) for national-level reporting. 6. Adoption of effective fiscal and non-fiscal incentives for increased investment in different types of small to medium scale renewable energy technologies. Finding: Met. Examples of ICED II Contribution: ICED II presented the results of a legal review of the Energy Resilience Fund to the Various Energy Director of DGNREEC. ICED II supported the clean energy and energy efficiency market assessment by facilitating a series of interviews with ICED II key stakeholders in the market. ICED II worked with International Finance Corporation (IFC) to initiate green bonds in Indonesia. ICED II supported Nikko Securities in developing a clean energy fund, in particular on solar PV market and technical assessment. A series of meetings and consultations have been concluded. ICED II discovered and corrected significant misunderstandings of Nikko Securities about market readiness. They were under the impression that currently many solar PV projects are available and ready to be financed. ICED II corrects this with actual status of project cycle information for these projects. 7. Establishment of regulatory framework and standardized feed-in tariffs (FITs) for the development of renewable energy projects with generating capacity larger than 10 MW. Finding: Met. Explanation: OJK is a regulatory entity that with ICED II’s help has developed mechanisms for sustainable energy finance. MEMR is responsible for the FIT. Standardized FITs were produced in 2015 and 2016 with input from ICED II. Examples of ICED II Contribution: OJK and IFC Indonesia learned how the Department of Energy of the Philippines developed a subscription capacity model in tendering the FIT for wind farms. ICED II presented different mechanisms in Indonesia for wind farms and was advised on the adoption of the subscription method as a viable mechanism for both the utility and increased RE penetration. ICED II shared the findings of its study on the technical potentials of solar PV in PLN systems, and the implications for the design and structure of the FIT for solar PV, market size, and quota system mechanisms. The study was presented to MEMR’s Directorate of Various Energy. ICED II also provided inputs to the Directorate on the mechanism for private sector engagement in solar PV development as described in MEMR Regulation 19/2016. ICED II provided inputs to PLN for the modification of the existing standard PPA on hydro in response to the new MEMR Regulation 19/2015 on the FIT for Mini-hydro. ICED II provided input to the Task Force on the Acceleration of New Renewable Energy Development on the development of the Sustainable Energy Fund concept note. 8. Public outreach strategies developed at the national/subnational level to effectively communicate the impact of policy/regulatory reform. Finding: Met. Explanation: ICED II does not create the outreach strategies but does provide inputs in the form of analyses, presentations and other contributions. Example of ICED II Contribution: ICED II contributed in developing the data set, energy model and four guidelines in the kit, as well as providing trainers during the National Workshop on RUEN Dissemination and Implementation workshop. ICED II chaired a discussion on the energy and transportation working group. The forum was conducted in two days and discussed sustainable finance, energy and transportation, land based/ agriculture, maritime and others. The forum was attended by relevant stakeholders such as the regulator, financial institutions, donor and civil society organization (CSO). ICED II supported OJK and IFC in the ASEAN Sustainable Finance forum in Manila. ICED II supported data on Indonesia clean energy market condition and update on latest development Component 2 MINIMUM RESULT FINDING 20 1. Facilitated installation of at least 400MW-equivalent of clean energy generation capacity. Finding: On course to meet by EOP. Examples of ICED Contribution: • Sangir Hulu Mini Hydro Project (10 MW) • Cilaki IB Mini Hydro Project (9.69 MW) • Parmonangan Mini Hydro Project (9 MW) • Cibalapulang I Mini Hydro Project (9 MW) • Pakkat Mini Hydro Project (10 MW) • Taludaa 2 Mini Hydro Project (2.3 MW) • Cianten 2 Mini Hydro Project (4.8 MW) • Cianten 1B Mini Hydro Project (6.2 MW) • Cianten 3 Mini Hydro Project (5.6 MW) 2. Adoption of standard business practices for clean energy development, integration and related procedures with PLN. Finding: Met. Examples of ICED II Contribution: ICED II supported PLN’s adoption of the following standard business practices: Methodology for calculating solar PV project developments with low penetration on PLN’s grid was used in determining the initial capacity for bidding under MEMR Regulation 19 of 2016 for solar PV FIT, including the 85% of based allowable tariff calculation for subsystems with high production costs Internal guidelines for PLN for renegotiating price of power purchases from geothermal independent power producers (IPPs) that were applied to two projects during PLN negotiations in West and South Sumatra (implemented) Methodology for estimating the levelized cost of energy for solar PV projects was used in evaluating proposal for projects in Lombok, NTB and in Minahasa, North Sulawesi 3. Established funding and implementation frameworks for mitigation actions in the clean energy sector at the subnational level. Finding: Met. Examples of ICED II Contribution: ICED II supported the introduction of various counterparts to Nikko to introduce new funding sources and/or financial products. OJK, Indonesia’s Financial Services Authority, has approved a Renewable Energy Mutual Fund that will be managed by Nikko Securities. ICED II provided training to Nikko prior to OJK’s decision to approve the fund and continued to support by introducing project developers to the Fund Manager and facilitating detailed discussions. Support to PT SMI’s financing mechanism for Makassar City Government to scale up their street lighting program (proposed). Support the Indonesia Energy Security Fund, which included a provision for funding the subsidy for payments to renewable energy IPPs (proposed). Support Clean Energy for Rural Electrification Program (proposed). 4. Documented assessment of human resources needs of relevant stakeholders and action-plan for designing and delivering associated technical assistance to enhance the capacity of local organizations in providing relevant services and trainings in clean energy aspects. Finding: Met. Examples of ICED II Contribution: Training Provider Inventory has been developed and reported seven potential training institutions list along with other relevant information. Assisted the IFC in conducting training needs assessments of 10 banks/financial institutions and in developing a communication strategy for sustainable finance initiatives. Identified business schools or faculty of economics at major universities with programs or research centers as possible programs or centers that could bring ESRA training to the next level (i.e., Economic Environment Indonesia –Faculty of Economics, University of Indonesia; Center for Sustainable Development Goal Studies, Faculty of Economics, Padjajaran University; Center for Financial Inclusivity and Sustainability, Faculty of Economics, Trisakti University). Agreed to assist the Human Resources Development Agency (BPSDM) of the MEMR to help improving its capacity as renewable energy and energy efficiency training center (at the Renewable Energy and Electricity Education and Training Unit. Assessed the capacity needs for the local action plan to reduce GHG emissions (RAD-GRK) and 21 the national action plan (RAN-GRK), as well as provides training, data and tools to build capacity. ICED II held a focus group discussion to determine the gaps between current and expected training modules to support the development of a curriculum for solar PV projects. Conducted a training needs assessment in PLN EBT, and resulted the needs of developing guideline to review mini hydro project feasibility study. 5. Pilot modules for study and practical training in at least five institutions for at least five different technologies. Finding: On course to meet by EOP. Examples of ICED II Contribution: ICED II finalized the ESRA training package and provided it to OJK, which then transferred it to seven universities to initiate clean energy financing capacity building. Three of the modules are related to clean energy technology (mini-hydro power financing, solar photovoltaic power financing, and biogas from palm oil mill effluent). ICED II disseminated modules developed for Clean Energy Financing and Sustainable Finance Training (formerly known as the Environmental and Social Risk Assessment or ESRA) to BRI Corporate University and Pertamina University. Finalized two guidelines for a RUED technical training series: 1) Guidelines on Geospatial Mapping of Electrification Data as RUED starter kit, the guideline to process the raw data of village electrification. 2) Guidelines on Geospatial Mapping of 20 kV line as RUED starter kit, the guideline to prepare a geospatial map of 20 kV electricity network. 6. Piloting of at least five renewable energy/energy conservation innovations with a roadmap for scaling￾up and/or rapid replication. Finding: On course to meet by EOP. Examples of ICED II Contribution: ICED II completed topography mapping by using a drone for the 5 MW Cikaengan Mini Hydro Power Plant project site in Garut District, West Java. Formulated the scope of the ICED II Municipal Energy Efficient Street Lighting Program, which includes technical support in mapping existing and planned streetlights, procurement and budgeting/financing, controls, testing and metering. In total, ICED II has facilitated the installation of approximately 195,000 LED lamps in municipal streetlights in Makassar City. 7. Improved business practices of project design and feasibility assessment in the clean energy market to reduce risk and increase likelihood for attracting investment. Finding: Met. Examples of ICED II Contribution: ICED II provided technical advisory services to project developers, banks and non-bank financial institutions. For example, the project: Conducted 3 training courses for 103 persons in ESRA covering hydropower, solar PV, biomass and biogas Prepared 65 Project Memos for projects based on prefeasibility studies, feasibility studies, interconnection studies, project design and construction, and system operations. Conducted 8 in-house training workshops for the following partner organizations: MEMR, PLN, Bank BRI and Bank Sumut. ICED II had been and is working with PLN develop a number of guidelines and business practices including: 1) Guidelines on Geospatial Mapping of Electrification Data, 2) Guidelines for Interconnecting Renewable Energy Generating Plants to the 20 kV medium voltage network, 3) PLN’s internal guideline for geothermal renegotiation, and, 4) Guidelines for hydropower project feasibility studies. Component 3 MINIMUM RESULT FINDING 22 Question 1 (continued): In what manner could either implementation or monitoring efforts be enhanced? ICED II implementation and monitoring are on track to achieve the project’s objectives by EOP with one major exception, GHG emission reductions. CONCLUSIONS ICED II has 5 HLR and 18 TCMR. Based on a review of the ICED II documents and KIIs with stakeholders, the team concluded that ICED II has met or will meet and exceed most of its HLR targets and TCMRs by EOP, with the major exception of GHG emission reductions. Additional information is needed to establish how much of the GOI emission reductions counted by RAD/RAN-GRK can be attributed to ICED II’s assistance. The tool and process has been and continues to be critical to the GOI to fully account for the emission reductions as a result of its combined activities. This is the amount of emission reductions reported by Bappenas. For ICED II to be able to claim the emission reductions, it needs to be able to establish that the assistance they provided was directly linked to the 1. Regular updates of USAID’s efforts in the energy sector disseminated to interested stakeholders/media highlighting their integration with GOI initiatives and other donor/lender activities . Finding: Met. Explanation: ICED II fulfills this MR through fact sheets, news and success stories that are routinely published on its website. It also conducts specialized meetings and participates in industry related meetings to disseminate USAID’s efforts. Examples of ICED II Contribution: ICED II produced its first success story: “PTPN X Leads the Way in Indonesia’s Development of Clean Energy through its Bioethanol Power Plant.” ICED II participated in the US – Indonesia Energy Policy Dialogue. The program featured presentations by representatives of the US. Department of Energy, US Department of Commerce’s Foreign Commercial Service (USDOC FCS), U.S. Department of State, US Treasury Department, US Trade Development Agency, Millennium Challenge Corporation, and USAID. At the dialogue, USDOE confirmed its interest in supporting MEMR on the Center of Excellence. At the November 23, 2015 International Seminar on Sustainable Finance to Support Sustainable Development Goals, Indonesia’s OJK launched the Guideline for Understanding Environmental Permits in Clean Energy. Designed for banks and financial institutions, the guideline was developed by ICED II, with support from OJK and the Ministry of Environment and Forestry. ICED II facilitated the signing of an MOU between USAID and PLN. 2. Annual workshops, forums, and other public outreach campaigns to inform local stakeholders about GOI and USG’s collaborative efforts. Finding: Met. Examples of ICED II Contribution: ICED II gave a presentation and staffed an exhibition booth at the 4th annual EBTKE Conference and Exhibition. Additionally, on August 21, ICED II held a training session as part of the EBTKE ConEx 2015 exhibition. The “Challenge and Opportunities in Accelerating Hydro Power Development in Indonesia” session also served as part of the project’s launch. ICED II met with AFD, the French development agency, which had a loan facility dedicated to clean energy (both renewable energy and energy efficiency). Participated in USAID’s reception and exhibition, which was held on November 11. 2015 in Jakarta. The event showcased USAID/Indonesia’s projects to USAID Assistant Administrator, Jonathan Stevers, during his visit to Indonesia. ICED II participated and opened an exhibition desk in OJK’s International Sustainable Finance Seminar in November 2015, where many seminar participants were exposed to ICED activities and its support to the Sustainable Finance Roadmap. On October 16, 2015 ICED II and ASHRAE held a one-day seminar at the International Indonesia Seafood and Meat Expo 2015 as part of ICED II’s scoping energy efficiency program in the industrial sector. 23 emission reductions. In other words, how much of those reductions was the result of implementing entities using the tools, training and assistance that ICED II provided versus the activities that were already planned and would have been implemented without ICED II’s assistance. ICED and Bappenas did not have that level of information for the evaluation to review. While the project appears to be behind in reaching its target for HLR 5) Increased energy services, the evaluation’s data review finds that ICED II is on track to achieve its target by EOP. RECOMMENDATIONS 1) USAID should consider refining the indicator for Component 1, TCMR 2 that reads “Preliminary RUEDs and RUKDs are developed that reflect integrated resource planning, are responsive to modeled energy demands for the region, and are aligned with the respective RAD- GRK” to something such as “Strengthening the ability of Government entities at the Provincial and National level to develop RUEDs and RUKDs that reflect integrated resource planning, are responsive to modeled energy demands for the region, and are how aligned with the respective RAD-GRK.” 2) ICED II should work with Bappenas to disaggregate its GHG reporting to determine how much can be attributed to ICED II as compared to how much is simply counted using the tool and process which ICED II has provided. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS FOR QUESTION 1A 1) Do the goal, results, indicators, and targets selected for ICED II appropriately and completely track all program activities, and accurate tools for monitoring and reporting on implementation? (2) Is there a need to refine, add, omit and/or more clearly define the selected results, indicators, and targets? The overall goal of USAID’s ICED II project is, “To strengthen the foundation for a low-carbon energy system in Indonesia.”10 In attaining this goal, ICED II has 11 performance indicators, the first five of which are covered as HLRs in Question 1. This section addresses the remaining six, presented in Table 4. 10 ICED II Statement of Work. TABLE 4. LOWER LEVEL INDICATORS PERFORMANCE INDICATORS BELOW HIGH LEVEL TARGETS UNITS TARGET RESULTS (LOP) AS OF 9/30/2017 % COMPLETED 6. Clean energy generation capacity installed or rehabilitated as a result of USG assistance Megawatt (MW) 400 89 22 7. Proportion of females who report increased self-efficacy at the conclusion of USG supported training/programming. Numbers of females 70% 85% Exceeded 8. Clean energy generation capacity supported by USG assistance that has achieved financial closure Megawatt (MW) 400 279 69 24 FINDINGS 1) The team conducted interviews with stakeholders and conducted an online survey to determine if there were any activities that did not appear in the project M&E reports, work plans and annual reports. ICED II is tracking all their activities. 2) As can be seen from the lower level indicator table above, ICED II has already exceeded two of the targets for performance indicators 7 and 9, is over 50% of the targets for indicators 8 and 10. ICED II appears to be well below on two of the six targets: 1) Clean energy generation capacity installed or rehabilitated as a result of USG assistance, and 2) Expected lifetime energy savings from energy efficiency or energy conservation (ELES). a) Clean energy generation capacity (CEGC). There is a time lag here and a connection between this indicator and the indicator investment mobilized. First, the investment is mobilized, financial closing is achieved, and second, construction takes places, is completed and then commissioning, and finally the operational plant can be counted in clean energy generation capacity. This lag can be more than two years. The team reviewed the projects counted in the investment mobilized indicator that had not yet been counted in the CEGC indicator and this together with KIIs indicate that most of these projects will reach operational status during the remainder of ICED II’s time. ICED II is expected to meet its target. b) Expected lifetime energy savings from energy efficiency or energy conservation (ELES). Due to circumstances beyond its control, energy efficiency activities that would yield the required results on this indicator are not a priority for MEMR. The Director General of New and Renewable Energy at MEMR wanted ICED II to stop its direct assistance in activities such as municipal street lighting that would yield the required project results during life of project and focus on activities such as appliance or equipment standards that would not have yielded results during the LOP. CONCLUSIONS ICED II has done a very good job of aligning its activities to the goal, results, indicators, and targets and of accurately monitoring implementation and progress towards meetings those goals, results, indicators, and targets. 9. Projected greenhouse gas emissions reduced or avoided through 2030 from adopted laws, policies, regulations or technologies related to clean energy as supported by USG assistance** Metric tons (millions) 27 34 Exceeded 10. Number of people trained global in clean energy supported by USG assistance Numbers of persons 5,000 2,783 56 11. Expected lifetime energy savings from energy efficiency or energy conservation, as a result of USG assistance Gigajoules (thousands) 2,800 753 27 25 The goal, results, indicators, and targets selected for ICED II appropriately and completely track all program activities, and tools for monitoring and reporting on implementation are accurate with the one exception of GHG emission reductions. ICED II has provided a number of important tools and databases and provided capacity building. However, in reviewing how the Government entities in energy interact11 and the budgeting issues they face with regard to the timing of RUED, the TCMR “Preliminary RUEDs and RUKDs are developed that reflect integrated resource planning, are responsive to modeled energy demands for the region, and are aligned with the respective RAD-GRK” is well beyond the ability of ICED II to sufficiently influence. In other words, it is enough outside the control of ICED II that ICED II’s activities may not be sufficient to move the needle. RECOMMENDATIONS 1) USAID should remove of the energy efficiency indicator, acknowledging that the ICED II is no longer working in this area because of MEMR’s change in direction. FINDINGS AND CONCLUSIONS FOR QUESTION 1B 1) Are there any activities in which ICED II engaged that are making limited contribution to the expected results and/or overarching goal of the program? (2) If so, to what extent and what is the rationale for continued engagement? FINDINGS 1) Are there any activities in which ICED II engaged that are making limited contribution to the expected results and/or overarching goal of the program? Yes, there are or were two activities in ICED II that are now making a limited contribution. The first is Energy Efficiency, which had been an important activity in ICED I and held the potential to be important in ICED II. Many municipalities had a strong economic incentive to invest in energy efficient street lighting and this was planned in ICED II. However, MEMR wanted ICED II to move in the direction of working on energy efficiency standards. There appear to be two problems with this change. Another GOI entity is legally charged with creating standards, and the results on which ICED II is judged would not likely appear within the LOP. The second is the work on RAD/RAN￾GRK. ICED I and ICED II have made significant contributions in this area but now the GOI has placed increased importance on the energy plans, RUED/RUEN. While continued ICED II efforts would in the legal sense meet its TCMR, there is a greater need for assistance for RUED/RUEN. CONCLUSIONS 11 There are several examples of this. The first was in North Sumatra where the provincial ESDM, the provincial counterpart of MEMR, could not get the data they needed from their Federal Counterpart to complete a joint product, RUED. Another example was from KII (7), who stated, “Coordination at the Ministry level is still difficult. They still have 2 institutions that want to lead this work. Cannot answer whether ICED II is helping or not to improve this problem. It is very difficult for ICED II to serve all institution. The coordination is also likely confusing at the provincial level, because do they deal with Bappenas vs. MEMR vs. National Energy Council.” 26 2) If so, to what extent and what is the rationale for continued engagement? ICED II has dropped the energy efficiency work. RAD-GRK has been scaled down to a minimum necessary to maintain ICED II’s already substantial efforts. The scaled back assistance to RAD-GRK is justified because RAD-GRK is still an important part of the GOI’s integrated energy program, which seeks to develop RUED-RUEN and integrate it with RAD-GRK. FINDINGS AND CONCLUSIONS FOR QUESTION 1C (1) Are there any activities in which ICED II is engaged that show proportionally high contributions to the expected results in consideration of resources allocated? (2) If so, to what extent have efforts been made to build on these activities to intensify their contribution to ICED II’s goal? FINDINGS There are two activities that appear to show proportionally higher contributions to expected results relative to the resources expended. They are: 1) Support to PLN including feasibility reviews, guidelines and grid integration. PLN is both a renewable energy project developer and a buyer of energy from RE projects. Working with PLN in a specific area yields greater results than working with a project developer using relatively the same amount of resources. This is similar to the proportionally higher results that are achieved by working with a multi-project developer as compared to a developer that will build only one project. Not only does working with PLN move and improve projects that PLN is developing but it also helps to remove roadblocks to outside project developers. 2) The use of geospatial data and tools. ICED II has introduced the use of geospatial data and tools into the planning process and this has allowed national and provincial energy authorities as well as PLN to get a more accurate picture of electrification and develop more optimal routes. It has been developed in detail for Nias Island. ICED II applied the experience from Nias Island to develop and process datasets and geospatial maps of village electrification in 34 provinces. This was then used for the RUED starter kit for all provinces to begin the planning process. CONCLUSIONS Two ICED II activities exhibited proportionately higher contributions relative to the resources used and ICED II made plans and followed through with concentrated efforts in those areas. FINDINGS AND CONCLUSIONS FOR QUESTION 1D How do the implementation approaches and management of ICED program affect the ability to achieve expected results (outputs, outcomes and strategic impacts) effectively, and what constraints are limiting its effectiveness? In addressing this question, the following aspects must be considered: project management approach, relationship with stakeholders, human resources availability, geographic focus, and local versus national stakeholder engagement. FINDINGS 27 This was a key question in the key informant interviews and the results for each area are presented below. Project Management Approach: Respondents report that overall project management has been strong and no KIIs reported major issues. “ICED II project is very important.12” Only two KIIs report minor communication issues at early stage engagement in Component 1. One respondent raised a programing issue here: They felt ICED II should provide more direct funding to bring all regional RUED participants to Jakarta for training. Relationship with Stakeholders: ICED II works alongside stakeholders that don’t communicate well among themselves and ICED II is seen as either the stakeholder’s consultant or a neutral party. ICED II is able to play the role of coordinator. Overall, KIIs noted the importance of ICED II’s assistance and the quality of that assistance provided to financial entities and project developers was rated a 4 on a scale of 1 to 5, with 5 being the highest quality13. ICED II’s neutral coordination role led one provincial government official to ask the team if ICED II could help his office get much needed data to complete the RUED from MEMR in Jakarta. Two stakeholders reported that early in their relationship that ICED did not communicate well or frequently and that the terms of engagement were not proposed early enough. Human Resources Availability: All KIIs reported that ICED was very responsive to their needs, acting in a timely manner and ICED consultants were “very good”. The North Sumatra Bappeda office said that ICED II’s experts were “easily reachable”. Project developers and financial entities alike indicated that ICED II’s consultants provided very useful advice. Geographic Focus: Has ICED II’s geographic focus, i.e., five focus provinces, impacted effectiveness in terms of achieving its targets? Based on the review of project documents, it would not appear that this is the case. However, in the KII process, differences of opinion occurred with regard to Component 1. ICED II is not limited in Component 2 to the focus provinces. ICED I and II provided a good deal of direct assistance with the RAD-GRK process to the focus provinces and in training to all provinces. All provinces appear to be fairly adept at the process and modelling. This was done over a number of years with direct assistance to focus provinces and other provinces were provided training and data that ICED II developed, as the work is rolled out. As one KII said, “This is not a problem because with ICED II works on the five provinces and at the national level, it will be replicated.” RUED follows a similar process. ICED direct assistance was at first focused in North Sumatra and then rolled out to other focus provinces. The work, data and tools used in these provinces are made available to other provinces, as is training at a national or regional level. However, due to the need to meet a tight timeline of a Presidential decree, GOI does not have the luxury of following the ICED process. It is of note that other than West Nusa Tenggara, only those provinces receiving donor support for RUED are on track or close to completing the RUED process. One MEMR KII believes that ICED II’s focus on 12 North Sumatra Bappeda. 13 This is based on questions asked during the KII and through the online survey and is based on a total of 18 KIIs with project developers and financial entities and 8 surveys from project developers (out of the twenty invited) or 60% of the ICED II stakeholder list in these two areas. 28 only the five provinces limits the ability of other provinces to come up to speed on RUED because these other provinces are not getting direct support. In other words, a MEMR KII wants ICED II help for all provinces to implement RUED. Several national and provincial officials requested greater direct assistance in implementation. There is an important trade-off though. The more that ICED II does in implementation, the less self-sufficient these entities are. ICED II provides them with the critical training, data and tools to do the job and in focus provinces has or will provide more hands-on efforts. ICED II then uses the results of this hands-on effort to refine the data, tools and training made available to all provinces. Local versus National Stakeholder Engagement: USAID wanted to determine if there were differences in the effectiveness of technical assistance by working through different stakeholders. In particular, the focus was on whether the work done at the local level was then taken up at a national level and disseminated to other local stakeholders. This was focused on component 1. The team found that much of the work done at a local level was used to develop guidelines and procedures. It was then replicated at the national level and used to train national and other local stakeholders and disseminate standard guidelines, tools and data. Planning for Energy Access (PLEA) is an example of how local level work was then taken to a national level. PLEA “is designed to help local governments identify and utilize their resources for electrification planning in order to improve energy access. PLEA is being implemented in three stages: 1) projecting the capability of PLN’s grid to supply electricity in the next 5 and 10 years using Network Planners software, 2) finding the best energy source(s) potentially available in areas outside the PLN grid’s range using HOMER software, and 3) analyzing supply-demand and designing an electrification system based on the results of Stage 2 (solar PV in a specified remote area).” 14 The results were then used in a workshop at the national level to introduce national level stakeholders to PLEA’s data process, methodology and initial findings. Next the results were then presented in a regional workshop in Medan so that other North Sumatra energy related entities could benefit from the Nias experience. Finally, the experience from the Nias Electrification work was applied to developing datasets and geospatial maps of village electrification in 34 provinces. These data then became part of the starter kit for RUED development and distributed to the respective Provincial Government. CONCLUSIONS ICED II has been effectively managed during the evaluation period. No systemic issues surfaced in the evaluation and ICED II has been able to work with several national and provincial entities that often see themselves as competing and yet been seen as a neutral third party. In this role, ICED II has been able to coordinate and facilitate among these entities where left own their own, this does not appear to happen so quickly or at all. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS FOR QUESTION 1E 14 ICED II Annual Report FY 2016, page 12. 29 1) What are emerging external factors (locally and nationally) that are affecting/have the potential to affect the implementation or effectiveness of ICED II’s interventions? (2) What mitigating measures are recommended for ICED II to address and adapt to these factors? FINDINGS The team found two major factors emerging during the evaluation period: 1) Regulation 10/2017 to address Power Purchase Agreements (PPAs) 2) Regulation 12/2017 addressing Feed-in Tariffs Regulation 10/2017 does not apply to solar or wind, hydro below 10MWe or biogas and waste to energy power plants. Nor does it apply to existing PPAs or projects with a letter of intent to purchase from PLN. It changes the economics of IPPs in ways that are so far detrimental15. Earlier the PPA for dispatchable projects was based on availability with a two-part tariff: a capacity charge design to service debt equity and fixed operations and maintenance (O&M) costs16 and an energy charge to cover variable O&M costs based on actual dispatch. Thus, debt coverage was not dependent on PLN’s dispatch of power. Regulation 10 changes this so that PLN is required to take and pay for electricity for a “period of time” which is undefined and should be agreed between the parties. It also mentions considering the period of repayment to the IPP’s lenders. This last aspect can be interpreted two ways. First, it could mean that the period of time is that period consistent with a take or pay mechanism to allow recovery of debt and return on equity in the earlier, normal fashion. Or, second, it could mean that PLN Is not required to make take or pay payments once the project’s debt is repaid. PLN has thus far assumed this latter meaning. It is interpreted by PLN in such is way that repayment is dependent entirely on PLN’s dispatch instructions. This alone would significantly change the risk for future IPPs. Regulation 10 also stipulates that new IPPs be undertaken on a build-own-operate-transfer (BOOT) basis rather than the current build-own-operate (BOO) model. For developers that are dependent upon project finance not balance sheet finance, this is a major setback because it makes difficult to obtain funding on a project finance basis. Finally, Regulation 10 seems to suggest that PLN will be excused from Force Majeure issues and further shifts risk to developers and sponsors and it stipulates that the project sponsor can’t transfer share before commercial operations date. While this does not impact any of the projects being support by ICED II, it does influence the decision of developers and financial entities to continue future IPP activity. Most that have not signed PPAs indicate that future projects are on hold except for those associated with palm oil and geothermal energy. Regulation 12/2017 applies to: 15 The regulation introduces vague terms, which PLN has thus far defined in ways that reduce the return or increase the risk to IPPs. 16 Earlier PLN had take or pay provisions and was obligated to pay and provide debt coverage even if it did not take the power. 30 • Solar PV • Wind • Hydropower • Biomass • Biogas • Municipal waste • Geothermal Regulation 12/2017 regulates: • The price at which electricity generated from these renewable energy sources is to be sold to the Indonesian State-owned power utility, PLN. • The manner in which PLN is entitled to procure electricity supply from a number of these renewable sources. Regulation 12/2017 in essence shifts the FIT to be based on an avoidable cost. The tariffs on the above projects17 are capped at 85% of the local production cost where the local production cost is higher than the national average production cost. If the local production cost is the same or lower than the national average production cost, then the reference price will be 100% of the local production cost. This introduces geographic specific tariffs and favors the use of RE over conventional power in those areas that are not served by large thermal or large hydro plants. Regulation 50/2017 revokes 12/2017 and replaces it. The major changes are in the way in which PLN procures power, the tariff, and extension of BOOT to other IPPs. Solar PV and wind IPPs went from open tender to direct selection and from BOO to BOOT as did biomass and biogas. It changes the tariff for projects where the local cost of production is less than the national cost of production that were limited to 100% of the local production cost to direct negotiation with PLN. Only one of the project developers interviewed indicates that they have received a higher tariff on an existing project under Regulation 12/2017. All others, that did not have signed PPAs, indicated that they would suffer under this regulation and several indicated that this was their reason to put on hold current projects that had not signed a PPA. All project developers we interviewed indicated that the existing regulations overall provide a disincentive to undertaking new projects. CONCLUSIONS Two major factors emerged during the evaluation period that are negatively impacting investment in renewable energy and these come at the same time that the legislature cut subsidy payments to PLN18. These changes have not had an impact on projects that ICED II is assisting since ICED II made the decision to only support projects that had a signed PPA. However, it negatively impacts the entry of new projects into the ICED II pipeline since both financial entities and most developers indicate that they will not be investing in RE until there is a more favorable economic environment. It should also be noted 17 Except those projects where procurement is made using the direct selection method and geothermal projects. Geothermal will receive 100% of the avoided cost of power (ACP) in areas where the ACP is higher than the national average and determined by negotiation between PLN and the IPP. 18 While this does not directly impact RE, it reduces the incentive PLN has to negotiate power from high priced sources. 31 that there are still a large number of projects that can benefit from ICED II assistance that are in the pipeline. RECOMMENDATIONS What mitigating measures are recommended for ICED II address and adapt to these factors? Many of the project developers the team interviewed recognized that there was little that the project could do to influence policy but did wanted the US Government to stress the importance of a favorable environment to the GOI. ICED II early on took the steps that would shield it from the impact of these specific emerging factors by deciding to work only with late stage projects. Additionally, ICED II made the determination that it could not influence policy and so decided to work with policy that had already been established but not implemented and to develop the implementing rules, regulations, guidelines and procedures. The team recommends that ICED II stay the course, continue working with those late state projects that needs it help, and increase its efforts with the PLN in addressing standards, guidelines and business practices that remove barriers and reduce costs for renewable energy. The team also recommends that USAID continue its dialogue with the GOI to stress the incompatibility of these recent emerging factors, which are themselves GOI driven, and Indonesia’s stated goals of significantly increasing RE penetration. FINDING AND CONCLUSIONS FOR QUESTION 1F How has monitoring, learning and research been used to inform adaptive management in terms of adjusting ICED II’s technical approaches and intervention designs? FINDINGS The period in which ICED II has been operating has been subject to changes that had potentially negative impacts on ICED II. Examples include: (1) The two changes in regulation (EQ 1e), that reduce investment, (2) Government priorities shifted from implementation of energy efficiency to development of standards (EQ 1b), and (3) changes in the structure of how the GOI coordinates energy planning. In the RAD-GRK process, ICED had helped the Bappenas create a secretariat to coordinate the different government entities. It prepared plans for a similar secretariat when the GOI brought out the RUED￾RUEN planning process but it was rejected and the MEMR is the lead entity. The first two examples and how ICED II responded have been discussed above. The third example of adaptive management was discussed in ICED II annual reports and by KIIs. ICED II changed its approach by shifting resources from RAD-GRK to RUED-RUEN and, therefore, from Bappenas to MEMR and shifting resources in Bappenas from the Director of Environment to the Director of Energy in response to changing national priorities. In detailed discussions with ICED II staff, there is a conscious effort at work planning time to incorporate the lessons learned from the past, including ICED I, and input from stakeholders about their needs and the future of their respective spheres of influence and activity. For example, based on the past success of the RAD-GRK secretariat to promote interagency cooperation and coordination, ICED II proposed a similar arrangement to support the RUED-RUEN process. As events unfolded and it looked as though this approach would not be adopted, ICED II quickly adapted its approach and resource programming. 32 Another example is that of grid integration studies. Early on ICED II, seeing that the PLN’s rigid view on RE limits on the grids was an unnecessary obstacle to greater RE investment, sought to provide assistance to PLN to conduct these studies. PLN was not ready and ICED II did not drop the activity from its future plans. As more RE proposals came forward and developers were insistent that the grids could handle more RE, ICED II continued to share both its views and willingness to provide assistance to PLN. So, when PLN was ready to receive the assistance, so was ICED II ready to provide it. CONCLUSIONS ICED II’s project management remains flexible and proactive. Monitoring of the project’s performance together with a good understanding of stakeholders needs and directions is used both to measure past performance and inform future assistance. Thus, ICED II remains a highly effective project. FINDINGS AND CONCLUSIONS FOR QUESTION 1G (1) Has ICED II identified key factors and strategies to ensure sustainability and future scaling of program results? (2) If yes, has ICED II incorporated these aspects in the work plans? FINDINGS ICED II has put in place measures that will ensure the sustainability and future scaling19 of ICED II assistance in all but one area, the technical advice given to project developers. In component 1, ICED II’s plan for technical assistance delivery was designed to maximize the use of local resources, both personnel and institutions, so that the training and tools that ICED II provided would be taken up and be available to provide the GOI with future services. While GOI entities have training centers, specialized training such as the Long Range Energy Alternatives Planning (LEAP) model training is provided at the centers but drawing on local experts and local universities. “ICED II’s work will be sustained since RUED-RUEN is a Presidential decree”, meaning that ICED II laid the foundations, provided training and tools and now the end result, RUED-RUEN, must be conducted annually. Another KII said that ICED II’s work is sustained when it develops guidelines that are then adopted by the government such as the solar PV feasibility study guidelines. At the provincial level, one KII says this about sustainability and scaling, “Yes, ICED is making sure that enough people are trained in LEAP model and in RUED.” Two KIIs at Bappenas have completely different views: “ICED II has training of trainers on the modeling. Bappenas has a center of training and hires the university to conduct the training for Bappenas.” ICED II’s “support so far has been in capacity building, the knowledge is there – perhaps if Bappenas put this knowledge at the university level, this could be sustained – this has not happened yet – this is not the job of Bappenas (which is capacity building) – but ICED could help facilitate this and Bappenas would be happy to help with this.” 19 ICED II has put in place a number of measures that contribute to possible scaling in Component 2 but these rely on the Government not disadvantaging renewable energy. Since 2017, the GOI has created an unfavorable investment climate for most RE, as discussed above, and it is beyond the capability of ICED II assistance to obviate these measures. ICED II has put the measures in place and prior to 2017, these measures would have ensured scaling. 33 The team confirmed through discussion that Bappenas provides inhouse training for routine topics and goes outside for detailed training such as that provided by ICED II. ICED II has used the consultants that these outside institutions use to provide both technical assistance and training and in this way both builds current capacity and provides the support for sustainability. This second KII appears not to be aware of how extensive ICED II’s assistance has been. Supporting sustainability, another KII reports: “Every year, OJK meets with ICED program to determine priorities, good dialogue. Good relationship so far”. In component 2, ICED II has worked with three classes of stakeholders: financial entities, project developers, and the utility, PLN. To ensure sustainability and scaling in the financial area, ICED II worked with OJK, the financial services regulator, to provide training and guidelines in Sustainable Finance and Energy. ICED II supported OJK’s Sustainable Finance program and developed guidelines for the financial services industry to increase investments in clean energy projects. It further developed training modules in technologies hydro, solar PV, and biogas and worked with seven universities to train and house these modules in the university. According to OJK, these modules have since been adopted by three more training institutions. In further support of scaling, ICED II worked on new source financing “scaling up USAID and GOI credit guarantees, developing green bonds, and working to free up financing from non￾bank sources and worked with the International Finance Corporation (IFC) to initiate green bonds (a debt security to raise capital specifically to support climate-related or environmental projects) in Indonesia.”20 At the utility, ICED II works with PLN to increase the safe and reliable penetration of RE power on its grids in a variety of ways that ensure both sustainability and scaling. It is developing standards and guidelines which when adopted are both sustainable and lead to scaling. For example, ICED II worked with PLN and the MEMR DG Electricity to revise the South Sulawesi grid code in FY 2016. In FY 2017, it began updating “Guidelines for Interconnecting Renewable Energy Generating Plants to the 20 kV medium voltage network, thus allowing PLN Wilayah to better estimate the capacity that can be included in the next RUPTL as well as determine the feasibility of proposed small scale renewable energy projects (< 10MW)”21 and is finalizing the guidelines for Methodology for Power System Impact Studies of Interconnection of Wind and Solar Photovoltaic Power Plants. Second, ICED II provides training in these areas. ICED II provides technical assistance to project developers to facilitate their transaction and the KII’s indicated that this assistance was critical, well-received and high quality. Yet, the team found no evidence that ICED II had made any direct22 effort to ensure sustainability. ICED II staff explained that it has not found a mechanism to create the direct demand for these services or advice. CONCLUSIONS ICED II has made concrete efforts to sustain and scale its efforts and they appear to have taken hold with two exceptions. The first exception is in scaling with the financial and project developer community. In 2017, the GOI issued two regulations that significantly change the economics of 20 ICED II Annual Report FY 2017. 21 ICED II Annual Reports. 22 By direct effort, it is meant that ICED II has not made any efforts directly with the project developer community. 34 renewable energy and appear to have put future RE investment in many areas on hold. It is possible that ICED II’s efforts with financial entities and project developers will not be scaled until the economics is more favorable. This is well beyond ICED II’s control. The second exception is in efforts to develop sustainability mechanisms for the work it has done with project developers. Unlike its work in the power finance sector, where it worked directly with the community to develop mechanisms to sustain the tools, guidelines and advice it provided, it has not done so with project developers. However, ICED II’s work with PLN is an indirect method of sustaining the work it has done with project developers. By developing guidelines and tools such as for feasibility studies, grid integration, procurement and pricing, ICED II helps to ensure that these become standards for the project development community since PLN is the major purchaser of renewable energy. FINDINGS AND CONCLUSIONS FOR QUESTION 2 With the new U.S. administration, there has been an increased emphasis on USAID demonstrating how its interventions overseas are making a positive contribution to American economicprosperity, i.e. domestic US job creation, expanded market opportunities for US firms, etc. Though not a criterion against which ICED II’s performance would be evaluated, to what extent can evidence be collected that demonstrates the program’s contribution to either or both of these emerging areas of potential interest? FINDINGS While not directly charged with assisting US companies, ICED II has been able to assist a number of US companies as they explore the Indonesian market or deepen their penetration of the market. ICED II is part of the U.S. Power Working Group for Indonesia. The U.S. Embassy Jakarta has established the U.S. Power Working Group (PWG) as an American team program for firms supporting President Jokowi’s “35 GW” power development plan. The U.S. Power Working Group serves as a platform for American firms to dialogue with the Indonesian government on how they can contribute to Indonesia’s power infrastructure. Since the creation of the program, the group has expanded to approximately 60 leading U.S. energy firms. ICED II represents USAID by participating in a series of group-proposed activities aimed at introducing, supporting, and developing commercial projects, and providing constructive input into relevant legislation. U.S. Companies that ICED II has advised or assisted: • Allotrope Partners - Oakland, California • American Capital Energy and Infrastructure- Annapolis, Maryland • Black and Veatch - Oakland Park, Kansas • Caterpillar - Peoria, Illinois • Change Wind - Jewett City, Connecticut • Clark Technologies – Minneapolis, Minnesota • Cummins Power Generation - Minneapolis Minnesota • Dearstone Consulting – Anchorage, Alaska • Deloitte Consulting – Arlington, Virginia • EPS Capital – Villanova, Pennsylvania • First Solar - Tempe, Arizona 35 • Fluidic Energy - Scottsdale, Arizona • GE Power – San Ramon, California • Island energy Resources – Honolulu, Hawaii • Natel Energy - Alameda, California • Ormat Technologies - Reno, Nevada • Pacific Private Equity Group – San Ramon, California • Skychaser Energy – Petaluma, California • Tesla - Palo Alto, California • The Innovation Network – Lexington, Massachusetts • UL – Northbrook, Illinois • UPC Renewables - Boulder, Colorado • Wellons - Vancouver, Washington • Winrock International - Boulder, Colorado U.S. Small Business Subcontractors • IWE – Jacksonville, Florida (subcontractor) • H&M Engineering – Ellicott City, Maryland (subcontractor) • BBI International - Grand Forks, North Dakota (subcontractor) U.S. Utility Partners • Hawaii Electric Light Company – Hilo, Hawaii • Pacific Gas and Electric – San Francisco, California • Sacramento Municipal Utility District – Sacramento, California • California Independent System Operator - Folsom, California Beyond the information from ICED II, the evaluation asked project developers and financial entities if they purchased or plan to purchase any US equipment or services on their projects. None reported using or planning to use US equipment or services. When asked if they knew of any US small turbine manufacturers, only one reported that he knew GE provided this equipment. When asked if they knew of any US equipment being used on similar size projects in the country, none reported knowing of any US equipment being deployed in Indonesia in similar size projects. Discussions with the ICED II team indicate that they are helping PLN move in an important area, hybrid grids, where US companies do have a competitive advantage. Moreover, ICED II plans on collaborating with other US Government entities to provide solutions that can provide openings to US companies. CONCLUSIONS ICED II is not designed to directly promote US companies but it has been heavily involved in assisting US companies. ICED II can’t create markets nor can it create the economic conditions that would give US companies a competitive edge. ICED II can, when it sees the right opening, use the collective tools of the US Government to foster solutions that US companies can take advantage of. Whether or not US companies take advantage of these openings is beyond the remit of ICED II. 36 ANNEX 1 EVALUATION STATEMENT OF WORK I. PURPOSE OF THEEVALUATION The purpose of this Midterm Performance Evaluation (henceforth, “the evaluation”) of the second iteration of the Indonesia Clean Energy Development Program (ICED II) is to provide USAID/Indonesia with an independent review of progress made by the Mission’s largest energy sector activity after its first two years of implementation. The evaluation is intended to measure the effectiveness and the impact of the program implementation by Tetra Tech ES, as the prime contractor, against the performance indicators and high level target results as set forth in the Contract. It is expected that the findings from the evaluation will assist USAID/Indonesia to make informed decisions regarding which aspects of the program have been most effective in advancing the goal of ICED II to strengthen the foundation for a low-carbon energy system in Indonesia. In addition, it is also expected to provide a recommendation for USAID/Indonesia in what manner/to what extent resources could be reallocated to achieve more significant results in consideration of accomplishments made thus far, and external factors that may be influencing the program’s implementation Beyond USAID/Indonesia, key stakeholders of this evaluation include representatives of the Government of Indonesia (GOI) – primarily those housed within the Ministry of Energy and Mineral Resources (MEMR), the National Development Planning Agency (Badan Perencanaan Pembangunan Nasional – BAPPENAS), and the Financial Services Authority (Otoritas Jasa Keuangan – OJK); the National Electricity Utility (Perusahaan Listrik Negara – PLN); sub-national government offices in ICED II’s area of implementation; international donors and multilateral institutions that are also working within Indonesia’s energy sector; and private sector entities, including financial services institutions and renewable energy project developers. The ICED II implementer, Tetra Tech ES, Inc., will be responsible for operationalizing any of the evaluators’ recommendations should USAID/Indonesia deem them appropriate/necessary and within the scope of the existing contract. If changes to ICED II’s expected results and/or performance indicator targets are required, those changes will be discussed with Tetra Tech, and may involve modifications to the contract. Recommendations may also lead to requested changes in implementation approaches, resource allocations, personnel structure, or other adaptive actions to lead to programmatic success across components. This evaluation will focus on ICED II’s national activities conducted between May 2015 and May 2017 with central GOI offices, PLN, and financial institutions, as well as on sub-national activities conducted within the program’s primary target provinces, including Aceh, North Sumatera, South Sulawesi, East Java and West Nusa Tenggara (NTB). 37 II. SUMMARY OF PROGRAM INFORMATION III. BACKGROUND ICED II is a five-year (May 2015 – May 2020), $17.18 million program implemented by Tetra Tech ES, with a consortium of partners/subcontractors that includes BBI International, Inc., Bradford Brook Associates, H&M Engineering, IWE, Nexant, PT Castlerock, GreenMax Sustainability and Finance, and PLN Enjiniring. The program builds on the achievements made during the Mission’s first iteration of ICED, which ran from February 2011 – February 2015 (also implemented by Tetra Tech). The overall goal of ICED II is to strengthen the foundation for a low-carbon energy system in Indonesia. This goal is the same as the Mission’s Sub-Intermediate Result (Sub-IR) 3.3.1 of the USAID/Indonesia 2014-2018 Country Development Cooperation Strategy (CDCS)23, and it contributes to the CDCS Intermediate Result (IR) 3.3: ‘Climate Change Mitigation and Resilience to Support a Green Economy Strengthened,’ and the 23. Investing in Indonesia: A stronger Indonesia advancing national and global development. USAID Strategy for Indonesia 2014 – 2018. https://www.usaid.gov/sites/default/files/documents/1861/Indonesia CDCS FINAL Version.pdf Program Name Indonesia Clean Energy Development Program II (ICED II) Implementing Partner Tetra Tech ES, Inc. Contract No. Task Order No. Contract No. AID-OAA-1-13-000019 Task Order No. AID-497-TO-15-00002 Total Estimated Ceiling of the Evaluated Program $17,180,713. Life of Strategy, Project, or Activity May 2015 – May 2020 Active Geographic Regions National support at the Central Level, and within five Target Provinces: Aceh, East Java, North Sumatra, Nusa Tenggara Barat (NTB) and South Sulawesi. Contributions to Mission Development Objective(s) (DOs) and Intermediate Result(s) (IRs) From USAID/Indonesia Country Development Cooperation Strategy (CDCS) 2014 - 2018 DO 3: Global Development Priorities of Mutual Interest Advanced; IR 3.3: Climate Change Mitigation and Resilience to Support a Green Economy Strengthened; Sub-IR 3.3.1: Foundation for Low Carbon Energy System Strengthened DO 4: Collaborative Achievement in Science, Technology, and Innovation Increased USAID Office USAID/Indonesia Environment Office (ENV) 38 corresponding CDCS Development Objective (DO) 3: ‘Global Development Priorities of Mutual Interest Advanced.’ In recognition of the potential for program interventions to support technological and human capacity advancements in the clean energy space, ICED II also complements the work of the DO 4: ‘Collaborative Achievement in Science, Technology and Innovation Increased.’ To achieve the goal, ICED II was designed in line with the USG’s Enhancing Capacity for Low Emissions Development Strategies (EC-LEDS) initiative, launched in 2011. The EC-LEDS program was launched in 20 countries worldwide, with Indonesia as one of the key partners, and focused primarily in two aspects: (1) Low Emissions Planning, and (2) Low Emissions Actions/Implementation. USAID/Indonesia’s clean energy programs are major part of EC-LEDS initiative in Indonesia and are expected to directly contribute to successful implementation of low emission development in the Country. Under this design framework, ICED II works in collaboration with the GOI and supports its energy sector objectives, contributing to national electricity generation and access targets and efforts to reduce sector-specific greenhouse gas (GHG) emissions. ICED II aims to assist the GOI in establishing an effective policy, regulatory and incentive environment for low-emission growth, while simultaneously attracting public- and private- sector investment in clean energy development. As such, ICED II’s programmatic activities are framed within three interrelated technical components: (i) Improve the enabling environment for rapid co-investment in clean energy; (ii) Accelerate the mobilization of private and public sector co-investment in clean energy; and (iii) Outreach on USAID and US Government (USG) inter-agency activities in Indonesia’s energy sector. Within these technical components, ICED II has developed associated Tasks and Work Streams that guide the design and implementation of its specific interventions (presented in Annex A). ICED II also operates a Clean Energy Technology and Innovation Fund that is used to augment project interventions. The purpose of the Innovation Fund is to be a flexible means, where necessary and/or useful to the project’s objectives, to encourage participation and/or collaboration of private sector and local entities toward the achievement of ICED II’s components and tasks. A. Description of the Problem, Development Hypothesis, and Technical Approach During the design stage of ICED II, the GOI estimated that national electricity demand would increase six times over the next two decades. To meet this demand, it was approximated that 230 GW of additional generation capacity would need to be integrated into the national energy system – effectively increasing the national installed capacity from less than 50 GW to 320 GW. USAID/Indonesia recognized that in the absence of a concerted effort to improve the foundation for a low emission energy system, it would be likely that a heavier reliance on coal-fueled power generation would meet the electricity demand of the rapidly growing economy, while remote, underserved populations would continue to rely on expensive and dirty diesel-fueled power. As such, the Mission developed the ICED II technical assistance program to engage with a range of public and private sector partners to implement policies, programs, and projects that would make measurable contributions to increasing the portion of Indonesia’s energy mix that is derived from renewable sources. Indonesia has a wealth of renewable energy resources – ranging from hydro, solar, wind, biomass, and geothermal power. However, as of June 2016, only 6.3 percent of Indonesia’s 55.4 GW installed generation was from renewables. Since the inception of ICED II, a number of new measures and commitments have been made by the GOI in regards to increasing the use of renewable energy, 39 expanding energy access, and reducing GHG emissions. In accordance with the 2014 National Energy Policy (Kebijakan Energi Nasional – KEN), Indonesia aims to have 23 percent of its energy demand supplied by new and renewable sources by 2025; The country aims to increase energy access from 85 percent to 100 percent by 2020; and in line with its Nationally Determined Contributions (NDC) reaffirmed at the 2015 United Nations Climate Change Conference of Parties (COP 21), aims to reduce its GHG emissions by 29 percent by 2030 (when compared to a business as usual scenario). Indonesia has stated that that this reduction could be as high as 41 percent with international donor support. However, in spite of these targets, Indonesia has not in recent years made significant notable steps in bringing large amounts of renewable energy generation facilities online – particularly when compared to many other nations in the region (i.e. Malaysia, Thailand, and the Philippines). This has largely been attributed to the persistence of an under-supportive policy environment, exacerbated by government administration changes, shifts in planning priorities and incentive regimes, and a growing disconnect between the MEMR and PLN. Consequently, a number of renewable energy projects - of various sizes and technologies – have stalled in their planning stage, and have been unable to secure power purchase agreements (PPAs) with PLN. The development hypothesis of ICED II is that by reinforcing assistance to national and local governmental institutions, the program improves clean energy and climate change mitigation programs. The establishment, or refinements, of a law, policy, strategy or regulation is the foundation of the program. ICED II trains individuals in critical functional areas associated with implementing the program. Coupled with guidelines, tools, and standard operating procedures, this leads to improved institutional capacity. Conversely, improved institutional capacity will result in internal training of new or transferred managers and key staff, that should, over the longer term, result the design and implementation of more effective programs. Specific ICED II activities and accomplishments are documented in the program’s work plans and quarterly/annual reports – which are referenced in Annex C, and available online. ICED II recognizes that the implementation of clean energy and climate change mitigation programs requires public and private sector investment in renewable energy and energy efficiency projects. ICED II assistance to project developers, banks and financial institutions, and consultants and engineering companies, is designed to improve the viability of clean energy projects. Once operating, the electricity generated or energy saved results in a reduction of GHG emissions. The additional generation also results in improved energy services to new or existing consumers. As a final contextual matter, it is important to note that with the new U.S. presidential administration there remains some uncertainty in regards to possible refinements to the USG’s foreign assistance priorities. By this evaluation’s completion, there will likely be further guidance available regarding the expected impact of USAID’s programming worldwide. However, in the interim there have already been requests from the new administration for all USG agencies to demonstrate how the federal funding they receive will have a positive impact on American economic prosperity (i.e. domestic US job creation, expanded market opportunities for US firms, etc.) and/or enhancing international security. Though recognizing that ICED II was not designed with these specific priorities in mind, the evaluation will serve the additional purpose of collecting evidence to demonstrate the extent in which ICED II interventions are complementary to this effort. B. Summary of the Project/Activity Monitoring and Evaluation (M&E) Plan 40 In March 2017, USAID/Indonesia approved the revised ICED II Monitoring and Evaluation Plan (M&E Plan), which describes the continuous process that Tetra Tech will use for performance monitoring and management. The M&E Plan is designed to demonstrate how it supports: (i) Alignment of ICED II with GOI related targets and associated monitoring, reporting, and verification (MRV) protocols; (ii) Effective adaptive management decision-making based on practical experience (lessons learned) in implementing the M&E Plan; (iii) Accountability and cost-effectiveness of delivering ICED II resources; and (iv) Provision of information on implementation. Included in the M&E Plan is the ICED II Results Framework (Annex B), which Tetra Tech says illustrates the cause-and-effect linkages between the program’s goal and relevant USAID/Indonesia Development Objectives (DO) and Intermediate Results (IR). The ICED II Results Framework also includes five high-level results the program is expected to achieve by 2020 by the Contract. Additionally, for each of the three technical components, ICED II has 15 “minimum results” which it will be accountable for achieving. On a quarterly basis, ICED II reports on progress made against 11 performance indicators – five of which are directly aligned with the program’s five high- level results. All of these indicators are derived from USAID’s Global Climate Change (GCC) performance indicators. The M&E Plan outlines how data is collected and stored, explains in what manner results achieved will be attributed to ICED II, and includes Performance Indicator References Sheets (PIRS) which describes how each of the performance indicators (and by extension, the high-level results) are defined and measured. Though some of the technical-component (“minimum”) results are directly aligned with individual performance indicators (i.e. “Facilitated installation of at least 400MW￾equivalent of clean energy generation capacity”), not all are. ICED II - HIGH-LEVEL PROGRAM RESULTS • At least 4.5 million tons of GHG emissions, estimated in metric tons of CO2e, reduced, sequestered, and/or avoided as a result of US Government (USG) assistance • At least US $800 million of investment mobilized from private and public sources for climate change mitigation as supported by USG assistance • At least an additional 5 million people with access to clean energy • At least 20 institutions with improved capacity to address climate change issues as a result of USG assistance • At least 20 laws, policies, strategies, plans, or regulations addressing climate changemitigation officially proposed, adopted, or implemented as a result of USG assistance. ICED II - TECHNICAL COMPONENT-LEVEL (“MINIMUM”) RESULTS24 Component 1: Improve Enabling Environment for Rapid Co-Investment in Clean Energy • Enhanced integration of Regional Action Plan for Greenhouse Gas Emissions Reduction (RAD-GRK) with the National Action Plan for Greenhouse Gas Emissions Reduction (RAN-GRK), 24 Those Technical Component-Level (“Minimum”) Results demarked with a “*” correspond directly with one of the ICED performance indicators and associated targets. 41 National/Regional General Plans for Energy (RUEN/RUED), Regional General Plans for Electricity (RUKD), the Regional Medium-Term Development Plans (RPJMD), Provincial/District/City Regional Spatial Management Plan (RTRWP/K), and associated strategic planning. • Preliminary RUEDs and RUKDs are developed that reflect integrated resource planning, responsive to energy demands for the region, and aligned with the respective RAD-GRK. • Improved capacity of institutions at local level to operationalize and refine RAD-GRK plans based on sound quantitative data and analysis and synchronization of methodologies at national and local level; and to develop, operationalize and align RUEDs and RUKDs with local development priorities. • Enhanced participation of district and municipal institutions in RAD-GRK, RUED and RUKD planning and implementation process. • Streamlined reform of existing policies and regulations relevant to the clean energy growth. • Public outreach strategies developed at the national/subnational level to effectively communicate the impact of policy/regulatory reform. • Facilitated installation of at least 400MW-equivalent of clean energy generation capacity.* • Adoption of standard business practices for clean energy development, integration and related procedures with PLN. • Piloting of at least five renewable energy/energy conservation innovations with a roadmap for scaling￾up and/or rapid replication. Component 2: Accelerate Mobilization of Private and Public Sector Co-Investment in Clean Energy • Adoption of effective fiscal and non-fiscal incentives for increased investment in different types of small to medium scale renewable energy technologies. • Establishment of regulatory framework and standardized feed-in tariffs (FITs) for the development of renewable energy projects with generating capacity larger than 10 MW. • Established funding and implementation frameworks for mitigation actions in the clean energy sector at the subnational level. Component 3: Outreach on USAID and USG Inter-Agency Activities in Indonesia’s Energy Sector • Improved business practices of project design and feasibility assessment in the clean energy market to reduce risk and increase likelihood for attracting investment. • Documented assessment of human resources needs of relevant stakeholders and action-plan for designing and delivering associated technical assistance to enhance the capacity of local organizations in providing relevant services and trainings in clean energy aspects. 42 • Pilot modules for study and practical training in at least five institutions for at least five different technologies. PERFORMANCE INDICATORS25 • Greenhouse gas emissions, estimated in metric tons of CO2e, reduced, and/or avoided through clean energy activities as supported by USG assistance (LOP Target: 4.5 MT)* • Amount of investment mobilized (in USD) for clean energy as supported by USG assistance (LOP Target: $800 million)* • Number of beneficiaries with improved energy services due to USG assistance (LOP Target: 5 million)* • Number of institutions with improved capacity to address clean energy issues as supported by USG assistance (LOP Target: 20 institutions)* • Number of laws, policies, regulations, or standards addressing clean energy formally proposed, adopted, or implemented as supported by USG assistance (LOP Target: 20 laws, policies, regulations or standards)* • Clean energy generation capacity installed or rehabilitated as a result of USG assistance (LOP Target: 400 MW) • Proportion of females who report increased self-efficacy at the conclusion of USG supported training/programming (LOP Target: 70 percent) • Clean energy generation capacity supported by USG assistance that has achieved financial closure (LOP Target: 400 MW) • Projected greenhouse gas emissions reduced or avoided through 2030 from adopted laws, policies, regulations or technologies related to clean energy as supported by USG assistance (LOP Target: 27 million MT) • Number of people trained in global clean energy supported by USG assistance (LOP Target: 5,000) • Expected lifetime energy savings from energy efficiency or energy conservation, as a result of USG assistance (LOP Target: 2, 800). IV. EVALUATION QUESTIONS ANDDESIGN The ICED II mid-term performance evaluation is framed around two primary evaluation questions. The evaluators will thoroughly integrate each of these questions and associated sub-questions within the design of the evaluation and their methodology. When feasible and practical, the evaluation team will 25 Those Performance Indicators demarked with a “*” are the five that are directly aligned with ICED II’s high-level results. 43 assess gender equity and women’s empowerment aspects of each evaluation question. Some of the questions have a clearer and more relevant gender dimension than others. EVALUATION QUESTIONS 1) To what extent is ICED II on track to meet its expected high-level and technical component-level (“Minimum”) results, and in what manner could either implementation or monitoring efforts be enhanced? a) Do the goal, results, indicators, and targets selected for ICED II appropriately and completely track all program activities, and accurate tools for monitoring and reporting on implementation? Is there a need to refine, add, omit and/or more clearly define the selected results, indicators, and targets? b) Are there any activities in which ICED II engaged that are making limited contribution to the expected results and/or overarching goal of the program? If so, to what extent and what is the rationale for continued engagement? c) Are there any activities in which ICED II is engaged that show proportionally high contributions to the expected results in consideration of resources allocated? If so, to what extent have efforts been made to build on these activities to intensify their contribution to ICED II’s goal. d) How do the implementation approaches and management of CED program affect the ability to achieve expected results (outputs, outcomes and strategic impacts) effectively, and what constraints are limiting its effectiveness? In addressing this question, the following aspects must be considered: project management approach, relationship with stakeholders, human resources availability, geographic focus, and local versus national stakeholder engagement. e) What are emerging external factors (locally, nationally, regionally and globally) that areaffecting/have the potential to affect the implementation or effectiveness of ICED II’s interventions? What mitigating measures are recommended for ICED II address and adapt to these factors? Note: a Risk Analysis Framework should be used to identify external and internal risks and corresponding recommendations. f) How has monitoring, learning and research been used to inform adaptive management in terms of adjusting ICED II’s technical approaches and intervention designs? g) Has ICED II identified key factors and strategies to ensure sustainability and future scaling of program results? If yes, has ICED II incorporated these aspects in the work plans? 2) With the new U.S. administration, there has been an increased emphasis on USAID demonstrating how its interventions overseas are making a positive contribution to American economic prosperity, i.e. domestic US job creation, expanded market opportunities for US firms, etc. Though not a criterion to which ICED II’s performance will be evaluated, to what extent can evidence be collected that demonstrates the program’s contribution to either or both of these emerging areas of potential interest? 44 EVALUATION DESIGN AND METHODOLOGY The evaluation team will be expected to develop a detailed design and methodology based on their expertise, understanding of the evaluation requirements, and available program documentation. USAID/Indonesia recognizes that limitation on data quality and availability; multiple sets of public- and private-sector stakeholders, often with differing objectives; logistical complications associated with field visits; and other external factors may affect the ability to conduct a comprehensive performance evaluation within the time and budget available. In this section, some general guidelines are provided on methodological aspects of the evaluation. The evaluation team will present its recommendations on how to balance these factors in developing its approach and methodology. The overall design will rely on mixed methods for data gathering and analysis. Quantitative data from ICED II reports and other sources will be combined with qualitative data, gathered through the evaluation team’s primary research and discussions with stakeholders. Each evaluation question is likely to be best addressed through a combination of data collection from both internal (USAID/Washington, USAID/Indonesia, and Tetra Tech) and external (all others sources), of quantitative or qualitative information, and of methods for gathering and analyzing that information. Tools used to gather information will also vary, depending on the evaluation question. Collecting primary quantitative data will be impractical, but otherwise the evaluation team may recommend a combination of literature review (including, but not limited to: external reports, Indonesian energy sector analyses, relevant scientific literature, and credible news articles), review and verification of ICED II monitoring data, information gleaned from key informant and focus group interviews, or other data gathering methodologies. Effort should be made to obtain diverse views and triangulate key findings across different stakeholder groups. The team will propose data collection and analysis methods that will be reviewed and approved as part of the design deliverable. The team will be expected to analyze and synthesize the information rather than merely summarize it. That is, USAID/Indonesia will be less interested in recitation of interview responses than in an analysis of what those responses mean in the context of addressing the evaluation questions and defensible recommendations. USAID/Indonesia also expects to receive an indication of the quality of evidence associated with each finding and recommendation. To the greatest extent possible within available time and budget, the evaluation team will make field site visits to at least one of three provinces where ICED II has been mostly active in the past two years, i.e. North Sumatera, South Sulawesi and West Nusa Tenggara (NTB). The Evaluation Design will include a recommended set of sites to visit; indicative timing of visits; personnel to be involved from the evaluation team, ICED II team, USAID/Indonesia, and other stakeholders; and other relevant details on field site data gathering (e.g., roundtables, key informant interviews, focus group discussions, etc.). It may not be necessary, efficient or productive to conduct site visits in all focus provinces where ICED II operates. V. DELIVERABLES AND REPORTINGREQUIREMENTS The following deliverables will be prepared and submitted by the evaluation team at different stages of conducting the ICED II performance evaluation. Changes to the deliverables – including timing, content, approvals, etc. – must be agreed in advance by USAID/Indonesia before they are implemented. The 45 actual submission date for the following deliverables and reporting will be determined in the work plan and must be approved by the COR. 1. Evaluation Design and Workplan: The evaluation team must submit to USAID/Indonesia a draft evaluation design (which will become an annex to the Evaluation Report) of no more than 30 pages. The Evaluation Design will include: (i) A detailed evaluation design matrix (sample attached as Annex C) that links the evaluation questions to data sources, methods, and the data analysis plan; (ii) Draft data collection instruments or their main features; (iii) A list of potential interviewees and sites to be visited and proposed selection criteria; (iv) Assumptions and known limitations to the evaluation design; and (v) A dissemination plan for sharing the findings/recommendations of the Evaluation Report. The evaluation design will also include a brief discussion of how the evaluation team plans to mitigate potential conflicts of interest between the evaluation contractor and the ICED II prime contractor, given their shared corporate ownership. The draft work plan will be 10 pages or less and include: (i) The anticipated schedule and logistical arrangements necessary to conduct the evaluation; (ii) A list of the members of the evaluation team, delineated by roles and responsibilities; and (iii) A preliminary indicative timeline for each deliverable, factoring in review periods. USAID/Indonesia will provide comments, request changes, or approve the Evaluation Design and Workplan – as appropriate. 2. In-briefing / inception report: Within three business days of arrival in Indonesia, the evaluation team will have an in-briefing with the USAID/Indonesia Environment Office and Program Office. This in￾briefing will serve introductions and to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and workplan, and/or to adjust the Statement of Work (SOW), if necessary. At USAID/Indonesia’s discretion, the in-briefing may also engage representatives from the GOI who are directly engaged in the implementation of ICED II. The evaluation team’s meeting notes will serve as an Inception Report deliverable. 3. Interim Briefings: The team will provide USAID/Indonesia with periodic briefings on the team’s findings, as agreed upon during the in-briefing. If desired or necessary, weekly briefings by phone can be arranged. 4. Exit Briefing / Presentation: The evaluation team will participate in an exit briefing prior to leaving Indonesia to discuss the status of data collection, analysis, and preliminary summary of findings. It is understood that recommendations would be premature until analyses are completed; however, if any “red flag” issues have emerged during the evaluation, those should be discussed at the exit briefing so that USAID/Indonesia may take more immediate corrective action as appropriate. The presentation portion of the exit briefing will consist of an electronic deliverable in PowerPoint or other agreed format, and given in a level of detail appropriate for the audience. Details for the exit briefing and presentation will be agreed upon during the in-briefing, consisting of approximate schedule, location, expected maximum length, preliminary list of attendees (e.g., whether or not to invite ICED II staff, GOI officials, or other stakeholders), and other details as necessary. These details may need to be adjusted if specific sensitivities arise during the evaluation, and will be finalized no later than five business days before the exit briefing and presentation is delivered. 46 5. Draft Evaluation Report: The draft evaluation report should be consistent with the guidance provided below pertaining to the final report format. The report will address in detail the findings associated with each of the evaluation questions presented in this SOW, and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation report will be determined in the evaluation work plan. [A good practice is for the evaluation team to share an early draft or detailed outline that includes main findings and bullets before completing the draft evaluation report.] Once the initial draft evaluation report is submitted, USAID/Indonesia will have 10 business days in which to review and comment on the initial draft, after which point consolidated comments will be provided to the evaluation team. The evaluation team will then submit a revised final draft report not more than 10 business days after receiving the Mission’s comments, and USAID/Indonesia again will review and send comments within 15 business days of its submission. 6. Socialization / Dissemination of Recommendations: After USAID review of the draft evaluation report, the evaluation team will organize and convene one or more roundtable discussion with stakeholders at USAID/Indonesia and possibly with contractor homeoffice staff. 7. Final Evaluation Report: The evaluation team will take no more than 10 business days to respond to and incorporate final comments from USAID/Indonesia. The evaluation team leader will then submit the final report to the COR. All project data and records will be submitted in full and should be in electronic form in easily readable format, organized and documented for use by those not fully familiar with the intervention or evaluation, and owned by USAID. VI. EVALUATION TEAM COMPISITION The Evaluation team is to be comprised of experts/evaluators who are independent of the current ICED II program implementers and the implementers’ staffs. The Evaluation team should have technical expertise in clean energy, particularly in energy policies, planning, renewable energy, energy efficiency, and project financing. The team should comprise not more than four individuals including as many as three local evaluators. All should be proficient in English and have excellent analytic, writing and communication skills. • Team Leader/Evaluation Specialist. At least fifteen years’ professional experience with significant experience evaluating energy sector programs in developing countries. The Team Leader should have strong managerial and operational experience with complex, national and sub-national activities involving coordination of multiple program partner institutions; subject matter expertise in clean energy, environmental policy and institutional development, global climate change and low carbon development, policy and market constraints, and effective approaches; and strong task output leadership and communications skills. Familiarity with the political, social and cultural context of Indonesia and Indonesian language is a strong plus. The Team Leader will focus on leading the team for the evaluation. In addition to being responsible for preparation and submission of all deliverables, the Team Leader will be tasked with synthesizing the findings and recommendations of the team. • Deputy Team Leader/Institutional and Human Capacity Development Specialist. Masters level with at least ten years’ professional experience, ideally in the energy sector. Will also play major role in organizing, editing and producing the evaluation report. Must have excellent writing skills. 47 • Renewable Energy Institutional/Policy Specialist. Master level with at least seven years’ professional experience in research, evaluation or program management of energy policy, planning or relevant projects. Practical experience in project implementation or evaluation, knowledge of the landscape of energy policies and enabling environment in Indonesia, focus on approaches for climate change mitigation and low emission development are a must. • Renewable Energy Project Development and Financing Specialist. Masters level preferably in engineering field or renewable energy integration with at least seven years’ professional experience in implementation and or evaluation of approaches for renewable energy and energy efficiency project development and project financing transactions. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. The evaluation team shall demonstrate familiarity with USAID’s evaluation policies and guidance included in the USAID Automated Directive System (ADS) in Chapter 200. VII. FINAL REPORT FORMAT The evaluation final report should include an abstract; executive summary; background of the local context and the strategies/projects/activities being evaluated; the evaluation purpose and main evaluation questions; the evaluation design/methodology; the assumptions/limitations to the evaluation; findings, conclusions, and recommendations. For more detail, see “How-To Note: Preparing Evaluation Reports” and ADS 201mah, USAID Evaluation Report Requirements. An optional evaluation report template is available in the Evaluation Toolkit. An abstract of not more than 250 words will briefly describe what was evaluated, evaluation questions, methods, and key findings or conclusions. The abstract should appear on its own page immediately after the reports cover. The executive summary should be 2–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation design and methodology shall be explained in the report in detail or summarized in the report and detailed in an Annex. Assumptions and limitations to the evaluation shall be disclosed in the report, with particular attention to any constraints associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.). The annexes to the report shall include: • The Evaluation SOW – attaching a revised version if applicable and based on COR agreement; • Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; 48 • All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; • All sources of information, properly identified26 and listed; • Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of interest; and • Summary information about evaluation team members, including qualifications, experience, and role on the team. In accordance with ADS 201, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within three months of the evaluation’s conclusion. VIII. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluation against the following criteria to ensure the quality of the evaluation report.5 • Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. • Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. • The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. • Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. • Evaluation methodology should be explained in detail and sources of information properly identified. • Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. • Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. 26 APA format preferred but not required; other acceptable formats are Council of Science Editors or Chicago Manual of Style 27 See ADS 201mah, USAID Evaluation Report Requirements and the Evaluation Report Review Checklist from the Evaluation Toolkit for additional guidance. 49 • If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. • If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. IX. OTHE REQUIREMENTS All quantitative data collected by the evaluation team must be provided in machine-readable, non￾proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the SOW of the contract/agreement, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the COR. Any revisions should be updated in the SOW that is included as an annex to the Evaluation Report. 50 ANNEX 2 REFERENCES ADB. 2016. Energy Sector Assessment, Strategy, And Road Map, Manila. Allen and Overy. 2017. Indonesia power sector: New regulation on tariff and tendering for renewable energy projects. http://www.allenovery.com/publications/en-gb/Pages/Indonesia-power-sector-New-regulation-on-tariff￾and-tendering-for-renewable-energy-projects.aspx Ecofys. 2016. The impact of risks in renewable energy investments and the role of smart policies ICED II. 2017. Various. http://www.iced.or.id. IRENA.2017. Indonesian Renewable Energy Policy and Investment Opportunities: Indonesia’s New Regulation and its Impact on Renewables OJK. 2017. Renewable Energy for Indonesia, 2017. http://www.ojk.go.id/sustainable-finance/id/publikasi/riset-dan￾statistik/Pages/RENEWABLE-ENERGY-FOR-INDONESIA-2017.aspx USAID. 2017. Climate Risk Management for USAID Projects and Activities USAID. 2016. GCC Standard Indicator Handbook USAID. 2013. Indonesia Energy Sector and Strategic Program Assessment USAID. ICED I Final Report USAID. 2015. ICED I Project End Results and Achievements USAID. 2015. ICED II FY 2015 Annual Report USAID. 2016. ICED II FY 2016 Annual Report USAID. 2017. ICED II FY 2017 Annual Report USAID. 2015. ICED II M&E Plan USAID. 2016. ICED II Work Plan Fiscal Year 3 USAID. 2015. ICED II Work Plan Fiscal Year 2 USAID. 2015. ICED II Work Plan Fiscal Year 1 World Wildlife Fund. 2014. Sustainable Energy Finance in Indonesia. 51 ANNEX 3 KII PROTOCOLS STAKEHOLDER GROUP: USAID/INDONESIA INTERVIEW TRACKING DATA This section will be completed by the interviewer prior to the key informant interview Date of Interview Location of Interview Name of Data Collector Name of Respondent Role or Position/Title of Respondent Male/Female/Non-binary Phone number/Email of Respondent INTRODUCTION Read the consent statement [This is a statement that tells them what we are doing and why and then asks for their consent to conduct the interview], and offer respondents the opportunity to ask questions. Have copies of the statement available in case the respondent prefers to read it. Once they have provided consent, proceed with the interview. INTERVIEW QUESTIONS Ask the interview questions below. Points to probe throughout the interview are shown in brackets [ ]. Question 1. Please tell me about your role as it relates to the ICED II Project. a. When and how did you start working with the ICED II Project? b. Did your interactions of involvement change over time? How so? 2. [Before asking this question, show the activities which ICED II shows in the work plan and monitoring plan.] Are there any activities, which ICED is doing that are not listed here? If yes, then what are they? 3. Are any of these activities working particularly well in terms of reaching the expected results? If so, why? 4. Are any of these activities making limited contribution to the expected results and/or overarching goal of the program? If so, why? If so, then why continue to do them? 5. In your own words, what were the key successes so far? 52 Question 6. Have implementation approaches and management of the ICED II program affected the ability to achieve expected results (outputs, outcomes and strategic impacts) effectively? If yes, please explain. In addressing this question, the following aspects must be considered: project management approach, relationship with stakeholders, human resources availability, geographic focus, and local versus national stakeholder engagement. 7. Are there constraints that are limiting its effectiveness? If yes, please explain. 8. In your own words, what have been the most meaningful or significant achievements of the ICED II Project? [PROBE: why?] 9. To what extent, if any, do you think any of the ICED II results will be sustained after the Project ends? [PROBE: Why?] 10. In your own words, could you describe how your component contributed to the ICED II? 11. [Before asking this question, show the Risk Framework Diagram.] Are there any emerging external factors (locally, nationally, regionally and globally) that are affecting/have the potential to affect the implementation or effectiveness of ICED II’s interventions? What are they and how are they or will they impact ICED II type interventions? [If asked for example, give recent FIT change] 12. In response to these factors you have identified, has ICED II recommended mitigating measures to address and adapt to these factors? 13. Has monitoring, learning and research been used to inform adaptive management in terms of adjusting ICED II’s technical approaches and intervention designs? 14. If so, how has this been done? 15. Has ICED II identified key factors and strategies to ensure sustainability and future scaling of program results? 16. If yes, has ICED II incorporated these aspects in the work plans? 17. We realize that ICED II was not setup to promote US exports or US jobs. However, this is often an outcome of US foreign assistance. In clean energy it often takes the form of purchases of US equipment, US expertise in project engineering or financial advice or even financing. With that in mind, are you aware of any evidence that demonstrates the program’s contribution to domestic US job creation and/or market opportunities for US firms? 18. What developers, banks or equipment manufacturers should we talk to? 19. Do any have offices that you know of in Indonesia? 20. Looking back on ICED II’s performance, is there anything you recommend to make it better? 21. Why? 22. Is there activity you would have dropped? 23. Why? CONCLUSION Before concluding the interview, ask: “Is there anything else you would like to add?” Once the interview is over, thank the respondent for their time and let them know that the team will deliver a debrief in March 2018 53 STAKEHOLDER GROUP: ICED II COP, DCOP AND COMPONENT LEADS INTERVIEW TRACKING DATA This section will be completed by the interviewer prior to the key informant interview Date of Interview Location of Interview Name of Data Collector Name of Respondent Role or Position/Title of Respondent Male/Female/Non-binary Phone number/email of Respondent INTRODUCTION Read the consent statement [This is a statement that tells them what we are doing and why and then asks for their consent to conduct the interview], and offer respondents the opportunity to ask questions. Have copies of the statement available in case the respondent prefers to read it. Once they have provided consent, proceed with the interview. INTERVIEW QUESTIONS Ask the interview questions below. Points to probe throughout the interview are shown in brackets [ ]. Question 24. Please tell me about your role as it relates to the ICED II Project. a. When and how did you start working with the ICED II Project? b. Did your interactions of involvement change over time? How so? 25. What are the activities that your component provided to stakeholders? Please describe them the way your stakeholders would describe them. 26. Are any of these activities working particularly well in terms of reaching the expected results? If so, why? 27. Are any of these activities making limited contribution to the expected results and/or overarching goal of the program? If so, why? If so, then why continue to do them? 54 Question 28. Have there been any key successes so far? 29. Have any implementation approaches or management methods such as the provinces in which ICED works, having a presence inside MEMR, or national vs. local offices affected the ability to achieve expected results (outputs, outcomes and strategic impacts) effectively? Or, if yes, please explain. Then ask about working with local versus national stakeholder engagement, the availability and quality of personnel, the project’s relationships with stakeholders. 30. Are there constraints that are limiting ICED II’s effectiveness? If yes, please explain. 31. In your own words, what have been the most meaningful or significant achievements of the ICED II Project? [PROBE: why?] 32. To what extent, if any, do you think any of the ICED II results will be sustained after the Project ends? [PROBE: Why?] 33. In your own words, could you describe how your component contributed to the ICED II? 34. [Before asking this question, show the Risk Framework Diagram.] Are there any emerging external factors (locally, nationally, regionally and globally) that are affecting/have the potential to affect the implementation or effectiveness of ICED II’s interventions? What are they and how are they or will they impact ICED II type interventions? [If asked for example, give recent FIT change] 35. In response to these factors you have identified, has ICED II recommended mitigating measures to address and adapt to these factors? 36. Does ICED II use monitoring, learning and research in the program? In other words, how does ICED II use M&E data to make course corrections? Has it changed course as a result of changes and if so, why and how? 37. If so, how has this been done? 38. Has ICED II done anything to address sustainability and future scaling of program results? 39. If yes, has ICED II incorporated these aspects in the work plans? 40. We realize that ICED II was not setup to promote US exports or US jobs. However, this is often an outcome of US foreign assistance. In clean energy it often takes the form of purchases of US equipment, US expertise in project engineering or financial advice or even financing. With that in mind, are you aware of any evidence that demonstrates the program’s contribution to domestic US job creation and/or market opportunities for US firms? 41. What developers, banks or equipment manufacturers should we talk to? 42. Do any of them have offices that you know of in Indonesia? 43. Looking back on ICED II’s performance, is there anything you recommend to make it better? 44. Why? 45. Is there activity you would have dropped? 46. Why? CONCLUSION Before concluding the interview, ask: “Is there anything else you would like to add?” Once the interview is over, thank the respondent for their time and let them know that the team will deliver a debrief in March 2018. 55 STAKEHOLDER GROUP: ICED II M&E MANAGER INTERVIEW TRACKING DATA This section will be completed by the interviewer prior to the key informant interview Date of Interview Location of Interview Name of Data Collector Name of Respondent Role or Position/Title of Respondent Male/Female/Non-binary Phone number/email of Respondent INTRODUCTION Read the consent statement [This is a statement that tells them what we are doing and why and then asks for their consent to conduct the interview], and offer respondents the opportunity to ask questions. Have copies of the statement available in case the respondent prefers to read it. Once they have provided consent, proceed with the interview. INTERVIEW QUESTIONS Ask the interview questions below. Points to probe throughout the interview are shown in brackets [ ]. Question 1. Please tell me about your role as it relates to the ICED II Project. a. When and how did you start working with the ICED II Project? b. Did your interactions of involvement change over time? How so? 2. [Before asking this question, show the Activities which ICED II shows in the work plan and monitoring plan.] Are there any activities, which ICED is doing that are not listed here? If yes, then what are they? 3. Are any of these activities working particularly well in terms of reaching the expected results? If so, why? 4. Are any of these activities making limited contribution to the expected results and/or overarching goal of the program? 5. In your own words, what have been the most meaningful or significant achievements of the ICED II Project? [PROBE: why?] 6. To what extent, if any, do you think any of the ICED II results will be sustained after the Project ends? [PROBE: 56 Question Why?] 7. Has monitoring, learning and research been used to inform adaptive management in terms of adjusting ICED II’s technical approaches and intervention designs? 8. If so, how has this been done? 9. Has ICED II identified key factors and strategies to ensure sustainability and future scaling of program results? 10. If yes, has ICED II incorporated these aspects in the work plans? 11. Prior to making the trip you were sent a request to map out the activities to the targets so we could understand what activities are making measurable contributions to the targets and by how much. Will you please walk us through that? 12. [The following only if they are including ICED only projects in the targets.] 13. Please explain the rationale for including projects or activities that took place exclusively before ICED II began work or to which ICED II did not contribute? CONCLUSION Before concluding the interview, ask: “Is there anything else you would like to add?” Once the interview is over, thank the respondent for their time and let them know that the team will deliver a debrief in March 2018. 57 STAKEHOLDER GROUP: ICED II NATIONAL AND PROVINCIAL STAKEHOLDERS INTERVIEW TRACKING DATA This section will be completed by the interviewer prior to the key informant interview Date of Interview Location of Interview Name of Data Collector Name of Respondent Role or Position/Title of Respondent Male/Female/Non-binary Phone number/email of Respondent INTRODUCTION Read the consent statement [This is a statement that tells them what we are doing and why and then asks for their consent to conduct the interview], and offer respondents the opportunity to ask questions. Have copies of the statement available in case the respondent prefers to read it. Once they have provided consent, proceed with the interview. INTERVIEW QUESTIONS Ask the interview questions below. Points to probe throughout the interview are shown in brackets [ ]. Question 1. Please tell me about your role as it relates to the ICED II Project. a. When and how did you start working with the ICED II Project? b. Did your interactions of involvement change over time? How so? 2. Please describe the activities which ICED is working with [Insert name of Government Entity] on? 3. Are any of these activities working particularly well in terms of reaching the expected results? If so, why? 4. In your own words, what have been the key successes so far? 5. Are there any aspects of the assistance that have not been as productive as you thought they would be? If so, what are they and why do you think they have not been so productive? 6. As you look at how ICED II assistance has taken place, do you so any aspects of project management that have affected the ability to provide the assistance your organization was expecting? If yes, please then proceed to next question. 7. Which project management aspects are they? 58 Question • General approach, • Relationship with stakeholders, • Availability of experts, • Geographic focus, • Local versus national stakeholder engagement • Other (Specify) 8. Are there constraints that are limiting ICED II’s effectiveness? If yes, please explain. 9. In your own words, what have been the positive achievements of the ICED II Project? [PROBE: why?] 10. To what extent, if any, do you think any of the ICED II results will be sustained after the Project ends? [PROBE: How?] 11. [Before asking this question, show the Risk Framework Diagram.] Are there any emerging external factors (locally, nationally, regionally and globally) that are affecting/have the potential to affect the implementation or effectiveness of ICED II’s interventions? What are they and how are they or will they impact ICED II type interventions? [If asked for example, give recent FIT change] 12. In response to these factors you have identified, has ICED II recommended mitigating measures to address and adapt to these factors? 13. Have you discussed sustainability and future scaling of program results with ICED? If yes, has ICED II identified key factors and strategies to ensure sustainability and future scaling of program results? 14. If yes, have you seen these aspects incorporated into the ICED work? 15. We realize that ICED II was not setup to promote US exports or US jobs. However, this is often an outcome of US foreign assistance. In clean energy it often takes the form of purchases of US equipment, US expertise in project engineering or financial advice or even financing. With that in mind, are you aware of any evidence that demonstrates the program’s contribution to domestic US job creation and/or market opportunities for US firms? 16. What developers, banks or equipment manufacturers should we talk to? 17. Do any have offices that you know of in Indonesia? 18. Looking back on ICED II’s performance, is there anything you recommend to make it better? 19. Why? 20. Is there activity you would have dropped? 21. Why” CONCLUSION Before concluding the interview, ask: “Is there anything else you would like to add?” Once the interview is over, thank the respondent for their time and let them know that the team will deliver a debrief in March 2018. 59 STAKEHOLDER GROUP: ICED II FINANCIAL STAKEHOLDERS INTERVIEW TRACKING DATA This section will be completed by the interviewer prior to the key informant interview Date of Interview Location of Interview Name of Data Collector Name of Respondent Role or Position/Title of Respondent Male/Female/Non-binary Phone number/email of Respondent INTRODUCTION Read the consent statement [This is a statement that tells them what we are doing and why and then asks for their consent to conduct the interview], and offer respondents the opportunity to ask questions. Have copies of the statement available in case the respondent prefers to read it. Once they have provided consent, proceed with the interview. INTERVIEW QUESTIONS Ask the interview questions below. Points to probe throughout the interview are shown in brackets [ ]. Question 1. Please tell me about your role as it relates to the ICED II Project. a. When and how did you start working with the ICED II Project? b. Did your interactions of involvement change over time? How so? 2. Was [Insert name of Financial Entity] working in the area of clean energy prior to working with the ICED project (either ICED I or ICED II)? 3. If yes, then what was it that [Insert name of Financial Entity] saw ICED as providing, that you did not already have? 4. If not, then what was it about ICED that encouraged you to work with them? 5. Please describe the activities which ICED is working with [Insert name of Financial Entity] on? 6. Are any of these activities working particularly well in terms of reaching the expected results? If so, why? 7. In your own words, what have been the key successes so far? 60 Question 8. [Show list of projects/loans that ICED II reports for this entity]. Is this list accurate in terms of the size (value) and status? 9. Could your organization have closed these loans without ICED II assistance? 10. Are there any specific projects or loans that reached financial closure on which ICED assistance was critical in helping you that are not on this list? 11. If yes, can you please provide us the names, IDR value and MW size of the project? 12. Are there any aspects of the assistance that have not been as productive as you thought they would be? If so, what are they and why do you think they have not been so productive? Now we will ask some project specific questions. 13. How many projects are ICED assisting your company with? For each project, beginning with the first, please answer the following. Project 1 14. What is the name:_______________________________________ 15. Size in MW: ____________________________________________ 16. Status :________________________________________________ 17. What areas of assistance did ICED II providing? Mark X for assistance provided by ICED On a scale of 1 to 5 where 5 is very good and 1 is very poor, how would you rate the assistance ICED II provided? _______ Develop Project Memo _______ Develop Project Memo _______ Discussion meeting with bank _______ Discussion meeting with bank _______ Project proposal review _______ Project proposal review ______ Provide Advice in Grid Interconnection _____ Provide Advice in Grid Interconnection _______ Review Engineering Design _______ Review Engineering Design _______ Review feasibility study _______ Review feasibility study _______ Review PPA _______ Review PPA _______ Review Prefeasibility Study _______ Review Prefeasibility Study _______ Site visit _______ Site visit _______ Solar Potential Study _______ Solar Potential Study 18. On a scale of 1 to 5 where 5 is very important and 1 is not very important, please rate each of the areas: _______ Develop Project Memo _______ Discussion meeting with bank _______ Project proposal review _______ Provide Advice in Grid Interconnection _______ Review Engineering Design _______ Review feasibility study _______ Review PPA _______ Review Prefeasibility Study 61 Question _______ Site visit _______ Solar Potential Study Project 2 19. What is the name:_______________________________________ 20. Size in MW: ____________________________________________ 21. Status :________________________________________________ 22. What areas of assistance did ICED II providing? Mark X for assistance provided by ICED On a scale of 1 to 5 where 5 is very good and 1 is very poor, how would you rate the assistance ICED II provided? _______ Develop Project Memo _______ Develop Project Memo _______ Discussion meeting with bank _______ Discussion meeting with bank _______ Project proposal review _______ Project proposal review ______ Provide Advice in Grid Interconnection _____ Provide Advice in Grid Interconnection _______ Review Engineering Design _______ Review Engineering Design _______ Review feasibility study _______ Review feasibility study _______ Review PPA _______ Review PPA _______ Review Prefeasibility Study _______ Review Prefeasibility Study _______ Site visit _______ Site visit _______ Solar Potential Study _______ Solar Potential Study 23. On a scale of 1 to 5 where 5 is very important and 1 is not very important, please rate each of the areas: _______ Develop Project Memo _______ Discussion meeting with bank _______ Project proposal review _______ Provide Advice in Grid Interconnection _______ Review Engineering Design _______ Review feasibility study _______ Review PPA _______ Review Prefeasibility Study _______ Site visit _______ Solar Potential Study 24. Project 3 25. What is the name:_______________________________________ 26. Size in MW: ____________________________________________ 27. Status :________________________________________________ 28. What areas of assistance did ICED II providing? Mark X for assistance provided by ICED On a scale of 1 to 5 where 5 is very good and 1 is 62 Question very poor, how would you rate the assistance ICED II provided? _______ Develop Project Memo _______ Develop Project Memo _______ Discussion meeting with bank _______ Discussion meeting with bank _______ Project proposal review _______ Project proposal review ______ Provide Advice in Grid Interconnection _____ Provide Advice in Grid Interconnection _______ Review Engineering Design _______ Review Engineering Design _______ Review feasibility study _______ Review feasibility study _______ Review PPA _______ Review PPA _______ Review Prefeasibility Study _______ Review Prefeasibility Study _______ Site visit _______ Site visit _______ Solar Potential Study _______ Solar Potential Study 29. On a scale of 1 to 5 where 5 is very important and 1 is not very important, please rate each of the areas: _______ Develop Project Memo _______ Discussion meeting with bank _______ Project proposal review _______ Provide Advice in Grid Interconnection _______ Review Engineering Design _______ Review feasibility study _______ Review PPA _______ Review Prefeasibility Study _______ Site visit _______ Solar Potential Study 30. In your own words, what have been the key successes so far from ICED? 31. Are there any aspects of the assistance that have not been as productive as you thought they would be? If so, what are they and why do you think they have not been so productive? 32. We realize that ICED II was not setup to promote US exports or US jobs. However, this is often an outcome of US foreign assistance. In clean energy it often takes the form of purchases of US equipment, US expertise in project engineering or financial advice or even financing. With that in mind, are you aware of any evidence from your project that demonstrates the program’s contribution to domestic US job creation and/or market opportunities for US firms? 33. Looking back on ICED II’s performance, is there anything you recommend to make it better? 34. Why? 35. Is there activity you would have dropped? 36. Why? CONCLUSION Before concluding the interview, ask: “Is there anything else you would like to add?” 63 Once the interview is over, thank the respondent for their time and let them know that the team will deliver a debrief in March 2018. 64 STAKEHOLDER GROUP: ICED II PROJECT STAKEHOLDERS27 INTERVIEW TRACKING DATA This section will be completed by the interviewer prior to the key informant interview Date of Interview Location of Interview Name of Data Collector Name of Respondent Role or Position/Title of Respondent Male/Female/Non-binary Phone number/email of Respondent INTRODUCTION Read the consent statement [This is a statement that tells them what we are doing and why and then asks for their consent to conduct the interview], and offer respondents the opportunity to ask questions. Have copies of the statement available in case the respondent prefers to read it. Once they have provided consent, proceed with the interview. INTERVIEW QUESTIONS Ask the interview questions below. Points to probe throughout the interview are shown in brackets [ ]. Question We will ask some general questions first and then ask project specific questions. 1. Please tell me about your role as it relates to the ICED II Project. a. When and how did you start working with the ICED II Project? 2. Did your interactions of involvement change over time? How so? 3. Was your company undertaking clean energy projects prior to working with the ICED project (either ICED I or ICED II)? 4. If yes, then what was it that convinced your company to work with ICED II? 5. If not, then what was it about ICED II that encouraged you to work with them? 27 PLN is included as an offtaker here but questions will probe deeper into other areas. 65 Question 6. As you look at how ICED II assistance has taken place, do you see any aspects of project management that have affected the ability to provide the assistance your organization was expecting? If yes, please then proceed to next question. 7. Which project management aspects are they: • General approach, • Relationship with stakeholders, • Availability of experts, • Geographic focus, • Local versus national stakeholder engagement • Other (Specify) 8. In your own words, what have been the positive achievements of the ICED II Project? [PROBE: why?] 9. Will you complete your project even if ICED II ends first? [PROBE: How and Why?] 10. [Before asking this question, show the Risk Framework Diagram.] Are there any emerging external factors (locally, nationally, regionally and globally) that are affecting/have the potential to affect the implementation or effectiveness of ICED II’s interventions? What are they and how are they or will they impact ICED II type interventions? [If asked for example, give recent FIT change] 11. In response to these factors you have identified, has ICED II recommended mitigating measures to address and adapt to these factors? Now we will ask some project specific questions. 12. How many projects are ICED assisting your company with? For each project, beginning with the first, please answer the following. Project 1 13. What is the name:_______________________________________ 14. Size in MW: ____________________________________________ 15. Status :________________________________________________ 16. What areas of assistance did ICED II providing? Mark X for assistance provided by ICED On a scale of 1 to 5 where 5 is very good and 1 is very poor, how would you rate the assistance ICED II provided? _______ Develop Project Memo _______ Develop Project Memo _______ Discussion meeting with bank _______ Discussion meeting with bank _______ Project proposal review _______ Project proposal review ______ Provide Advice in Grid Interconnection _____ Provide Advice in Grid Interconnection _______ Review Engineering Design _______ Review Engineering Design _______ Review feasibility study _______ Review feasibility study _______ Review PPA _______ Review PPA _______ Review Prefeasibility Study _______ Review Prefeasibility Study _______ Site visit _______ Site visit _______ Solar Potential Study _______ Solar Potential Study 66 Question 17. On a scale of 1 to 5 where 5 is very important and 1 is not very important, please rate each of the areas: _______ Develop Project Memo _______ Discussion meeting with bank _______ Project proposal review _______ Provide Advice in Grid Interconnection _______ Review Engineering Design _______ Review feasibility study _______ Review PPA _______ Review Prefeasibility Study _______ Site visit _______ Solar Potential Study Project 2 18. What is the name:_______________________________________ 19. Size in MW: ____________________________________________ 20. Status :________________________________________________ 21. What areas of assistance did ICED II providing? Mark X for assistance provided by ICED On a scale of 1 to 5 where 5 is very good and 1 is very poor, how would you rate the assistance ICED II provided? _______ Develop Project Memo _______ Develop Project Memo _______ Discussion meeting with bank _______ Discussion meeting with bank _______ Project proposal review _______ Project proposal review ______ Provide Advice in Grid Interconnection _____ Provide Advice in Grid Interconnection _______ Review Engineering Design _______ Review Engineering Design _______ Review feasibility study _______ Review feasibility study _______ Review PPA _______ Review PPA _______ Review Prefeasibility Study _______ Review Prefeasibility Study _______ Site visit _______ Site visit _______ Solar Potential Study _______ Solar Potential Study 22. On a scale of 1 to 5 where 5 is very important and 1 is not very important, please rate each of the areas: _______ Develop Project Memo _______ Discussion meeting with bank _______ Project proposal review _______ Provide Advice in Grid Interconnection _______ Review Engineering Design _______ Review feasibility study 67 Question _______ Review PPA _______ Review Prefeasibility Study _______ Site visit _______ Solar Potential Study 23. Project 3 24. What is the name:_______________________________________ 25. Size in MW: ____________________________________________ 26. Status :________________________________________________ 27. What areas of assistance did ICED II providing? Mark X for assistance provided by ICED On a scale of 1 to 5 where 5 is very good and 1 is very poor, how would you rate the assistance ICED II provided? _______ Develop Project Memo _______ Develop Project Memo _______ Discussion meeting with bank _______ Discussion meeting with bank _______ Project proposal review _______ Project proposal review ______ Provide Advice in Grid Interconnection _____ Provide Advice in Grid Interconnection _______ Review Engineering Design _______ Review Engineering Design _______ Review feasibility study _______ Review feasibility study _______ Review PPA _______ Review PPA _______ Review Prefeasibility Study _______ Review Prefeasibility Study _______ Site visit _______ Site visit _______ Solar Potential Study _______ Solar Potential Study 28. On a scale of 1 to 5 where 5 is very important and 1 is not very important, please rate each of the areas: _______ Develop Project Memo _______ Discussion meeting with bank _______ Project proposal review _______ Provide Advice in Grid Interconnection _______ Review Engineering Design _______ Review feasibility study _______ Review PPA _______ Review Prefeasibility Study _______ Site visit _______ Solar Potential Study 29. In your own words, what have been the key successes so far from ICED? 30. Are there any aspects of the assistance that have not been as productive as you thought they would be? If so, what are they and why do you think they have not been so productive? 31. We realize that ICED II was not setup to promote US exports or US jobs. However, this is often an outcome of US foreign assistance. In clean energy it often takes the form of purchases of US equipment, US expertise in project engineering or financial advice or even financing. With that in mind, are you aware 68 Question of any evidence from your project that demonstrates the program’s contribution to domestic US job creation and/or market opportunities for US firms? 32. Looking back on ICED II’s performance, is there anything you recommend to make it better? 33. Why? 34. Is there activity you would have dropped? 35. Why? CONCLUSION Before concluding the interview, ask: “Is there anything else you would like to add?” Once the interview is over, thank the respondent for their time and let them know that the team will deliver a debrief in March 2018. 69 ANNEX 4 ONLINE SURVEY 1. Was your company undertaking clean energy projects prior to working with the ICED project (either ICED I or ICED II)? Yes No 2. If yes, then what was it that convinced your company to work with ICED-II? 3. If not, then what was it about ICED-II that encouraged you to work with them? 4. How many of your projects did ICED-II assist with? 1 2 3 4 More than 4 Smaller than 1/2 MW 1/2 MW to 1 MW 1 MW to 5 MW 5 MW to 10 MW Larger than 10 MW Hydro Biomass Biogas Wind Solar PV Geothermal 5. For each project please provide the technology and approximate size in MW 70 Waiting for PPA to be signed PPA Signed Financial Closure Under Construction Operating Hydro Biomass Biogas Wind Solar PV Geothermal 6. What is the status of your projects 7. What assistance did you receive from ICED-II on these projects? Develop Project Memo Discussion with Financier Proposal Review Provide Advice in Grid Interconnection Review Prefeasibility Study Review Feasibility Study Review Detailed Engineering Study Site Visit Solar Potential Study Other (please specify) 71 1 2 3 4 5 Develop Project Memo Discussions with Financier Proposal Review Provide Advice on Grid Interconnection Review Prefeasibility Study Review Feasibility Study Review Detailed Engineering Study Site Visit Solar Potential Study Other 8. For each of the areas where ICED-II provided assistance to your projects please rate the quality of the assistance received on a scale of 1 to 5 with 5 being the best quality and 1 being the lowest or unacceptable. 1 2 3 4 5 Project Memo Discussions with Financier Proposal Review Advice on Grid Interconnection Review of Prefeasibility Study Review of Feasibility Study Review of Detailed Engineering Study Site Visit Other 9. On a scale of 1 to 5 where 5 is very important and 1 is not very important, please rate each of the areas: 72 10. Is any of the equipment on your project coming from a US Company? Yes No 11. What is the value of this US equipment in Millions of US Dollars? 12. Is there anything you would recommend to make ICED better 73 ANNEX 5 EVALUATION DESIGN MATRIX Evaluation questions Refined and/or Related Evaluation Questions Applicable to Several Stakeholder Groups Data collection methods, e.g., Records, structured observation, in-depth interviews, surveys Analytical methods, e.g., frequencies, crosstabs, Source(s) Method(s) content analysis a. Do the goal, results, indicators, and targets selected for ICED II appropriately and completely track all program activities, and accurate tools for monitoring and reporting on implementation? Is there a need to refine, add, omit and/or more clearly define the selected results, indicators, and targets? Additional question: Are there any activities, which ICED II is doing that are not listed here? If yes, then what are they? Original logic model; M&E Plan and Reports and Annual Reports. ICED II COP, DCOP and Component Leads, M&E Staff Records review In-depth interviews Structured Observation Online survey Descriptive Statistics Content analysis Quantitative Analysis b. Are there any activities in which ICED II engaged that are making limited contribution to the expected results and/or overarching goal of the program? If so, to what extent and what is the rationale for continued engagement? Are any of these activities making limited contribution to the expected results and/or overarching goal of the program? If so, why? If so, then why continue to do them? Monthly, quarterly reports, monitoring data, baseline data, audit/ strategy IEE. ICED II COP, DCOP and Component Leads, M&E Staff Records review In-depth interviews Online survey Contribution analysis Content analysis Are there any aspects of assistance that have not been as productive as you thought they would be? If so, what are they and why do you think they have not been productive? Provincial and national stakeholders Projects In-depth interviews Online survey Contribution analysis 74 Evaluation questions Refined and/or Related Evaluation Questions Applicable to Several Stakeholder Groups Data collection methods, e.g., Records, structured observation, in-depth interviews, surveys Analytical methods, e.g., frequencies, crosstabs, Source(s) Method(s) content analysis c. Are there any activities in which ICED II is engaged that show proportionally high contributions to the expected results in consideration of resources allocated? If so, to what extent have efforts been made to build on these activities to intensify their contribution to ICED II’s goal? Are any of these activities working particularly well in terms of reaching the expected results? If so, why? In your own words, what have been the most meaningful or significant achievements of ICED II? In your own words, what have been the key successes so far? Monthly, quarterly reports, monitoring data, project baseline data, audit/ strategy IEE and financial reports. ICED II COP, DCOP, Component Leads, M&E Staff Projects Financial stakeholders Records review In-depth interviews Content analysis Contribution analysis d. How do the implementation approaches and management of ICED II program affect the ability to achieve expected results (outputs, outcomes and strategic impacts) effectively, and what constraints are limiting its effectiveness? In addressing this question, the following aspects must be considered: project management approach, relationship with stakeholders, human resources availability, geographic focus, and local versus national Have implementation approaches and management of the ICED II program affected the ability to achieve expected results effectively? If yes, please explain. Are there constraints that are limiting its effectiveness? If yes, please explain. ICED II COP, DCOP, Component Leads In-depth interviews Online survey Contribution analysis As you look at how ICED II assistance has taken place, have any aspects of project management such as the approach, relationship with stakeholders, geographic Provincial and national stakeholders Projects Contribution Analysis 75 Evaluation questions Refined and/or Related Evaluation Questions Applicable to Several Stakeholder Groups Data collection methods, e.g., Records, structured observation, in-depth interviews, surveys Analytical methods, e.g., frequencies, crosstabs, Source(s) Method(s) content analysis stakeholder engagement. focus, etc. affected the ability to provide the assistance your organization was expecting? If yes, please explain. Are there constraints that are limiting its effectiveness? If yes, please explain. Financial stakeholders e. What are emerging external factors (locally, nationally, regionally and globally) that are affecting/have the potential to affect the implementation or effectiveness of ICED II’s interventions? What mitigating measures are recommended for ICED II address and adapt to these factors? Are there any emerging external factors (locally or nationally) that are affecting/ have the potential to affect the implementation or effectiveness of ICED II’s interventions? In response to these factors you have identified, has ICED II recommended mitigating measures to address and adapt to these factors? Other relevant projects and sector studies; ICED II technical reports ICED II COP, DCOP, Component Leads Provincial and national stakeholders Projects Records In-depth interviews Online survey Content Analysis Contribution Analysis f. How has monitoring, learning and research been used to inform adaptive management in terms of adjusting ICED II’s technical approaches and intervention designs? No change to this question. Follow up question: If so, how has this been done? Quarterly and Annual Reports ICED COP, DCOP, Component Leads, M&E Staff Records review In-depth interviews Content Analysis Contribution Analysis 76 Evaluation questions Refined and/or Related Evaluation Questions Applicable to Several Stakeholder Groups Data collection methods, e.g., Records, structured observation, in-depth interviews, surveys Analytical methods, e.g., frequencies, crosstabs, Source(s) Method(s) content analysis g. Has ICED II identified key factors and strategies to ensure sustainability and future scaling of program results? If yes, has ICED II incorporated these aspects in the work plans? No change to this question. Follow-up question: To what extent, if any, do you think any of the ICED II results will be sustained after the Project ends? Quarterly and Annual Reports ICED COP, DCOP, Component Leads, M&E Staff Records review In-depth interviews Online survey Content Analysis Contribution Analysis Have you discussed sustainability and future scaling of program results with ICED II? If yes, has ICED II identified key factors and strategies to ensue sustainability and future scaling of program results? Projects Provincial and national stakeholders In-depth interviews Online survey 2. To what extent can evidence be collected that demonstrates the program’s contribution to domestic US job creation and/or market opportunities for US firms? ICED II was not setup to promote US exports or jobs. However, clean energy often does this in the form of purchases of US equipment, US expertise in engineering, or financial advice and financing. Are you aware of any evidence that demonstrates ICED II’s contribution to domestic US job creation and/or market opportunities for US firms? Quarterly & Annual Reports, Success Stories ICED II Staff & stakeholders Projects Records review In-depth interviews Online survey Descriptive Statistics Content Analysis 1 | USAID REPORT TITLE HERE USAID.GOV ANNEX 6 LIST OF ENTITIES INTERVIEWED ENTITY DATE ICED II 2/5/18- 2/6/18 MEMR 2/7/18 Bappenas 2/7/18 Bank BNI 2/8/18 Kreasi Mini Hidro 2/8/18 Indonesia Infrastructure Finance 2/9/18 Bank Central Asia 2/9/18 Bumi Investco Energy Group 2/12/18 PLN 2/12/18 PT Tirtadaya Group 2/12/18 Bappenas 2/13/18 DJK 2/13/18 Nikko Securities 2/13/18 North Sumatra, Bappeda 2/13/18 North Sumatra, ESDM 2/13/18 US State Department 2/14/18 Citra Multi Energi 2/14/18 US Commercial Service 2/15/18 PT Nusantara Infrastructure 2/15/18 Site Visit at Parmonangan Project 2/14/18 Site Visit at Pakkat Project 2/14/18 PLN South Sulawesi 2/19/18 Equis Energy 2/19/18 PLN South Sulawesi 2/20/18 PT Energi Toba Prima 2/21/18 PT Petro Hidro Prima 2/22/18 2 | USAID REPORT TITLE HERE USAID.GOV Sewatama Group 2/22/18 East Java --- Office of Energy and Minerals 2/22/18 ICED 2/22/18 MEMR 2/23/18 MEMR -- Renewable Energy Division 2/27/18 OJK 3/1/18 3 | USAID REPORT TITLE HERE USAID.GOV ANNEX 7 DISCLOSURE OF CONFLICT OF INTEREST FORMS Matthew Addison, Team Lead 4 | USAID REPORT TITLE HERE USAID.GOV Joseph Viandrito, Institutional and Human Capacity Development Specialist 5 | USAID REPORT TITLE HERE USAID.GOV Rislima Sitompul, Energy Policy Specialist 6 | USAID REPORT TITLE HERE USAID.GOV Mark Bershatsky, Renewable Energy Project Development and Financing Specialist Disclosure of Conflict of Interest for USAID Evaluation Team Members I cemify {1} that I have cornpleted thi$ disclosure form fully and to the best of my ability and (2) that I will update this dlsclosure form promptly if relevant circumstances change. lf I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or dlsclosure for as long as it remains propri€tary and Name Mark Bershatsky Title Renewable Energy Project Development and Financing $pecialir Organization lntegra Govemment Services lnternational LLC Evaluation Position? Ll Team Leader lr Team member Evaluation Award Numher (contract or other instrument) AlD497-C-16-00006 USAID Proiect(sl Evaluated (l nclude proj e ct na me (s), im pleme nte r name{s) and oward number(s), if applicable) I have real or potential conflicts of interest to dlsclore, fYes ENo lf yes answered above, I dixlose the following facts: Real or Wtential conflicts aJ interast fioy include, but drc not limited to: 7. Clox fa*ily member who ls on employee af ths USAID opemting unlt mofiaging the praject{s) beiilg evoluated il the ifiplementiflg orgonlzatian(s) whose prgject(s) are being evoluated. 2. Finonciat inteftttthst is dlrect, ot is significant though indkect, iil the implefirentlng otganimttah(sJ whose pruiects are being evoluoted or in the autcome of the evaluatlon. 3. Cunent or prevlous dlrect or stgnificont though indtrcct experience with the prolect{s) being evaluoted, includlng involvefient ifi the prolect deslgn or previous iterotlons of the prolect. 4. Cwrcnt ar previous wotk experience or seeking employment with the USAID operqting unit mcnoging the evaluatian or the knplernenting organizcttion(s) whase proiect{s) dre being evoluated. 5, Current or previaus workexperience with an orgdnizstion that fiay be seen N an lnduitry cafipetitar with the implemeMiog orgonizatton(s) whose WJectls) dte betng evaluoted. 6, Preconcelv*d ideos towad individucls, gtaups, orgdnizatlons, or ,bkctives ofthe particulor prciects and organizdtions being evaluated thot cauld Nas the evoluation. refrain from usine the infornration for that for which it was furnished. Slgn*ture Oate fhol.eW --tib,tlti