1 October 2015 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Marc Shapiro, George B. Schaeffer, Anand Subbiah, Zaheer Ahmed Athar, Husain Babur, Faisal Haye, and Muhammad Danish under Management Systems International’s Performance Management Support Contract (PERFORM). ENERGY POLICY PROGRAM’S END-OF￾PROJECT PERFORMANCE EVALUATION The evaluation team receives a briefing at the Tarbela Dam control room ENERGY POLICY PROGRAM END-OF-PROJECT PERFORMANCE EVALUATION October 2015 Contracted under Order No. AID-391-C-15-00004 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i ACKNOWLEDGMENTS First and foremost, we wish to thank Pakistani government officials, and stakeholders who generously gave us their time, shared relevant information and patiently answered our endless questions. Without their participation the evaluation would not have been possible. Special thanks to the United States Agency for International Development Pakistan Mission’s Energy office, Performance Management Unit, and implementing partner, Advanced Engineering Associates International, Inc. (AEAI), for their invaluable guidance and assistance. Thank you. Sincerely, Marc Shapiro George B. Schaeffer Anand Subbiah Zaheer Ahmed Athar Husain Babur Faisal Haye Muhammad Danish ii CONTENTS Program Summary ...................................................................................................................................................... v Executive Summary ................................................................................................................................................... 1 Program Background ................................................................................................................................................. 6 Development Problem and USAID’s Response ........................................................................................... 7 Evaluation Purpose and Questions ....................................................................................................................... 10 Evaluation Methods and Limitations ..................................................................................................................... 12 Findings and Conclusions........................................................................................................................................ 15 Recommendations ................................................................................................................................................... 38 Annexes ..................................................................................................................................................................... 41 Annex 1: Evaluation Statement of Work .................................................................................................... 42 Annex 2: Table of Evaluation Question by Data Sources, Collection and Analysis Methodologies .. 80 Annex 3: Data Collection Instruments ........................................................................................................ 82 Annex 4: Bibliography of Documents Reviewed ...................................................................................... 103 Annex 5: List of Individuals and Agencies Contacted and Places Visited ............................................. 105 Annex 6: Evaluation Team Bios .................................................................................................................. 109 Annex 7: Key Factors Affecting Sustainability .......................................................................................... 111 Annex 8: Private Power and Infrastructure Board Upcoming IPPs ....................................................... 113 Annex 9: Analysis of Power Plants Administrative Data ........................................................................ 116 Annex 10: Assignment Work Plan ............................................................................................................. 150 Tables and Figures Table 1: Summary of Energy Policy Program ......................................................................................................... v Table 2: G2G Grant Power Plant Project Implementation ............................................................................... 16 Table 3: PESCO Operating Statistics .................................................................................................................... 19 Table 4: System Operating Statistics .................................................................................................................... 22 Table 5: List of Individuals and Agencies Contacted and Places Visited ....................................................... 105 Table 6: Private Power & Infrastructure Board Upcoming IPPS .................................................................... 113 Table 7: Installed Capacity and Effective Capability (MW) .............................................................................. 116 Table 8: Running Hours Sheet at Jamshoro TPS .............................................................................................. 117 Table 9: PESCO Interventions ............................................................................................................................. 118 Table 10: NTDC Support ..................................................................................................................................... 121 Table 11: Monitoring and Support of Transmission Project Implementation .............................................. 123 Table 12: Due Diligence ....................................................................................................................................... 124 Table 13: Training and Capacity Building ........................................................................................................... 126 Table 14: Planning Commission ........................................................................................................................... 129 Table 15: Ministry of Finance ............................................................................................................................... 131 iii Table 16: Ministry of Petroleum and Natural Resources ................................................................................ 138 Table 17: Ministry of Water and Power ............................................................................................................ 143 Table 18: Generation Technical Assistance ....................................................................................................... 146 Table 19: Import of Power ................................................................................................................................... 148 Table 20: Jamshoro Power Plant Maximum Load Attained During Jan. 2014, by Unit............................... 149 Table 21: Load at Jamshoro Units Before and After Rehabilitation .............................................................. 149 Figure 1: Location of EPP Activities ........................................................................................................................vi Figure 2: Mission Strategic Framework for DO1 ................................................................................................. 8 iv ACRONYMS ADB Asian Development Bank AEAI Advanced Engineering Associates International, Inc. AEDB Alternative Energy Development Board ASSIST Applying Science to Strengthen and Improve Systems COR Contracting Officer’s Representative CPPA Central Power Purchasing Authority DGPC Directorate General Petroleum Concessions DISCO Power Distribution Company DO Development Objective FESCO Faisalabad Electric Supply Company EPP Energy Policy Program G2G Government to Government GENCO Power Generation Company GIS Geographic Information System GoP Government of Pakistan GWh Gigawatt Hour IESCO Islamabad Electric Supply Company IMF International Monetary Fund IPP Independent Power Producer IR Intermediate Result ISGS Inter State Gas Systems Limited kV Kilovolt kWh Kilowatt-hour LESCO Lahore Electric Supply Company LLM Live Line Maintenance LNG Liquefied Natural Gas M&E Monitoring and Evaluation MoF Ministry of Finance MPNR Ministry of Petroleum and Natural Resources MSI Management Systems International MW Mega-watt MWP Ministry of Water and Power NEPRA National Electric Power Regulatory Authority NPCC National Power Control Centre NTDC National Transmission and Dispatch Company O&M Operation and Maintenance PC Planning Commission PDP Power Distribution Program PEPCO Pakistan Electric Power Company PESCO Peshawar Electric Supply Company PPIB Private Power and Infrastructure Board PQA Port Qasim Authority SOW Scope of Work USG United States Government USAID United States Agency for International Development WAPDA Water and Power Development Authority v PROGRAM SUMMARY TABLE 1: SUMMARY OF ENERGY POLICY PROGRAM Title/Field Program/Activity Information Contract Number AID-391-TO-12-00002 Contracting Officer’s Representative (COR) Saeed Anwar (COR); Rabia Bukhari (alternate COR) Start Date February 12, 2012 Completion Date October 15, 2015 Location Nationwide Name of Implementing Partner (IP)  Advanced Engineering Associates International, Inc. (AEAI) USAID/Pakistan Mission Strategic Framework Linkages Development Objective 1: Increased Sustainable Energy Supplied to the Economy Intermediate Result 1.1: Increase Energy Supply Intermediate Result 1.2: Improved Energy Sector Governance Budget $67,328,410 (Technical assistance and major procurement for transmission equipment by EPP) $291,710,000 (Government-to-Government Direct Investment) vi FIGURE 1: LOCATION OF EPP ACTIVITIES 1 EXECUTIVE SUMMARY PROGRAM BACKGROUND Pakistan's energy sector is in crisis. To help the Government of Pakistan (GoP) address the underlying causes and consequences of the crisis, the United States Agency for International Development (USAID) initiated the Energy Policy Program (EPP) in 2012. EPP provides economic and technical assistance to the GoP to enhance capacity to generate and transmit power, secure fuel supplies, and alleviate rolling blackouts, otherwise known as load shedding.1 More specifically, EPP focuses on reforming the energy sector; conducting and preparing due diligence reports for new projects under consideration for U.S. government support; supporting the Peshawar Electric Supply Company (PESCO) to serve as a turnaround distribution company; building capacity of energy sector entities; and providing technical assistance to support oil and gas policy, exploration, and importation. Advanced Engineering Associates International (AEAI) is implementing EPP under a $67.3 million contract for technical assistance. EPP also provides monitoring and implementation support to seven Government-to-Government (G2G) energy projects. These projects were ongoing under a G2G direct investment ($291.7 million) at the time of EPP’s commencement. EVALUATION PURPOSE AND QUESTIONS This final performance evaluation aims to assess whether EPP achieved its planned results and prospects for their sustainability. In addition, the evaluation identifies lessons and provides recommendations to guide implementation of the follow-on activity designed by USAID/Pakistan’s Energy Office to support the energy sector in Pakistan. The primary audiences for the evaluation are the USAID/Pakistan Energy Office, the U.S. State Department Special Representative for Afghanistan and Pakistan, the GoP, and the implementing partner AEAI, Inc. The specific evaluation questions are: 1. How and to what extent has the project achieved planned results? 1.1. What is the perceived effect of the 1,300 MW added to the grid through USAID energy projects?2 2. What is the likelihood that the results EPP has achieved are sustainable beyond the life of the project? METHODS The evaluation used a mixed-methods design for data collection and analysis, drawing data from both primary and secondary sources. The team collected primary qualitative data through semi-structured 1 “Load shedding” refers to deliberately disconnecting parts of a power-distribution system to reduce the quantity demanded when demand exceeds supply. 2 In the Statement of Work (SOW) document this sub evaluation questions was incorrectly numbered as 1.2. 2 interviews with EPP stakeholders and conducted site visits to observe program interventions. The evaluation team relied on secondary quantitative data from EPP’s monitoring and evaluation system, administrative data related to the Pakistan power sector, and plant and unit-level activity logs from facilities providing data. The evaluation also used secondary qualitative data collected from program reports and other documents. These included contractual documents, quarterly and annual reports, monitoring and evaluation reports, EPP training evaluation forms, and technical assessments and studies. The team triangulated findings across sources and methods whenever possible. FINDINGS AND CONCLUSIONS Questions 1 and Sub-Question 1.1: Results Generation, Transmission, and Distribution USAID’s investments in generation through G2G grants and facilitated by EPP were cost effective with high rates of return (Table 2). Results include: increased generation capacity, energy production, and reliability of power generation that occurred more quickly because of USAID funding. Although there were moderate delays related to procuring equipment, implementation was orderly, and the program achieved results mostly on schedule and with higher than initially-expected outcomes, e.g., 1,0133 megawatts (MW) of generation capacity restored to the grid versus the initial estimate of 430 MW. Through EPP’s interventions, transmission capacity throughout Pakistan increased modestly. EPP’s interventions in the Peshawar Electric Supply Company (PESCO) resulted in localized improvements in performance. The limited result is likely due to the short time period in which EPP worked with PESCO and the company’s aging 132 kV (kilovolts) transmission lines. EPP’s technical audits of the transmission system were a significant achievement. The audits identified congestion points and key requirements for maintaining, enhancing, and restoring infrastructure as well as recommendations to the National Transmission and Dispatch Company (NTDC) and the National Power Control Centre (NPCC) grid system operations and management. When addressed, these will reduce congestion and outages. Given the sheer size of the sector, the overall effects from USAID’s combined investments have been modest, but positive. USAID programs added or restored 1,518 MW of capacity including both generation and transmission. The added capacity represents 19 percent of the capacity deficit at peak deficit demand of about 8,000 MW.4 Considerable work and investment will be needed to achieve a reliable high voltage grid. Few managers interviewed could estimate how many MWs of capacity were added/restored as a result of EPP interventions or the benefits of the interventions. However, they indicated that the full benefits may not be visible or yet realized since progress has been slow in the commercial aspects of distribution. 3 Source: AEAI, EPP Quarterly Report, April –June 2015. 4 NTDC, 2014. “NTDC Power System Statistics 2013-2014: 39th Edition Planning Power NTDC.” 3 Governance and Policy EPP’s greatest impact and highest value added in governance and policy has been in the gas sector, where capacity and knowledge were most lacking. EPP was largely responsible for the contractual framework leading to the first importation of liquefied natural gas (LNG) to Pakistan. The Fast Track Regasification Terminal was established within a year and imports started in March 2015. The shale gas policy framework developed by EPP represents an important long-term potential that will contribute to the diversification of the fuel mix in Pakistan. EPP’s assistance was also instrumental in developing the Business Transfer Agreement between the Central Power Purchasing Agency (CPPA) and the NTDC. The agreement will allow an orderly separation of the two entities, which represents an important step toward developing an independent wholesale electricity market. Human Capacity Development EPP appropriately targeted human capacity building through trainings, study tours, seminars and internships designed to support generation, transmission, and distribution. Respondents (11 out of 18) reported that the content was relevant and well-presented and participants and managers had a favorable impression of the activities. The knowledge and skills delivered are expected to improve the operational capabilities of the power sector. All managers interviewed responded that live-line maintenance (LLM) training was very valuable, however they also reported that trainees’ skill development and retention was slowed by delays in receiving tools and equipment. Without tools and equipment, trainees were unable to practice and reinforce what they learned. EPP mitigated the effect of the delays, in part, by providing LLM refresher training courses for all Distribution Companies (DISCOs) beginning on/about March 2015 and providing tools at the end of the courses. Efforts to help the GoP reduce knowledge-related barriers to women’s employment in the sector have only just started. EPP’s trainings and internship program included a relatively high proportion of women, although it is unclear whether the women that were trained or participated in the internship program are currently or will continue to work in the sector. It is evident that more needs to be done and will require a multi-pronged approach and practical plan to address barriers to women’s participation in the sector. Question 2: Sustainability Conditions for system-wide financial sustainability are weak, resulting in insufficient spending on operations and maintenance (O&M) and long-term capital replacement. The evaluation team expects that EPP stakeholders will sustain O&M training for generation companies/plants, assuming sufficient O&M funding is available. This training will contribute to sustainability by improving daily operations. Maintenance will only be improved if funding is available, as in the case of Tarbela, where USAID encouraged protected (ring-fenced) funding from revenue generated by additional or restored capacity. The commercial shortcomings inherent in the system affect and overshadow USAID’s successes in increasing capacity. The loss of significant production without compensation from tariffs contribute to the GoP’s inability to support maintenance, not just at the facilities targeted by EPP but at others, where reduced capacity can offset hard-won gains. Over the past four years, distribution has had combined technical and commercial losses averaging 18 percent and collections of only 90 percent (Table 4). Furthermore, subsidies (Table 2) lead to even higher financial losses given the greater generation and transmission capacity now available. Unless the GoP increases tariffs and improves distribution 4 performance, for every kilowatt-hour (kWh) delivered, the GoP will lose even more money. Thus, USAID funding helps reduce shortages at the cost of additional financial losses. The live-line maintenance training requires additional support either by the end of EPP or afterward to cement learning. The refurbished NTDC Tarbela Training Center provides the institutional apparatus to sustain learning. The evaluation team does not expect that the knowledge or skills achieved through direct and indirect training in the policy area will be sustained due to the rotation of civil service staff every two to three years. Finally, the training with the Power SIM planning model is of questionable sustainability due to insufficient buy-in from the Ministry of Finance (MoF). MAIN RECOMMENDATIONS Generation and Fuels Leveraging its success, USAID should continue to facilitate importing and transporting liquefied natural gas. Doing so can lower generation costs and reduce greenhouse gas emissions from the use of heavy fuel oil. This may support development of more infrastructure and bolster technical capacity to increase imports. In particular, support will be required for the GoP’s planned construction of a land-based LNG regasification terminal. Transmission and Distribution To support overhauling transmission networks, USAID should encourage and support implementation of the technical audits undertaken during EPP. It should prioritize congestion issues and other activities that would be supported with the capital from the new Asian Development Bank (ADB) transmission loan. USAID should consider encouraging the development of market rules including eliminating the requirement that all electricity is sold to the CPPA. It should also continue to support the separation of the NPCC from the NTDC, so that an independent market operator is formed. Wholesale electricity markets will allow electricity buying and selling outside the single buyer regime now in place. Live-line maintenance training for DISCOs should continue for the balance of EPP’s period of performance and under the follow on activity if possible to enhance lineman’s skills. In addition, business plans developed under EPP for Power Generation Companies (GENCOs) and DISCOs including the Peshawar Electric Supply Company (PESCO) should be reviewed to ensure that tariffs are based on full cost recovery rates. Governance, Policy, and Human Capacity Development Governance and policy are areas in need of support that can be provided at low relative cost. These are also areas in which USAID has experience and a comparative advantage. USAID should consider supporting work on gas-related policies and agreements, subsidy targeting, amendments to the Electricity Act, and policies supporting improvements in DISCOs’ commercial operations. USAID’s future capacity development initiatives might be strengthened by including training in finance and accounting, program management (especially for Mangla Dam), commercial activities, resource management, human resources, and information technology. Future program assistance should consider upgrading software for basic customer information and supporting a billing system with telemetry capabilities at the DISCO level. 5 USAID should consider supporting the National Electric Power Regulatory Authority (NEPRA) to continue its bid to link tariffs to full-cost-recovery rates and to introduce energy efficiency demand-side interventions. USAID should review and support business plans developed under EPP to assure all elements of full-cost recovery have been considered. Assuming the GoP has sufficient interest in renewable power to help fill the capacity demand gap, USAID should consider governance-level support for the Alternative Energy Development Board (AEDB) and the Private Power and Infrastructure Board (PPIB). USAID should work with the next implementer to consider ways to get the most out of expat advisors including pairing local advisors with expatriate advisors to facilitate and enhance knowledge transfer while the advisor is embedded and to enhance prospects for sustainability of technical assistance and capacity building efforts. Given the small number of women in the energy sector, USAID should undertake an assessment and work with the GoP to expand employment opportunities for women including creating connections between power companies and universities to place women in jobs, helping the GoP monitor and track progress toward its goals, working with power plant managers to take practical steps to accommodate and increase women’s employment starting with providing appropriate facilities at targeted locations, and expanding opportunities for training. 6 PROGRAM BACKGROUND Between 1994 and 1998 electricity reforms in Pakistan moved the power sector from a monolithic, centrally-planned system toward more independent entities responsible for generation, transmission, and distribution. These reforms ceased when the GoP breached the power-purchase agreements with independent producers by demanding lower prices. Simultaneously, a lack of investment, especially in the private sector, contributed to Pakistan’s chronic power crisis. Overall, political interests and weak enforcement of energy regulations have stymied reform initiatives in the power sector. The current crisis is a function of increasing demand, a declining reserve of natural gas, and insufficient investment in generation capacity combined with a constrained transmission system that suffers a high rate of loss and low collection rates. In addition, the absence of a clear and long-term strategy to resolve the crisis is associated with high, untargeted subsidies that are not offset by tariffs thus exacerbating the GoP’s monetary deficits. The resulting nationwide electricity shortages progressively worsened after 2008 until they reached a current peak of approximately 8,000 MW. These shortages translated into power outages of up to 20 hours a day in many rural areas, from 8 to 10 hours in urban areas, and have had far reaching effects on the welfare of Pakistan’s citizens, industries, and macro-economic growth. The power sector’s technical and financial shortcomings, compensated for by the GoP’s direct monetary subsidies, created cumulative losses of $2.07 billion between 2012 to 2013.5 Electricity shortages in Pakistan have many causes, but the GoP’s continued insistence on maintaining low retail tariffs while using tax revenue to support an increasingly expensive system is among the most critical. The power sector’s debt, temporarily eased through payments of $550 million in 2013 and 2014, are growing once again. Termed circular debt, the subsidy to the power sector contributes to an increased budget deficit and continued need for funding. The State Bank of Pakistan must turn to international financial institutions (IFIs) to balance payments. Meanwhile, high, untargeted subsidies that benefit a few, low tariff collection rates, distorted pricing, high rates of transmission and distribution losses, and widespread corruption in public entities has left the private sector little incentive to invest in the sector. To encourage additional investment in the sector, the GoP will need to rationalize pricing—e.g., set tariffs on electric utilities at cost recovery levels—to enable power suppliers, power generators, and power distribution companies to recover their long￾term operating and investment costs. In order to help the GoP develop appropriate energy pricing, regulatory, and privatization policies, USAID, IFIs, and other bilateral donors provide various forms of assistance to the GoP. These efforts promote expediting energy sector reforms that will eliminate untargeted subsidies, promote private sector investment, and increase the supply of affordable power generation. They also support efforts to move toward a more appropriate and affordable fuel supply mix. In addition to these reforms, the GoP also needs to address power line losses and power theft, both of which increase costs to legitimate users. Implementing such reforms may be difficult in the near-term given the current political climate, but inaction will only exacerbate these problems over time. 5 Khan, Ahmad Fraz, 01 August, 2014. “Disastrous situation in power sector: Discos suffer Rs211bn loss,” The Dawn, accessed on 9 August, 2015 at http://www.dawn.com/news/1122459. 7 Pakistan’s power sector is currently in a transformational phase, with GoP-owned and managed entities moving toward attaining full autonomy with the ability to independently purchase, generate, transmit, dispatch, and distribute electricity. Traditionally, the GoP has administered the energy sector under a single-tiered structure, with the Water and Power Development Authority (WAPDA) Power Wing having the central authority. The WAPDA Power Wing provided the line and functional control of the Distribution Department and directed the operation of eight Area Electricity Boards across the country. In 1998, the GoP enacted legislation that led to WAPDA being restructured and the Area Electricity Boards being converted to publically-held DISCOs with the GoP owning all the equity. The bulk of power generation was assigned to generation companies (GENCOs), leaving WAPDA responsible only for hydropower and water dams. These reforms also established the National Electric Power Regulatory Authority (NEPRA) as a regulatory agency and the Pakistan Electric Power Company (PEPCO). Under the new reforms, these organizations assumed supervisory roles to oversee the transition of GoP￾owned entities to full autonomy. Over the course of the last 16 years, while the GoP made significant steps towards achieving the goal of full autonomy for the power sector (and with some GoP entities fully supporting reform), there have been few tangible results on energy shortfalls since the initiation of the reform agenda. The agreement between the International Monetary Fund (IMF) and the GoP calls for cutting subsidies to the power sector and increasing tariffs to compensate for the high transmission and distribution losses. The GoP has made some progress by increasing in the price of electricity, which has halved the rate of increase in deficits. Ongoing financial deficits, however, have led to operating deficits and insufficient maintenance and training, both of which require greater levels of funding and assistance to create a better functioning – and eventually sustainable – system. DEVELOPMENT PROBLEM AND USAID’S RESPONSE USAID/Pakistan designed the Energy Policy Program (EPP) to respond to these chronic energy shortfalls. In particular, EPP aims to support the GoP by (1) adding and saving megawatts, (2) decreasing transmission and distribution losses by investing in selected energy infrastructure projects and new technology, and (3) improving governance by supporting GoP reform efforts with technical assistance. Focus Areas and Groups EPP activities covered the nine distribution companies, key government ministries, and the transmission system. The Ministries engaged in EPP activities, largely the focus of the program’s policy reform initiatives, are the Ministry of Water and Power (MWP), the Ministry of Finance (MoF), the Planning Commission (PC), and the Ministry of Petroleum and Natural Resources (MPNR) and its sub-agencies. EPP worked with NTDC and NPCC to enhance transmission capacity and resolve existing bottlenecks. The focus on PESCO under Component IV as a “turnaround” DISCO because of its potential to improve is another important element in EPP’s initiatives to improve the transmission system. In addition, EPP provided inspection and verification services for USAID G2G projects in the energy sector. 8 DO 1: Increased Sustainable Energy Supplied to the Economy IR 1.1.1: Increased Generation and Transmission Capacity IR 1.2.1: Improved Policy Implementation IR 1.1.2: Improved Efficiency of Consumption and the Distribution Systems IR 1.1.3: Increased Financial Sustainability of Power Supply IR 1.2.2: More Autonomous Energy Sector Entities IR 1.2.3: Improved Capacity of USAID￾Supported IR 1.2.4: Increased constructive civil society engagement in the energy sector IR 1.1: Increased Energy Supply IR 1.2: Improved Energy Sector Governance IR 1.1.4: Increased Non-USG Investment in the Energy Sector Intended Results EPP contributes to Development Objective (DO) 1: “Increased Sustainable Energy Supplied to the Economy” and both intermediate results (IRs) (Figure 2).6 FIGURE 2: MISSION STRATEGIC FRAMEWORK FOR DO1 Approach and Implementation USAID designed its energy sector initiatives with the underlying belief that improving the energy sector and increasing the availability and reliability of energy supplies will lead to improved social stability and sustained economic progress (the results framework’s goal level results). EPP comprised four components.  Component I – Monitoring and Support of Project Implementation Under this component, EPP provided monitoring and implementation support for seven signature energy projects: four repair and maintenance projects (3 GENCOs and Tarbela Dam); two multipurpose dam completion projects (Gomal Zam and Satpara); and one dam rehabilitation project (Mangla). 6 Based on the SOW, EPP contributes to IR 1.1.1, IR 1.1.4. IR 1.2.1, IR 1.2.2, and IR 1.2.3. 9  Component II – Advice and Support of Energy Sector Policy Reform Energy sector policy reform was a critical focus of EPP and was implemented by undertaking activities that were requested by one or more GoP entities and by directly seconding staff to work in the offices of the requesting entities as advisors and specialized support staff. EPP also supported generation and transmission-related technical studies and policy reform activities assigned by USAID with the support of the GoP.  Component III – New Projects, Planning and Development The activities implemented under this component center on providing due diligence of projects which were candidates for U.S. Government (USG) support. EPP conducted and prepared due diligence reports on these projects to provide detailed information, reduce USG risk, and set a basis for creating Project Implementation Agreements for the selected projects.  Component IV – New Activities USAID/Pakistan expanded EPP’s original scope in September 2012. The expanded scope incorporated new activities aimed at responding to the GoPs efforts to address its current energy crisis, e.g., severe load shedding. The component covered support to PESCO, live line training for DISCOs, technical support to the GoP for upstream oil and gas activities, technical support to NTDC and NPCC, technical assistance to GENCOs and WAPDA, and expanded support for governance. 10 EVALUATION PURPOSE AND QUESTIONS PURPOSE AND USE OF THE EVALUATION This final performance evaluation will determine the extent to which the program has achieved planned results and the prospects for their sustainability. USAID/Pakistan will incorporate the evaluation’s conclusions and recommendations in decisions about future assistance to the GoP’s energy sector reforms and the type and scope of possible future USAID/Pakistan interventions in the energy sector. AUDIENCE AND INTENDED USE The primary audiences for the evaluation include: USAID/Pakistan, particularly the Energy Office; the U.S. Embassy in Pakistan, Applying Science to Strengthen and Improve Systems (ASSIST) Office; the U.S. State Department Special Representative for Afghanistan and Pakistan; the USAID Office of Afghanistan and Pakistan Affairs (OAPA); and the implementing partner AEAI, Inc. The GoP will receive the evaluation report’s executive summary and recommendations. EVALUATION QUESTIONS The evaluation statement of work (SOW) specified three evaluation questions, the second being a sub￾question to the first. 1. How and to what extent has the project achieved planned results? Explanation: The question addresses the effectiveness of EPP overall, and specifically its various components. The answer to the question should explore what has worked and how, what has not worked as anticipated and why, and highlight any unintended outcomes. It should also examine instances where EPP has changed its approach to overcome obstacles and the results of these changes and identify issues (political and operational) that have not been addressed and why. The answer should address EPP’s effectiveness in improving the “governance” of public sector power generation and transmission companies. It should also provide practical recommendations and lessons learned for making future programs more effective. This answer should also provide recommendations about which program activities to continue, revise, or drop in future energy programming. 1.1. What is the perceived effect of the 1,300 MW added to the grid through USAID energy projects? Explanation: The sub-question will specifically focus on examining EPP stakeholders’ perception of the effect—if any—of the 1,300 MW EPP activities added to the grid. The answer to this question will explore if and in what ways the additional energy has improved/changed/affected the GoP’s perception of the utility of EPP vis-à-vis USAID’s support to the energy sector. 11 2. What is the likelihood that the results EPP has achieved are sustainable beyond the life of the project? Explanation: This question should generate conclusions about which EPP results are likely to be sustainable beyond USAID support. It should also explore factors that contribute to sustainability or unsustainability and provide recommendations for how to enhance prospects for sustainable results including additional actions EPP or USAID can take that would enhance prospects for sustainability, e.g., continued advocacy after EPP ends or for follow-on activities. 12 EVALUATION METHODS AND LIMITATIONS EVALUATION METHODOLOGY The evaluation employed a mixed-methods approach, using both quantitative and qualitative data collection and analysis techniques to develop in-depth contextualized information. The evaluation used data from multiple sources and locations collected through a variety of methods, which allowed the team to triangulate information thereby enhancing the validity of findings, conclusions, and recommendations. DATA COLLECTION Before and during the fieldwork, the evaluation team collected and reviewed program documents from USAID and AEAI. These included quarterly and annual reports, program modifications, monitoring and evaluation (M&E) reports, technical assessments, studies, and policy documents generated by EPP for various power sector entities. The document review helped the evaluation team develop a strong understanding of EPP, the development needs it is designed to address, and the basis for the evaluation questions and the evaluation. The evaluation team split into three sub-teams, each with two members, so it could conduct fieldwork in multiple sites simultaneously. It conducted 69 semi-structure interviews over a two-week period in July 2015 (Annex 5). The team conducted a majority of the interviews in person and a few by telephone. The team also conducted follow-up interviews with a small subset of stakeholders to collect administrative information that was not available during the initial interview and for several stakeholders who had asked for extra time to provide additional information/data. The team interviewed seven AEAI staff members; six USAID/Pakistan Energy Office staff members; 17 individuals representing eight energy-related institutions;7 representatives of five of Pakistan’s nine DISCOs;8 13 managers and operational staff from the thermal generation plants9 and the hydro-power projects;10 managers from the Asian Development Bank (ADB), Port Qasim Authority (PQA), Sui Southern Gas Company Limited (SSGCL), Pakistan State Oil (PSO), Alternative Energy Development Board (AEDB), the Private Power and Infrastructure Board (PPIB); and EPP advisors seconded to MWP, NTDC, MoF, MPNR, and PESCO. The EPP-seconded advisors changed over time and several left as EPP 7 The Ministry of Water and Power (MWP), Planning Commission (PC), Ministry of Finance (MoF), Directorate General Petroleum Concessions (DGPC), Water and Power Development Authority (WAPDA), Ministry of Petroleum and Natural Resources (MPNR), National Power Construction Corporation (NPCC), and National Transmission and Dispatch Company (NTDC). 8 The Islamabad Electric Supply Company (IESCO), Multan Electric Power Company (MEPCO), Peshawar Electric Supply Company (PESCO), Lahore Electric Supply Company (LESCO); and Faisalabad Electric Supply Company (FESCO). 9 Muzaffargarh GENCO, Jamshoro Power Plant, Guddu Power Plant, and GENCO Holding Company. 10 Tarbela, Mangla, Gomal-Zam, and Satpara Dams 13 wound down. Therefore, all of the recent advisors were not available for interviews nor was it possible to assess the quality of advisors outside of feedback from stakeholders. Quantitative administrative data on program inputs and outputs were an essential component of the secondary-data review process. The team gathered these data from EPP documentation and from AEAI staff members and government entities including ministries, GENCOs, and NTDC. These data included monthly unit-level log sheets from power plants, unit running hours, and heat rates. Collecting and analyzing these data enhanced the team’s ability to triangulate findings and highlight areas requiring greater probing to assess outcomes. The evaluation also used these data to verify the number of MW added/restored to the grid and determine the avoided cost per MW restored. Due to time constraints, the evaluation design and statement of work (SOW) did not specify interviews with recipients of trainings provided by EPP. Nonetheless, the team gathered aggregated information from training evaluation forms provided by AEAI at the end of training sessions to help determine whether the trainings met recipients’ expectations. To assess the likelihood of EPP having achieved sustainable results, the team developed a preliminary set of questions and related draft indicators of sustainability. The indicators of sustainability focused on conformity of policy support with stakeholder objectives; organizational structures designed to improve continuity of reforms; persistence of human resource capacity building; institutionalization of policy; and financial sustainability. The questions and responses were incorporated during data analysis, as appropriate. (See Annex 3 for semi-structured interview protocols.) DATA ANALYSIS The evaluation team employed a structured and systematic approach to analyze the qualitative data. It first created a tally sheet with all questions and then identified typical responses and common themes that emerged from the various interviews. To ensure cross-verification of the evidence and to reach consensus on the themes that emerged from the interviews, the team reviewed the findings and coded the tally-sheet as a group. The tally sheet allowed the team to discern trends and dominant responses. Following this exercise, the team drafted a detailed outline summarizing key findings, conclusions, and recommendations for each evaluation question. Wherever possible, the team compared the results of the thematic analysis with quantitative and administrative data to assess the reliability of interview statements and validity of findings. Although the team did not receive systematic unit-level data from all generation facilities, it used available unit-level data to confirm plant-level data from EPP. Although the SOW did not include an assessment of the cost-effectiveness of interventions, the evaluation team compared administrative data from Pakistan with international benchmarks to calculate avoided-cost ratios, a common metric used in the power generation sector and central to integrated resource planning. The team also carried out quantitative analysis regarding effects at the system and DISCO-level for PESCO in order to assess potential effects EPP may have had more broadly and to lend further credence to the qualitative findings. In order to attribute results appropriately, the evaluation team uncovered other sources of financial support and development assistance that may have co-funded the investments. The team used the information collected from training evaluation forms to assess whether the trainings met recipients’ expectations. It triangulated these data with data from those who manage and supervise the trainees, which allowed greater depth of analysis. The sustainability of capacity built through training was assessed through managers’ impressions and not by response forms. 14 METHODOLOGICAL STRENGTHS AND LIMITATIONS The evaluation design permitted a multi-tiered triangulation process across the information spectrum, including both primary and secondary data sources thus ensuring that the final set of findings and corresponding conclusions were reliable and had a high degree of validity. However, there is a potential for selection bias inherent in a qualitative sampling approach that relies on input from the implementer and the stakeholders identified by AEAI and USAID. To reduce this source of potential bias, the evaluation team reviewed EPP work plans to identify additional stakeholders for interviews. Interview data represented individuals’ personal experiences and perspectives which may or may not reflect reality. To validate findings, the team cross-verified interviewee statements against primary and secondary quantitative data and inputs from other individuals when possible and asked clarifying follow￾up questions when inconsistencies arose. For example, when a GENCO respondent gave a figure for the MW of capacity added due to EPP initiatives, the team corroborated the data against log-sheets comprised of unit peak-load generation data, among other variables, when the data were available. The methodology of using ex-post data collection with treatments and no control means that exogenous contextual changes occurring at the same time as the treatment cannot be ruled out as causes of observed outcomes. However, discussions and administrative data should identify these confounding factors. Outputs, as direct results of inputs, do not require controls to determine attribution. The evaluation team also asked interviewees s about other sources of support and development assistance that could have also-funded the investments or otherwise explained results. As there were few overlaps with other donors and interviews focused on the interventions’ direct outputs, attribution of results to EPP was relatively strong. Some of the key stakeholders identified for interviews were either unavailable or lacked the knowledge required to answer the questions meaningfully because they had not interacted with the EPP or had not been in the position long enough to have sufficient perspective. In these cases, the team asked AEAI or contacts within the institution to suggest alternative appropriate individuals who could provide information or rescheduled the interview with the original subject. The team continued data collection on an as-needed basis during the period of data analysis and report writing, in order to conduct interviews with key informants who were not available earlier and/or to verify new information. Furthermore, some stakeholders had limited time for a full interview. Therefore, the evaluation team did not obtain responses from all interview subjects on all questions. Non-response bias is possible but unlikely and the direction of any bias is unknown. Finally, each section of findings notes the number of respondents. EPP post-training evaluation forms are typical of the industry and focus on whether the trainings met recipients’ expectations. They were not intended to evaluate broader outcomes and so did not include follow-up surveys to assess changes in practice or, more difficult, skills built. The team expected a positive bias from these simple training evaluation forms and weighted the training form data lower than responses from supervisors. EPP reported that they conduct surveys and focus group discussions (FGDs) of training participants six months after the training, however the evaluation team did not receive information from the FGDs that were conducted. 15 FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1: RESULTS Evaluation Question 1 – How and to what extent has the project achieved planned results? To answer the question, this section presents findings related to generation, transmission, and distribution, which includes interventions involving physical capacity and human capacity and the overall processes involved. Generation – USAID-funded Government-to-Government Grants This sub-section addresses increased generation capacity under Sub-IR 1.1.1, and implies results under Sub-IR 1.1.3, increasing financial sustainability. USAID provided $291.7million in G2G grants through fixed-amount reimbursable agreements to help the GoP increase generation capacity rapidly at seven plants: four repair and maintenance projects (three GENCOs and Tarbela Dam), two multipurpose dam completion projects (Gomal Zam and Satpara), and one dam rehabilitation project (Mangla). EPP provided procurement assistance and oversight, verified delivery of equipment, monitored progress, and produced monthly and quarterly reports. The projects succeeded in restoring capacity cost-effectively, roughly on budget, and ahead of plan in both capacity installed and production. Even though all plants had program interventions to increase capacity across a range of activities, some plants moved beyond just building capacity to focus on strengthened reliability and increased availability as well. Table 2 summarizes financial-related investments, achievements, and financial data related to sustainability at each of the seven plants. At Tarbela Dam, EPP restored 128 MW on three units and replaced electromechanical governors with digital governors for ten units, thus increasing reliability at the largest hydro station in the country. Managers at Tarbela reported that the Phase I work paid for by USAID would have occurred without the intervention though at a slower schedule and would not have included Phase II work that will occur after EPP closes. Tarbela was the only generator to suggest that the work would have been undertaken on a timely schedule without USAID support. EPP also helped complete two small dams at Gomal Zam and Satpara for which the GoP had halted construction due to financial constraints after the 2007 financial crisis. EPP helped put the activities back on track, adding 35 MW to the system and completing irrigation canals. The program restored capacity at the three thermal power stations with 500 MW restored at Muzaffargarh thermal power station representing the majority (49 percent) of the total restored capacity (Table 2). 16 TABLE 2: G2G GRANT POWER PLANT PROJECT IMPLEMENTATION Activity Planned Investment 1 Actual Investment 3 Additional Investment Project Objectives 2 Project Achievements Avoided Cost 5 Project Investment 6 Avoided Cost Ratio Sustainability-related Calculations Capacity Generation Achieved Capacity 4 Increase over expected Achieved Generation Additional generation over expected Regulatory Tariff Price 7 Full Cost Recovery Benchmark 8 Price - Benchmark (USD Millions) (USD Millions) (%) MW Million kWh (GWh) MW MW Over GWh GWh Over (USD per kW) (USD per kW) (Ratio) (USD per kWh) (USD per kWh) (USD per kWh) Muzaffargarh 9 $15.2 $15.7 3% 165 930 500 335 2,847 1,917 $1,000 $30 32.9 $0.12 $0.16 -$0.04 Jamshoro 10 $18.4 $19.2 4% 95 530 270 175 1,537 1,007 $1,000 $68 14.7 $0.10 $0.16 -$0.06 Guddu 11 $18.1 $18.9 4% 55 305 80 25 456 151 $1,000 $226 4.4 $0.05 $0.08 -$0.03 Thermal totals $ 51.7 $ 53.8 4% 315 1,765 850 535 4,840 3,075 $3,000 $325 9.2 Tarbela Phase 1 $16.5 $14.3 -13% 80 192 128 48 729 537 $3,000 $129 23.3 $0.001 $0.07 -$0.07 Gomal Zam $40.0 $45.0 13% 17.4 91 17.4 0 99 8 $3,000 $2,299 1.3 $0.001 $0.07 -$0.07 Satpara $26.0 $26.0 0% 17.6 105 17.6 0 100 -5 $3,000 $1,477 2.0 $0.001 $0.07 -$0.07 Mangla 12 $150 90 434 Grand Totals $284.2 $139.2 4% 520 2,587 1,013 583 5,768 3,615 17 Sources/details: 1, 2 Based on original contract dated October 2011 3, 4 Based on March 2015 Quarterly Report 5 Avoided cost is the price of new construction an additional kW. US Energy Information Administration. Updated Capital Cost Estimates for Utility Scale Generation Plants 6 Investment of project divided by addition kW 7 Price set by the Regulatory Agency NEPRA 8 Source: US Energy Information Administration. Levelized Cost and Levelized Avoided Cost of New Generation Resources in the Annual Energy Outlook 2015 9 Single Cycle Steam Plant using Heavy Fuel Oil 10 Single Cycle Steam Plant using Heavy Fuel Oil 11 Single Cycle Steam Plant using Gas 12 Being implemented By the end of USAID/Pakistan’s 2012 fiscal year, public-sector thermal power stations had an available capacity of 3,635 MW of an installed capacity of 4,841 MW. Public-sector utilities overall had an available capacity of 10,405 MW (measured during the summer, the highest capacity period for hydro-power plants) of an installed capacity of 11,357 MW. This indicates a loss of capacity of 952 MW, or 8 percent of original capacity.11 The USAID activity helped restore 1,013 MW of capacity in the public-sector hydros and GENCOS. The 850 MW restored at thermal power stations accounted for 84 percent of total restored capacity (Table 2). The rehabilitation achieved more than expected in energy production with 5,768 gigawatt hours (GWh) produced in 2014 relative to the plan of 1,838 GWh, or 214 percent over plan. Actual production at plants will be determined by the output of other units still requiring upgrades, actual demand given transmission and distribution constraints, and external factors that could limit production. The evaluation team found no overlap with other donors’ support at these plants and therefore attributes the results solely to USAID G2G assistance facilitated by EPP. All important technical parameters improved at USAID-funded public-sector generators. Heat rates at GENCOs improved by an average 10.6 percent which improved efficiency.12 Available capacity has increased by 10 percent relative to installed capacity in the summer and 12 percent in the winter. Between 2011 and 2014, available capacity increased by 10 percent in the summer and 12 percent in the winter. Over the same time period, available capacity as a percentage of installed capacity increased by one percent in the summer and three percent in the winter (Table 7 in Annex 9). Managers interviewed about physical capacity identified increased reliability (10 of 14), increased physical capacity and revenues (7 of 14), improved efficiency (6 of 14), and longer estimated useful life (5 of 14) 11 NTDC, 2012. “NTDC Power System Statistics 2011-2012: 37th Edition Planning Power NTDC”, Tables G-1 and G-2 12 The data necessary to calculate efficiency improvements at the unit level was not provided by plants uniformly across all plants. 18 as the most important outcomes of EPP. The increased reliability is an important factor in achieving the higher energy production. Except for a few units with high operating costs, most of the units in generation plants where EPP restored capacity operate as base power units. Base load units are run on a continuous basis, whereas peak load units are run as required to meet demand. For example, the January 2014 monthly record13 (Table 8 in Annex 9) shows that units in the Jamshoro plant operated between 69 percent and 81 percent of the time, implying that they were on standby 19 percent to 31 percent of the time. Projects supported by EPP were roughly on budget and cost effective (Table 2). Subtracting the costs for Mangla Dam, which will not begin until after EPP closes, actual expenditures totaled $139.2 million against a budget of $132.2 million. Across the utilities, the investments display medium to high avoided cost ratios of 1.3 to 32.9. Avoided costs in this case are the incremental savings from not having to replace the units entirely.14 The cost per kilowatt (kW) for rehabilitation ranged between $32 and $2,300 with the $32 to $242 per kW costs at thermal stations being substantially lower than the $1,000 per kW international benchmark for new installations. The comparison does not account for the shorter life-span of rehabilitated units relative to new units. The range for hydro-power is wider at $128 to $2,300 per kW and comes closer to the $3,000 per kW international benchmark for replacement costs. However, Satpara and Gomal Zam dams were under construction and should last longer than rehabilitated infrastructure. No generation investments required more than 13 percent additional funding, and one required less than planned. As a result, the package of investments cost only four percent more than planned. Conclusions: EPP-supported restoration and construction projects effectively increased the power sector’s generation capacity and reliability. The investments, individually and in total, were cost-effective with high rates of return. The added capacity has, or will, increase production throughout much, but not all, periods of the day and year. Most of the work would not have occurred or would have occurred at a slower schedule without USAID assistance. Transmission and Distribution Distribution Interventions (Live-line Maintenance Training) EPP launched a $2 million Live-line Maintenance (LLM) training at NTDC Tarbela Training Center for all nine DISCOs and provided specialized tools and vehicles used in LLM. The LLM program was designed to train LLM crews to service energized 132 kilovolt (kV) and 66 kV transmission lines. LLM eliminates the need to disrupt power to customers to perform routine maintenance thus increasing reliability. In addition to the training provided, EPP refurbished the Tarbela Training Center to working condition. Although the stakeholders rated the training favorably, the delay in providing equipment and vehicles for four DISCOs highlighted a need for refresher training and oversight for safety before workers begin practicing the skills. EPP started providing LLM refresher training for DISCOs in the second quarter of FY 2015. USAID/Pakistan cancelled the only other distribution-focused intervention planned under EPP. 13 The evaluation team did not receive other monthly data from Jamshoro plant nor from other thermal utilities. 14 Avoided cost metrics are central to integrated resource planning. This figure measures only direct costs and does not discount for the shorter lifespan of restored older facilities relative to new facilities. 19 The evaluation team did not find evidence of other donors or external partners working in or funding LLM training improvements, and thus attributed results solely to EPP. To date, seven of the nine DISCOs have received full LLM training. The evaluation team interviewed five of the nine DISCOs. All five DISCO managers overseeing the training confirmed that live-line repair will improve availability of the system and requested more practical training in the future. They confirmed that EPP selected the right people to undertake the training and that they believed the training was extremely valuable. Only one DISCO (PESCO) has received the equipment and trucks required for LLM and was using them in the field during the time of the evaluation. The manager at PESCO indicated that the use of LLM equipment enhanced PESCO’s capacity to reduce unplanned outages and improve system availability. The other four DISCO managers expressed concern that the skills learned would fade away with ongoing delays in receiving equipment. As noted above, EPP has addressed in part the effect of delays by starting LLM refresher training courses in the second quarter of FY 2015 at the conclusion of which DISCOs are supposed to be provided with tools. Transmission at the Peshawar Electric Service Company (PESCO) USAID selected PESCO as a “turnaround DISCO," because its systems need substantial improvement and PESCO management had a desire to improve them. The potential for significant improvement was intended to highlight to other underperforming DISCOs the value of undertaking strategic repairs and capacity building. EPP’s $8.2 million, year-and-a-half long assistance to PESCO under component IV included investments in grid repairs, LLM, technical studies, and capacity building. TABLE 3: PESCO OPERATING STATISTICS Units 2011 2012 2013 2014 Electricity Received GWh 10,331 11,246 11,190 10,391 Distribution losses % 37% 38% 37% 35% Billed 1 GWh 6,509 6,973 7,050 6,754 Collection Rate % 82% 82% 82% 85% Net Collections 2 GWh 5,337 5,717 5,781 5,741 Total Lost 3 % 48% 49% 48% 45% Source: Powers System Statistics. NTDC 1 Billed is electricity received less distribution Losses 2 Net collections is billed less the amount not collected for (1-collection rate) 3 Total lost is the Electricity Received less Distribution Losses less non collected amounts. 20 PESCO has an operational area of 74,251 square kilometers15 bordered by conflict-prone federally administered tribal areas (FATA) and serving some 2.6 million consumers16 with a large rural component. In 2011, PESCO experienced considerable losses and problems with collection (Table 3). Distribution losses reached 37 percent in 2011 and increased to 38 percent by 201217 and the collection rate on billings was 82 percent implying that PESCO realized payment on just over half of the electricity received. The loss figures are very high by international standards and domestically, compared to about 9 percent losses from distribution and collections for IESCO and 11 percent for FESCO.18 EPP’s interventions involved a number of activities (Table 9, Annex 9). The remainder of this section highlights the results. PESCO indicated that it was working with no other external donor partner to improve its systems, although it is working with the Power Distribution Program (PDP), also funded by USAID. Respondents also indicated that PESCO’s own internal work plan for improvement was limited and did not overlap with activities funded by USAID. Therefore, the team attributes results solely to USAID. The evaluation team isolated results to primarily EPP-focused inputs and thus attribution rests primarily with EPP, although some overlap with PDP cannot be excluded. With USAID support, EPP undertook numerous infrastructure enhancements at both the PESCO and NTDC. Interventions included civil works at capacitor banks and replacing/repairing transformers and cooling fans. For PESCO, all nine capacitor banks are now operational, up from three prior to the intervention. The net results are a gain of 505 MW over the entire system19 and major improvements within PESCO’s higher voltage network. To reduce technical and commercial losses, PESCO is developing a study based on telemetry data generated from USAID-supplied equipment. The PESCO manager said the studies conducted on transformer oil sampling, technical audits, and reactive power were very valuable and will lead to improvements in the grid infrastructure. The chief executive officer and the nine other managers interviewed indicated that the equipment and vehicles provided by EPP reduced unplanned load￾shedding in localized areas, improved system reliability, and faster response time to emergencies. The evaluation team was unable to obtain data on reliability in terms of unplanned outages or load-shedding to validate the managers’ assertions objectively in time for the report. Ten officers ranging from field officers to operational heads and the chief executive officer revealed a high level of appreciation and enthusiasm about the value of the EPP assistance, with one respondent stating “EPP has saved us [PESCO] from collapse.” A senior manager of PESCO heading the project management unit indicated no major problems working with EPP. However, four of five line managers voiced concern over delays in receiving hot tools and vehicles for live line repair crews. With one and a half years of support from EPP, the management indicated that barely 20 percent of the operational area has benefited from USAID assistance and that day-to-day operations have not yet improved dramatically even though some decrease in losses is apparent. However, PESCO’s 15 NEPRA, 26 June, 2013. “PESCO Tariff Petition 2013-2014.” 16 PESCO website (www.pesco.gov.pk) 17 According to PESCO, the Hattar Industrial Estate being a more dense service area has transmission and distribution losses comparable to other DISCOs such as FESCO. 18 NTDC, 2013. “NTDC Power System Statistics 2013-2014: 39th Edition Planning Power NTDC”, Table D 19 EPP quarterly report, April-June 2015 21 management has a priority improvement plan and said they were motivated to continue working with USAID to further improve service and efficiency. According to PESCO, the long length of feeders and high ratio of 11 kV to 400 volt lines, which represent the retail-level service, are the causes of high transmission and distribution losses and some reliability problems. PESCO hopes to reduce the length of feeders and decrease the ratio in future. In terms of future priorities, however, PESCO respondents unanimously emphasized the urgent need to rehabilitate the 3,470 circuit kilometers-worth of 132 kV lines. PESCO attributed 60 percent of their system deficiency to aging 132 kV lines turned over to the DISCOs during the unbundling of WAPDA. Some of the lines are over 50 years old and are considerably older than the more-recently installed 200 kV and 500 kV systems maintained by NTDC. The five-year budgetary requirement for improving the sub transmission grid, including rehabilitating 132 kV lines and reducing the length of feeders, is $120 million. PESCO is not hopeful about receiving the necessary funding in the near future. PDP support to PESCO was limited. According to the ADB, however, the offtake from the national-level Power Transmission Enhancement Project has been slow and the available funding will expire in a year. There remains the possibility of new funding through the ADB for the $1.8 billion Power Transmission Investment Project. Therefore, longer-term outcomes may not be fully attributable to USG support. Conclusions: EPP interventions at PESCO are associated with some initial improvements with localized effects, though not yet in system-wide performance. Even though EPP’s effect on the system overall has been marginal to date, the broader effect of USAID’s programming to support transmission should increase over time as a result of EPP support. PESCO’s expressed motivation for continued improvement, a priority improvement plan, and evidence of initial improvements rather than continued worsening points to it remaining a “turnaround DISCO”, but not yet a DISCO turned around. To achieve its potential and prevent regression given the heavy financial burden of rehabilitating 132 kV lines, PESCO requires additional funding from budgetary support and donors. Transmission Nationwide Including NTDC EPP focused on providing investments in Pakistan’s historically underperforming national grid, which incurred transmission and distribution losses of 21 percent and total system losses (including collections) of 30 percent in 2011 (Table 4). Under Component IV, EPP interventions to support transmission included physical capacity elements (i.e., conducting technical audits, augmenting grid￾network capacity through infrastructure rehabilitation/ enhancement) and human capacity development. 22 TABLE 4: SYSTEM OPERATING STATISTICS Units 2011 2012 2013 2014 Electricity Delivered to Transmission GWh 90,575 89,721 88,270 95,148 Transmission losses % 3.03% 2.82% 3.05% 2.48% Electricity Delivered to Distribution 1 GWh 87,831 87,191 85,578 92,788 Distribution Losses % 18.40% 18.17% 17.64% 17.54% Combined, Transmission and Distribution Losses % 21.4% 21.0% 20.7% 20.0% Billed 2 GWh 71,670 71,348 70,482 76,513 Collections Rate % 89% 86% 94% 90% Net Collected 3 GWh 63,786 61,360 66,253 68,862 Total Loss 4 GWh 30% 32% 25% 28% Source: 2014 Power System Statistics. NTDC 1 This represents the amount delivered to the distribution network for sale to consumers. 2 Billed is the amount delivered to the distribution network less Distribution Losses 3 Net collected is Billed less the non-collected rate (1-collection Collections Rate) 4 Total Loss is Electricity delivered to the distribution network less Distribution Losses less non collected amounts Tables 11 and 12 in Annex 9 document the complete list of EPP transmission interventions taken from EPP reports and augmented by EPP staff interviews along with stakeholder assessments on whether the item was significant within the work plan and an item-by-item rating. Fifteen of the 25 program interventions were completed at the time of report writing at a cost of $17.1 million for the physical and technical assistance components. The evaluation team found no direct overlap with other donors in work plans undertaken by USAID, although, as noted above, funds are available through the ADB loan underlying the Power Transmission Enhancement Project. The evaluation, therefore, cannot attribute outcomes solely to EPP. EPP conducted technical audits of 45 of the 48 grid stations within the NTDC-controlled grid of the transmission system. Technical audits assist in identifying operation and maintenance (O&M) issues, congestion, and worn out equipment. The EPP audits identified congestion points and key requirements for infrastructure maintenance, enhancement, and restoration and included recommendations to the NTDC and NPCC grid system operations and management staff. The items identified by the technical audit will be central to mitigating unscheduled national grid-level disruptions by maintaining and re￾enforcing the existing transmission network. All three respondents that will be affected by the audit, PESCO, NTDC, and the NPCC, stated that implementation of the audit recommendations will ensure capacity, enhance stability and reliability, reduce congestion, and reduce voltage drops caused by overload. It also will serve as a blueprint for future improvement under the pending agreement with ADB for the $1.8 billion Power Transmission Investment Project. NPCC and NTDC have already begun implementing some of the recommendations of the technical audits at Mardan and Nishatabad grid stations and attached a high priority to implementing the audit recommendations in the remaining grid. They believed that a slight expansion in power will push the transmission system to its limit and shared the view that implementing the audit findings is crucial to maintaining existing capacity and enhancing system performance to handle new generation. 23 As a result of policy-level advice from EPP advisors and recommendations from commissioned technical studies, the two managers at NPCC and NTDC saw improved grid-code compliance as one of the key immediate outcomes of the program and complementary to interventions designed to enhance system reliability. National-level data (Table 4) show a slight downward trend in transmission and distribution losses from 21 percent in 2011 to 20 percent in 2014, figures that remain considerably higher than found in many domestic power sectors. The reduction represents about $40 million in savings.20 Not surprisingly, all stakeholders working in transmission identified improving transmission as one of the higher priorities for USAID support. Even three stakeholders from other parts of the sector, including ADB, said that improving transmission is a priority issue. Managers at Tarbela Dam indicated that transmission and distribution problems will be key congestion points as additional generation capacity is developed. Limited nation-wide transmission and distribution capacity could inhibit any benefits that additional generation would provide. Conclusions: The overall effects from USAID’s combined investments in transmission and ADB loans are small but positive, contributing to a slight downward trend in transmission and distribution losses and modest increases in transmission capacity. Transmission losses remain higher than found in other countries and reveal a continuing weakness in the sector. The technical audits will be instrumental in identifying performance improvements associated with active implementation of O&M, identifying congestion points, and replacing and expanding capital. The audits revealed, however, that considerable work and investment is still required to realize a fully functioning and reliable high voltage grid. The pending ADB project will be critical to ensuring that power is delivered to the lower voltage level, but, given slow GoP offtake of ADB loans, technical assistance from USAID could sharply reduce the time in which improvements will occur. Transmission and Distribution Conclusions Stakeholders were united in the view that transmission and distribution constraints are key issues that need to be addressed in the near term. New generation will be pointless without addressing the problems of evacuation and delivery to the customer. EPP started the process by undertaking the critical technical audits. This large and time-consuming undertaking identified many problems. In terms of distribution, the five DISCOs interviewed praised the LLM training. However, the overall success of the training was constrained by EPP’s failure to deliver necessary tools and equipment on time. PESCO managers viewed the interventions as important and said they helped them devise a plan for improving operations. Generation, Transmission, and Distribution – Processes, Procurement, Quality Control, Unintended Outcomes, Changes of Approach The 14 respondents identified few serious process-related problems. All 14 responding institutions indicated that there was a project management unit, and all 8 generation utilities with significant 20 Calculated assuming constant 2014 energy flow levels, 2014 electricity prices, and 2011 exchange rates, 24 infrastructure projects underway or completed displayed evidence of following standard operating procedures, with appropriate internal reporting and quality-control processes in place for procurement.21 Internal need motivated 9 of the 13 institutions to engage with the interventions which imply a sense of ownership. More than half (8 of 14) of the institutions responding, as well as the five DISCOs, identified speed of procurement as a problem. EPP is still accepting delivery and handing over equipment to DISCOs as it closes down. Among the five DISCOs the evaluation team visited, the delays in delivering equipment and vehicles affected human capacity development through the inability to practice complicated new procedures and to reinforce skills from prior training at Tarbela Training Center. Although the evaluation team did not ask a formal question, some stakeholders, including some within USAID, spontaneously noted delays with regard to customs. However, they described the customs delays as short and inconsequential to the overall work process. Otherwise, most delays related to other procurements were minor, implementation proceeded in an organized fashion, and results were achieved mostly on schedule. With regard to the remaining processes, the G2G grants benefitted from existing feasibility studies, and there were no significant changes in implementation after EPP started its work. Satpara Dam was the only G2G grant project that required additional work to achieve anticipated results. No institution within the Gilgit-Baltistan government had ownership of the irrigation system associated with the dam. Consequently, no entity was responsible for cleaning the canals when landslides blocked them shortly after construction. The canals remained blocked thus limiting the agricultural outcomes anticipated from the dam. In addition, some stakeholders considered scaling back construction of canals by six percent. According to EPP, after a high-level meeting in February 2013 that included the Chief Minister of Gilgit￾Baltistan, Chairman of WAPDA, USAID Deputy Mission Director, and EPP, WAPDA agreed to complete the remaining works at the original scale. In either case, the adjustments would not have affected the power-side of the work undertaken. Only five of the 14 responding utilities, all generators, cited unanticipated positive outcomes, mostly in the form of higher than expected capacity and higher than expected savings from the fuel additives study (Jamshoro). Aside from the delays in delivering equipment, the only negative outcome (actually output) the 14 respondents identified was related to the functional inadequacy of the enterprise resource planning system for two GENCOs. Table 12, in Annex 9, shows the results of due diligence. This aspect of EPP was successful. USAID ultimately funded five of 17 energy projects on which EPP conducted due diligence studies. Relevant stakeholders in both the government and USAID valued the studies highly. The ADB also stated that the GoP is using the step one due diligence work conducted by EPP as part of its request for the $11.2 billion Diamer Basha Multipurpose Dam and identified no problems with the work reviewed thus far. Conclusions: In spite of moderate delays related to procurements and minor delays in customs, implementation proceeded in an organized fashion, and the results were achieved mostly on schedule. The evaluation concluded that processes functioned fairly well among the USAID-funded G2G sub￾projects within EPP’s oversight, and in the broader EPP portfolio. Since the G2G grants relied entirely 21 They follow the Pakistan Procurement Regulatory Authority rules and standard equipment specifications. 25 on existing internal management units and processes at the utilities and existing feasibility studies, the lack of significant changes in EPP’s implementation approach after the work plan was set is not surprising. Slow procurement that affected delivery of LLM equipment to DISCOs was the most noteworthy process problem identified in the evaluation, and one of the few variables entirely out of the utilities’ control. The time between training and receiving equipment to implement the skills learned may necessitate additional training to refresh skills. The few unanticipated outcomes were positive and associated with achieving higher-than-expected capacity improvements. The due diligence work was both useful and used by USAID to direct future investments in restoring or adding power to the sector. Overall Success and Conclusions across Evaluation Question 1 A majority (9 of 13) of managers of generation, transmission, and distribution institutions regarded the USAID-funded interventions as high value with the remainder indicating a medium or mixed response. In generation, USAID’s choice of ready-to-fund projects, combined with effective existing controls on the part of generators and EPP, successfully accelerated capacity increases in Pakistan's power sector through G2G grants. Improvements in the transmission and distribution systems have been modest in scale. As the turnaround DISCO, PESCO received considerable focus and support from EPP with highly￾rated studies on transformer oil sampling, technical audits, and reactive power and a telemetering system predicted to reduce technical and commercial losses in the future. PESCO noted reduced unplanned load shedding and improved response time and system reliability at the local level but no significant improvement system-wide in day-to-day operations thus far. In terms of the transmission system, EPP’s effects on system losses and reliability have been small, but the trend is toward progress. EPP successfully supported several constructive, cost-effective investments, such as cooling fans. EPP’s technical audit of the transmission system, which, if fully implemented, can ensure transmission capacity is maintained, enhance stability and reliability, and reduce congestion and voltage drops from overload. The transmission and distribution systems remain weak links in the system. Neither is ready to accommodate the additional electricity associated with the hydro-power and coal thermal projects expected to come online between 2017 and 2021. To several stakeholders, this indicates that upgrading transmission is a key to the revival of Pakistan's energy sector. Human Capacity Building Through its work under Components I, II, and IV, EPP provided stand-up trainings and seminars; on-the￾job training; indirect capacity building of facility staff by including them in rehabilitation and construction; study-tours; and internships. The team interviewed managers of the people trained through formal trainings, seminars, study tours, and capacity building, but did not have time or resources to contact trainees. The evaluation team, therefore, requested trainee post-training evaluation forms to augment its understanding of the immediate responses of participants. These evaluations assessed only trainee satisfaction and did not speak to outcomes such as knowledge, skills, or practices. The team found no evidence that other donors or external partners funded overlapping training, so results are entirely attributable to EPP. 26 Generation, Transmission, and Distribution – Capacity Building22 Overall Table 13, Annex 9 summarizes details of the 12 training and capacity building programs conducted by EPP over the span of the program as well as participants’ responses about the quality of training collected immediately following the trainings. The team had no measure of longer term outcomes of usefulness, use, or skill development, which can be lower than an immediate response to training quality. Of the 1,903 total responses, 1,717, or 90 percent, rated the training as satisfactory or better. The distributions of responses are skewed positively and most are not significantly different from each other. Trainees at thermal plants received trainings in O&M better than trainees at hydro-power plants (p<0.05)23, although the difference is small since most responses are satisfactory or better. Two-thirds of responses are very good or higher. The percentages are statistically the same for the three study tours – commercial, grid and market – 64 percent very positive or 94 percent positive. Most managers (11 of 18) expressed favorable impressions about the value of trainings, study tours, and capacity building. One-third (6 of 18) had mixed views and assigned a high value to some trainings and a low value to others. Three respondents expressed a preference for on-the-job training over stand-up training and the importance of obtaining the input of technical managers. According to the manager respondent, training at NPCC is contributing to making more electricity available to the consumer. Conclusions: The trainings, study tours, and seminars designed for generation, transmission, and distribution was relevant and communicated effectively. Trainees’ and managers reported positive responses about EPP trainings and they expect the knowledge and skills developed by trainees as a result of EPP’s training interventions will improve the operational capabilities of the power sector. Generation, Transmission, and Distribution – Capacity Building, Gender24 The energy sector is known worldwide for underrepresenting women. This is especially pronounced in developing countries. In Pakistan, women represent only 1 percent of the entire workforce in DISCOs.25 According to the trainee evaluations conducted immediately after the trainings, 18 percent of trainees (55 of 305) were female. No information was available to identify what proportion were actually employees of the utilities. Three managers interviewed offered suggestions to increase women's employment in the sector including encouraging their entry into positions in customer service centers and less-remote locations where the distance does not contribute to safety concerns and there is less conflict with home responsibilities. One manager pointed out that their power station did not even have separate toilet facilities for women and suggested improving working conditions for women. Other 22 Table 13, Annex 9 provides the 11 types of trainings plus internships undertaken through EPP. 23 Significance measured using Wilcoxon-Mann-Whitney rank sum test. 24 Although not included specifically within its SOW, the evaluation team assessed gender-related outputs of the EPP as a best practice of gender integration in line with ADS 205 “Integrating Gender Equality and Female Empowerment in USAID’s Program Cycle.” 25 “USAID helps boost gender equity in the energy sector,” 22 August, 2014, accessed 6 August, 2015 at http://www.pdip.pk/recent-activities/gender-equity-training-for-mepco-senior-staff/. See also “Women in energy: bias and barriers,” Al Jazeera, 14, March 2015, accessed 1 august, 2015 at http://www.aljazeera.com/programmes/countingthecost/2015/03/women-energy-bias-barriers-150313141633882.html. 27 suggestions included encouraging women interns to join the power sector, setting quotas, and having power sector managers report centrally on progress towards reaching targets. No other sex-differentiated data relevant to the EPP program were available. The program did not collect data or focus on the effect of intended outcomes by income level or by under-served populations. Conclusions: EPP has made a very modest contribution to helping the GoP reduce knowledge-related gender-based barriers in the sector by including a relatively large proportion of women in its trainings. Outcomes from the training are unknown and are expected to be small without further work to lower barriers. Without the leadership of the GoP or USAID and a clear plan of action to address barriers to women’s participation in the sector, training alone will not make a significant difference in women’s engagement in the sector. Engaging women will require making workplaces more suited to women’s participation, e.g., strategically selecting facilities and operations at which women can be integrated and upgrading facilities to accommodate women, e.g., providing separate restrooms. Governance and Policy Through its work under Component 2, EPP assisted key ministries and the Planning Commission of the GoP by embedding advisors to provide technical assistance to improve policy and governance and to help power-sector institutions transition to independence post-reform. In addition, EPP provided indirect on-the-job capacity building for ministry staff by overseeing their work. This section assesses EPP inputs and outputs by ministry, although the results from one ministry may overlap other ministries. Tables 14-17 in Annex 9 summarize interventions from EPP reports and augmented by EPP advisors and stakeholders’ assessments of whether the assistance was material and central to the plan of work. Highlights include EPP’s work on gas importation and a shale gas policy framework, assistance developing the business agreement between Central Power Purchasing Authority (CPPA) and NTDC, and general support to the MoF. The shale gas policy has the potential to enhance indigenous primary energy resources that are being depleted at an increasing rate. The other key players are the ADB and the private sector. Since the evaluation team directed questions to the results of identified, directly-attributable assistance, EPP is solely responsible for the results. The one staff member interviewed from the MPNR and three senior managers from its technical arm, the Directorate General Petroleum Concessions (DGPC), indicated that almost all assistance received from EPP was materially important and of high value to the work of the ministry (Table 16, Annex 9). The interventions provided by EPP were directly responsible for enabling imports of LNG to Pakistan within the short period of just over a year. EPP advisors were instrumental in developing modified petroleum concession agreements that reflected the new policy and allowed the DGPC to include the licensees in the new policy regime. In addition to LNG, EPP developed the Shale Gas Policy Framework and financed DGPC officials to visit NuTech Laboratories in Houston, Texas where they viewed shale gas samples being tested. EPP also provided assistance to Inter State Gas Systems Limited (ISGS), a public limited company set up by the GoP to diversify gas resources. The major inputs were the LNG Service Agreement and the Implementation Agreement. The supplemental gas agreement developed by EPP resulted in 82 license modifications to bring them under the new gas policy. ISGS included EPP advisors in its negotiation team with Qatar Gas and noted that the advisors had improved the knowledge base at ISGS. ISGS viewed the advisors as essential to its tendering, logistics management, and LNG scheduling and commissioning. EPP supported construction of one LNG terminal and ISGS expects to build two more terminals with 28 the supply and demand potentially rising exponentially. ISGS expressed a need for continued expert assistance to meet the two billion cubic feet per day LNG market potential in Pakistan. Closely linked to the successful ISGS assistance, Port Qasim Authority (PQA) indicated that EPP provided much needed training on Q-flex vessels. The tug boat captains’ training at Siport 21, Spain and at Ras Laffan, Qatar allows the PQA to safely lead and dock ships at the LNG terminal. The implementation agreement signed by PQA and Engro, the terminal owners, is based on the agreement developed by EPP advisors. PQA acknowledged EPP’s efforts, and its website displays the EPP￾developed manual, “Conditions of Use and Standard Operating Procedures.” EPP also provided advice on purchasing tugboats and the port tariff. PQA confirmed the quality and importance of the assistance. Based on these efforts, ADB has given a $30 million loan to support the regasification project at Port Qasim. The diversification and importing of natural gas opened a path to less capital-intensive and less-polluting electric power generation. For instance, the ISGS said that under the aegis of the GoP (and separately from EPP), it intended to set up a 1,000 MW combined-cycle gas turbine generation plant. The large￾scale activity could benefit from technical assistance and support from USAID. Table 15 in Annex 9 examines the work EPP conducted at the MoF. Most of the completed interventions were material and of high value to the ministry. The embedded advisor acted in the capacity of a secretariat to the MoF officers, who had insufficient technical knowledge to manage their work and time. They indicated that the assistance resulted in a number of very well-received papers that were useful and important to the ministry. A key initiative resulted in the ministry successfully defending the GoP’s decision to impose a ‘universal service obligation” surcharge to pay off the power producers’ liabilities before the High Court. If upheld this could help put the system on a path to greater profitability and sustainability. MoF officials considered the assistance of USAID-funded projects on the Circular Debt Study as helpful to them and others26 in calling attention to the mounting problems within the sector; assessing the budgetary requirement for the power sector subsidy; and settling the debt in two tranches – 2013 and 2014. The debt has risen again, although MoF officials indicate it is rising at only half the previous rate. Another important task EPP undertook was the 2013 Pakistan Energy Efficiency and Conservation Bill. The bill established procedures to increase the effectiveness of energy efficiency and conservation. Table 14 in Annex 9 shows EPP interventions with the Planning Commission (PC) which are more limited than those with the MoF because the PC has become less important to planning in the energy sector. The PC viewed the person seconded to it as an assistant rather than an advisor. Of the planned activities, half were completed and perceived to be of high value. EPP provided an integrated analytical tool for forecasting, the Power SIM, to a member of the PC and trained him and staff of MWP and MoF in how to use it. The PC reported that it has begun to use the Power SIM on practice exercises, but the staff did not have sufficient experience with the model to achieve any outputs. However, they indicated that staff will continue to use the tools over the next six to twelve months to gain more experience. 26 Sections were used by NEPRA in its “Policy Report of 2013.” 29 Table 17 Annex 9 examines EPP’s inputs and outputs from its work with the MWP. The MWP rated EPP’s level of success lower overall than the MoF or the PC on most inputs. The three officials interviewed at MWP considered the assistance as less consistent because of turnover among the expatriate advisors who required time to get up to speed and to be helpful. The MWP considered many of the interventions provided by EPP as not material, and those deemed material to their central work plan, generally had medium perceived value. The MWP singled out the Business Transfer Agreement between CPPA and the NTDC as the primary achievement. This agreement will allow an orderly separation of the CPPA (now, CPPA-G) from NTDC, which represents a step towards developing a wholesale electricity market. MWP did not highlight any other EPP assistance as contributing to longer￾term outcomes. As discussed previously, EPP provided an integrated analytical tool, the Power SIM, to a member of the PC as well as selected staff members within MWP and trained them in its use. Multiple staff members at the MWP indicated that while the Power SIM is a useful tool in theory, in practice the ministry is not using it because it is planning to use different software in the future. According to the EPP advisor at MWP, the ministry has not accepted the program, and the MWP has been reluctant to share the data needed to use the Power SIM with the PC because it is concerned that the incorrect use of data has the potential to distort findings. The other two agencies (PC and MoF) expect to continue using the Power SIM model. Overall, stakeholders were satisfied or very satisfied with the assistance provided by EPP advisors. However, one reported that the frequency with which expatriate advisors rotated resulted in periods where learning was inconsistent and advisors less effective. Conclusions: One of EPP’s biggest successes and greatest value-added in governance and policy was in the gas sector, where capacity and knowledge was most lacking. EPP’s efforts led to bringing the first gas imports to Pakistan within a year. Gas diversification will be an important legacy of EPP. The shale gas policy framework also represents an important long-term potential, although continued assistance will be required to realize impacts. Assistance developing a business transfer agreement to separate the CPPA and NTDC is an important step toward developing a wholesale electricity market. However, full utilization of EPP’s assistance will require continued support. Non-USG Investment in Sector Multilateral lenders, such as the ADB and the World Bank have been encouraged by the modest improvements in the energy sector. The ADB representative indicated that the state of the transmission system is only one of many factors and considerations in its decision to continue its public-sector lending and does not directly affect its lending decisions for generation projects. The EPP improvements parallel ADB funding for repairs that have only reached half the loan value for the 2007-2016 period and will be followed by the Power Transmission Investment Program that will provide $1.8 billion and focus on NTDC. Although USAID/Pakistan indicated that its interventions at Jamshoro encouraged ADB to approve the Jamshoro coal extension project, the ADB point-person in Pakistan has only held the post for a year and did not indicate that the EPP work had influenced ADB’s loan decision.27 The World Bank-funded hydro-power at Tarbela had been planned parallel to and independent of the USG decision 27 This may better align with Congressional priorities regarding USG funds. 30 to fund G2G grants there and not as an extension. The evaluation team found no evidence of donor interest at other EPP-supported facilities, such as Mangla. The gas sub-sector has emerged as an area in which USAID’s investment through EPP has begun to leverage private sector investment within a one-year period, specifically Engro’s investment in the LNG regasification terminal at Port Qasim. The ADB also has provided a $30 million loan to the GoP that will make possible the first imports of natural gas. EPP’s support to the NTDC may eventually lead to private sector investment. EPP worked with NTDC to overcome issues of intermittent supply and supported a study on intermittent load which is scheduled to be completed before the end of the project. EPP advisors also provided advice to amend the grid code in a way that intermittency of wind and solar could be regularly communicated to NTDC. Amending the grid code is a long process and will not be completed by the end of the project. Wind and solar projects require lower levels of capital and can come on line more rapidly, and there is some evidence this is happening. ADB has provided $34 million for the IPP development of the Zorlu Wind Farm and the BOOT Gulpar Hydro run-of-river generation plant. In addition, the ADB reported that it has nine renewable energy IPP projects under construction that will eventually add 467 MW to the grid, all of which were initiated in the last two years after the feed-in tariffs for solar and wind were announced. This responsiveness to market conditions points to an area that holds promise for future USAID investment if the goal is to leverage private-sector investment in building capacity and production. Conclusions: USAID’s investments through EPP have been only modestly effective in improving the environment for both international finance institution loans to support government projects in the energy sector and in leveraging private-sector investment. Renewables have seen a slight increase in recent years and represent an area for USAID to consider leveraging future private-sector investment. EVALUATION QUESTION 1.1: PERCEIVED EFFECT OF ADDED CAPACITY Evaluation Question 1.1 – What is the perceived effect of the 1,300 MW added to the grid through USAID energy projects? Taken together, the USAID-funded interventions that EPP helped facilitate through G2G projects (generation) or oversaw (transmission) translated to 1,363 MW of additional capacity to the transmission and generation infrastructure, among other important project outcomes. This figure represents a modest but important contribution to mitigating demand shortfalls. Managers of 17 power sector institutions the evaluation team interviewed were not aware of the exact amount of added capacity as a result of USAID energy sector investments. Eight of 17, including five respondents from government policy and regulatory bodies refused or were unable to offer an estimate of the amount of added capacity attributable to USAID investments. Answers from the nine other respondents ranged from 100 MW (one respondent) to 1,700 MW. Facility-level managers were not as aware of national level program outcomes as managers from energy policy and regulatory bodies, which is illustrative of an information gap between energy policy setting and regulatory bodies and plant or transmission-level 31 developments. The evaluation team found that there was no institutional approach to communicating results to stakeholders and the stakeholders interviewed often had a difficult time estimating how much capacity was added as a result of USAID’s investments. The stakeholders providing responses generally seemed aware of the contribution of EPP interventions. The two primary qualitative outcomes they mentioned included enhanced system reliability (8)28 and reduced load-shedding (4). Two respondents also reported an increase in manufacturing; increased customer satisfaction or positive public perceptions in their service area or plan vicinity; and reduced probability of broad blackouts. The team also looked for available survey data on public perceptions regarding the electricity sector. Time-series data through 2015 that could be parsed by EPP-serviced regions in time for the report were not readily available, although this type of analysis may be possible for future USAID-funded activities with up-front coordination. Conclusions: The absence of a mechanism to communicate intended or actual results left few respondents with a clear understanding of the physical outcomes of USAID’s investments. EVALUATION QUESTION 2: SUSTAINABILITY Evaluation Question 2: 'What is the likelihood that the results EPP has achieved are sustainable beyond the life of the project?' This section references previous findings including factors affecting sustainability. To avoid repetition, Annex 7 contains a more complete list of factors affecting sustainability. Sustainability: Physical Capacity and Financial Although all physical capacity in a power sector is destined to fail at some point, its longevity can be extended or shortened. Longevity, the key element of sustainability of physical capital, is inexorably linked to financial sustainability of the system. An improved, but still unsustainable, financial situation in Pakistan’s power sector will lead to shorter-than-typical lifespan for the physical capacity built under EPP until the system as a whole becomes more sustainable. EPP’s efforts in the governance and policy arenas have contributed in small but meaningful ways to improving sustainability, though longevity of physical capital remains shorter than it could be. The major barrier to overall sector reform and sustainability is financial. Table 4 outlines energy sector performance in its entirety. Production (around 90,000 GWh annually) and transmission losses (around 3 percent per year) have remained consistent through the 2011to 2014 period. Distribution, however, has experienced consistent losses in the 18 percent range, though with a slight downward trend. The 28 National-level survey results on “duration of load-shedding in your locality” showed a sharp increase to 41 percent in the proportion of respondents indicating more than 12 hours per day of load-shedding between June 2012 and April 2013. This proportion dropped to 20 percent by October 2013 matched with a big increase in those with only up to five hours per day rising to 29 percent (the remainder six to twelve hours per day). The proportion across the three categories stayed about even through March 2014 (Gilani Foundation-Gallup Pakistan, 2014). Underlying data was unavailable and not collected in a way to attribute changes to USAID investments. 32 norm for distribution losses is 3-4 percent. Furthermore, collections have averaged 90 percent of billing, while 99 percent is the norm. The overall effect is that between 25 and 30 percent of electricity produced disappears without compensation, a range with enough volatility that it is difficult to determine the validity of the apparent downward trend over the time period examined. The second issue is the price the electricity value chain receives. The last three columns of Table 2 examine the GENCO and Hydro current regulator price relative to an international benchmark. Thermal power plants came closer to full-cost recovery than hydro-power plants under 2015 tariffs, but they are still between $0.03 and $0.06 per kWh29 lower than benchmarks. This means that every unit of electricity sold creates a loss. Replacing heavy fuel oil with natural gas through imports would greatly improve the situation if tariffs remain static since decreases in GENCO’s energy cost will allow production cost to come more in line with their current tariffs. However, hydro-power plants are underpaid from even an O&M standpoint. Outside of EPP, USAID suggested that a portion of the revenue derived from the capacity added or restored at Tarbela Phase II be set aside for O&M. This exception will be helpful in maintaining the capacity built in both Phase II and Phase I. The approach may be challenging to apply universally and does not necessarily result in overall improvements across utilities if the GoP does not set aside O&M funds for all its hydro projects. Components in individual units restored through G2G grants will be sustainable for some time (10-15 years generally) with proper maintenance. However, the duration of their useful lives will depend on sufficient maintenance. Furthermore, other components in some thermal units may fail thereby reducing capacity or production. In addition, the overall plant infrastructure may go down for extended periods due to insufficient funding for maintenance and re-capitalization of worn out major components canceling out part of the gains paid for by USAID. USAID’s investment in PESCO was effective in creating localized improvements from a technical standpoint and some operational improvements. However, managers at PESCO indicated that improvements in outcomes were manifested only at a localized level, while broader improvements in service are too limited to analyze for sustainability. However, these improvements are not likely sustainable without additional external assistance since the DISCO’s turnaround is in the early stages and many weaknesses remain, such as the aging 132 kV lines, that can lead to unplanned outages. Power line losses, power theft, and collections shortfalls are problematic resulting in half of all electricity disappearing without compensation within the PESCO service area. This limits PESCO’s ability to maintain and build on improvements paid for by USAID so that its deteriorating system does not reach a point where it negates any visible localized effects. Furthermore, improvements in the critical areas of commercial operations were insufficient to extend to its overall operations. In terms of improvements in the transmission system, the specific items paid for by USAID that have augmented the grid-network capacity through infrastructure rehabilitation and enhancement are likely to be fairly sustainable well beyond the life of the EPP. Newly-replaced transformers have an expected life of 25 years, although it may be somewhat less in Pakistan due to system-wide problems such as voltage fluctuations. Human capacity outcomes are likely to be sustained if NTDC institutionalizes capacity building activities into its routine practice. 29 The price included the energy change and the capacity payment. 33 The technical audit pointed out that one of the shortfalls of the system was the lack of adherence to the grid code. The technical audits themselves are not sustainable without considerable funding, such as expected from the ADB’s Power Transmission Investment Project, and technical assistance. However, if the audit recommendations are implemented, they will contribute substantially to the sustainability of the grid. In particular, weaknesses in the 132 kV lines managed by DISCOs imperil system reliability. Some lines are over 50 years old and considerably older than the more-recently installed and better￾managed 200 kV and 500 kV systems under NTDC.30 This affects consumers’ perceptions of system￾wide improvements in transmission paid for by USAID. Fuel supply will be critical for future power sector development. The low levels of natural gas reserves inhibit thermal generation expansion at prices that are low enough to be politically appealing to sustain the regulator’s tariff determination. As discussed in the next section, EPP supported the foundation for the first imports of natural gas in March 2015, a major milestone that opens the door to importing natural gas, which will lower domestic energy costs and particularly affect generation. This assistance has not yet influenced gas prices but the sustainability of lower prices would be affected by the competition for the gas among agriculture, transportation, and the energy sector. To the extent that energy gets a significant share of the gas, it could affect the sustainability and growth of the sector through cheaper fuel more closely in line with current tariff rates. Sustainability: Governance and Policy Stakeholders generally valued EPP’s assistance in governance and policy. The most notable contributions to enhance the sustainability of the sector were EPP’s support to MPNR to facilitate gas imports, production, and exploration and to MWP to develop the business agreement to separate CPPA from NTDC which represents a significant step toward creating a wholesale electricity market. Work for the MoF may also contribute to more sustainable outcomes conditional on exogenous decisions. EPP developed work plans with five of six31 ministries involved in the sector. Although ministries might have strong ownership individually, they were often affected by competing internal drivers. This resulted in occasional policy development dead-ends, that were, by definition, unsustainable. This section focuses on those efforts that did not hit such dead-ends. In the gas arena, documents generated by EPP and processes and procedures developed for MPNR, ISGS, DGPC, and PQA, such as the model implementing agreement, standard operating procedures, and ship training are sustainable in that they are in use. Furthermore, skills transferred to local counterparts concerning the process of importing LNG are more likely to be sustainable with current and continued use. In fact, their sustainability seems assured by the planned expansion of import capacity, i.e., a second LNG terminal, which is expected to increase capacity by 500 million cubic feet of LNG per day.32 EPP’s work to establish the foundation for shale energy sources has not yet produced major outputs on which to assess sustainability. Stakeholders have indicated that the policy’s potential impacts are not likely to be sustainable without additional efforts by the GoP and technical support from others. 30 GoP has focused expenditures on expansion of service over maintaining a reliable system. 31 The exception was the Planning Commission, which indicated no formal work plan was generated with EPP. 32 Bhatti, Sohail Iqbal, (26 March, 2015), “First Qatar LNG Shipment Reaches Karachi Port,” Dawn. Accessed 10 August, 2015 at http://www.dawn.com/news/1172013 34 EPP’s assistance to MWP to develop a business transfer agreement separating CPPA from the NTDC is in the process of being formalized and institutionalized through follow-up agreements. These results represent an important step toward developing a wholesale electricity market. EPP’s support to MWP has been impactful, institutionalized, and sustainable. EPP supported MoF by conducting a study on the impacts of the electricity bill surcharge being levied through NEPRA on reducing the circular debt. The surcharge is designed to reduce the cost to the government, which must cover the difference between payments made for generation and payments received from distribution. At this point, the High Court is deciding on the legality of the surcharges. A decision in favor of NEPRA, which was supported by MoF, would be a major victory and improve the potential for sustainability of MoF efforts and the system of tariffs. EPP’s work with the MoF and contribution to reducing growth in the energy deficit has been tangential. EPP’s work on the 2013 Pakistan Energy Efficiency & Conservation Bill may contribute to more sustainable efforts to reduce growth in the energy deficit. Although the Circular Debt Study called attention to the mounting problems within the sector and helped to settle the debt through 2014, the debt has risen again. The rate of increase in debt has decreased by half, but it is not yet at a point that the debt is declining. Overall some elements of EPP’s work with the MoF may be sustained with lasting impact on reform. In terms of EPP’s work with the PC, the Five-Year Development Plan may be useful for forecasting, however, the PC’s lack of centrality to the energy planning process limits the sustainability of EPP’s efforts. The key assistance to the PC with the potential for longer-term sustainability was adapting the forecasting tool, the Power SIM, and training staff members in its use. The choice to house the tool within the PC, the apparent decision by the MWP to proceed with another tool, and the lack of buy-in from the main owner of energy planning processes within the GoP, may compromise prospects for achieving sustainable broader impacts in planning and forecasting. Sustainability: Human Capacity EPP contributed to human capacity building in many ways - formal training through seminars, overseas study tours, internships, and informal training through working with counterparts. EPP personnel reported that they used a variety of methods to assess the impact of their training efforts, including trainee evaluations completed immediately after the training, follow-up surveys to assess longer-term changes in knowledge and practice, and focus group discussions with training participants six months after trainings. EPP also reported that prior to receiving training, participants are required to formulate an 'Action Plan', highlighting the steps the participant will take after completing the training program. The evaluation team did not receive documentation of completed action plans or the results of follow up surveys or focus group discussion sessions. The evaluation team reviewed immediate post training evaluations completed by trainees and interviewed managers to obtain evidence of the degree to which EPP’s training efforts have enhanced the capacity of trainees and been institutionalized. O&M Training Participants gave thermal O&M training high ratings, and, if given the necessary funds to execute what they learned, the knowledge will be sustainable. However, outside of Tarbela Dam after Phase II construction is completed, funding for O&M training in unlikely to continue in the publicly-operated generation plants. Utility managers generally felt that they are able to follow international best practice in O&M and rehabilitation (9 of 12), and some reported that they could do so prior to EPP's intervention. It is unclear if EPP had formal mechanisms for passing along training to others who did not have the opportunity to participate in training activities. However (7 of 9) managers indicated that 35 participants of the O&M training trained others in their sections after receiving the training. Furthermore, of the three generation facilities that received training and responded, all indicated that some or all of the trainees would remain in their positions for extended periods. EPP’s scope did not include work on pre-service training, and no new training center was established or refurbished for O&M training for generation facilities.33 Rehabilitation Training Managers in all the supported utilities reported mixed levels of involvement with the rehabilitation work itself, which may be an indicator of greater sustainability. Half (6 of 12) opted for having the original equipment manufacturer undertake the rehabilitation, a third (4) could perform the work without oversight by the end of the program, and the remainder (2) performed the work themselves with oversight. All 11 energy utility managers interviewed expected that many or all those trained will remain in their positions for the immediate future. However, they said that some trainees will rotate to or be promoted to non-relevant positions within the same utility or related utilities. LLM Training EPP provided LLM training to nine DISCOs and reported that five out of nine DISCOs received tools. During the interviews, the managers consistently praised the quality of LLM training. However, only PESCO is confident that the skills learned are sustainable because they received both tools and training. Four out of five DISCOs interviewed reported delays in receiving tools, parts, and equipment and mentioned that without tools and equipment those who received training were not able to practice what they learned to cement the skills acquired through training. Moreover, DISCOs with insufficient funding for O&M also raised concerns about how trainees will retain newly acquired skills without practice unless DISCOs continue LLM training. EPP mitigated the effect of the delays in part by rolling out LLM refresher training courses in quarter 2 of FY 2015. At the conclusion of the courses DISCOs are supposed to receive tools. The first DISCO to complete training was IESCO in/about April 2015. The remaining DISCOs were scheduled to receive refresher trainings in the subsequent quarters. The refurbished NTDC Tarbela Training Center can institutionalize linemans’ capacity building to perform LLM. EPP reports that monitoring and evaluating functional crews is currently the primary mechanism for ensuring sustainability. Power SIM Training The Power SIM is new and not yet being used for planning. The model reflects the existing system and will need to be updated to retain its relevance and utility as the sector evolves. To the extent that the PC and MoF staff practice and use the model, the skills may be sustained without external support until changes in the sector render the model obsolete, perhaps in three to four years. If the PC and MoF invest in updating and adapting the model, it could be sustainable for a longer period of time without external assistance. However, given staff rotation, some external support will likely remain necessary. Because the MWP may choose to use a different model, the skills acquired by MWP staff in using the Power SIM may not last. This conclusion will depend on whether the MWP chooses to use an alternative model. 33 This was not asked as a formal question. 36 Informal Training through Work with Counterparts The sustainability of capacity built through day-to-day interaction with counterparts is limited by the relatively rapid rotation of civil service employees due to the GoP civil service and appointment system. All five of the EPP-supported ministries concerned with governance and policy expected regular staff turnover. Coupled with the lack of technical criteria to determine postings, it is doubtful that any of the knowledge transferred will be sustainable without continued donor assistance to train new employees. The evaluation team found no evidence of organizational structures to facilitate or improve continuity of reforms. Working with local partners through a process such as developing the LNG plant and terminal can build capacities and transfer skills. However, managers from DGPC indicated that they are not in a position to sustain policy initiatives without additional assistance, and no institutional system to transfer knowledge and data exists. To the extent that political will exists to import natural gas it could bolster sustainability of skills built in the gas sub-sector. Internships From the information available to the evaluation team, it is unclear what capacity EPP was able to build within the sector through internships. However, EPP reports that it placed 130 interns in 17 public energy sector entities throughout the country and over 20 percent of these interns have been women. EPP also reports that 35 percent of graduates of EPP’s internship program are employed full-time and 11 percent of those are female. EPP also reported that the internship program will continue after the close of EPP under the auspices of another USAID implementing partner. Sustainability: Conclusion The financial problems in the Pakistan power sector will in large part determine the longevity, and thus sustainability, of physical capital. The country cannot expect to lose 28 percent of production without compensation and expect critical infrastructure to be maintained or sustainable. Poorer functioning DISCOs exacerbate the problem, which, in the case of PESCO, loses half of the electricity it receives. In this context, each unit of production results in further net losses and the additional capacity funded by USAID ironically leads to higher financial losses since the GoP loses money for every kWh produced. An unsustainable financial situation limits the ability to fully fund O&M for routine maintenance or capital-intensive restoration. The result is shorter than typical lifespan for physical capacity. By helping close the tariff gap and halving the rate of losses, EPP’s support for governance should modestly increase O&M spending and extend the lifespan of capital equipment. However, higher tariffs and lower rates of loss will be required to end the circular debt cycle and allow substantial increases in O&M spending. The agreement to set aside a portion of the revenue derived from USAID-funded capacity at Tarbela Phase II for O&M may help sustain USAID’s investments. Given the limited duration of USAID support to PESCO, interventions will have limited sustainability while PESCO continues to operate so poorly. EPP’s work in the policy arena produced a few results that have been utilized and institutionalized and are thus more likely to be sustained and contribute to the sustainability of physical capital. The documents, processes, and procedures EPP produced in the gas sector point to a sustainable path to increasing LNG imports and sustaining the skills transferred to counterparts. The business transfer agreement separating CPPA from the NTDC is being formalized and institutionalized and represents a step toward a sustainable wholesale electricity market. The shale energy policy framework, however, will require additional technical assistance to generate outcomes that could be sustainable. Prospects for sustainable outcomes from the Power SIM forecasting tool are unclear since few people have the 37 expertise or experience to use it, the tool requires periodic updating, and the MWP is leaning toward an alternative tool. Broad and sustainable improvements in human capital require investments within the system itself. Frequent rotation in technical positions may also render skill development irrelevant. The financial situation in the sector has prevented such investment from occurring consistently, and the civil service rotation process inhibits specialization of skills in key ministries. In the utilities, the skills built through training and oversight of rehabilitation and maintenance may require less additional assistance among hydro-power plants than among thermal plants to achieve sustainability. Most of the plants themselves are attempting to broaden and sustain the knowledge built independently of EPP. O&M training for generation utilities has moderate prospects for sustainability depending on the level and timing of additional inputs. LLM training for DISCOs will require additional support by EPP or afterwards to become used and institutionalized. Skills built among the ministries are more tenuous partially due to external factors and lack of staff to pair with EPP expat advisors. Further, in cases such as the MPNR, the emphasis on meeting milestones may have led to insufficient skill transfer to sustain progress without additional external technical assistance. OVER-ARCHING CONCLUSIONS USAID interventions in physical capital, governance, and human-resources that EPP either implemented or, in the case of the direct G2G grants, facilitated, led to increased energy supply for the units and locations affected. The mid to long-term sustainability of the interventions may be less than that expected in countries further along in power sector reform, largely due to the system-wide unsustainable financial situation. Longer-term sustainability rests on the financial balance of the system. An EPP work plan with activities in many areas may have contributed to diffuse and often limited effects outside of facilitating direct G2G grants. However, this approach may also have allowed EPP to identify key areas – notably the gas sector – where inventions have resulted in tangible, innovative, and significant achievements with the potential for sustainability and significant future impacts. 38 RECOMMENDATIONS GENERATION AND FUELS USAID should consider leveraging its successes in the gas sector by continuing to facilitate imports and domestic transportation of liquefied natural gas. Doing so can lower costs and reduce the greenhouse gas emissions trajectory by crowding out coal in electricity generation. This could support infrastructure development and bolster technical capacity to increase imports. In particular, support will be required for the recent tender won by Akbar Associates to construct a land-based LNG terminal. If gas imports increase sufficiently and the share of new imports is prioritized for energy, it may be advisable for USAID-funded activities to work with the GoP to convert existing power generation facilities away from expensive (and higher greenhouse gas emitting) heavy fuel oil. From an operational standpoint, USAID could ask its implementer to work with the GoP to rationalize the gas market and keep it competitive. Among several options is helping set up a market with numerous regional gas companies and empowering a separate company to supervise development and maintenance of the pipeline. Given both its success under EPP and U.S. leadership and technological edge related to shale gas extraction, USAID should consider efforts to support shale gas development, perhaps by supporting studies of available reserves or facilitating initial private sector investment. Access to shale energy resource could enhance Pakistan’s energy security. To the extent that USAID chooses to continue supporting G2G investments in the generation sector, it should identify high-impact, low-cost investments and, for thermal plants, target base-load units for maximum effect. Consider funding low-cost, high-impact demand-side interventions to increase energy efficiency in physical stock such as industrial electric motors. The mission could also consider demand￾reducing renewables such as solar rooftop photo-voltaic panels, which is part of the USAID portfolio in India. This may require support to develop enabling policies. With pending increases in generation, USAID can build on EPP successes by helping update fuel supply and power purchase agreements and continuing to encourage their use. USAID might consider working with the Private Power and Investment Board to update its power purchase agreements. As a near-term intermediate step toward proper O&M budgets, USAID should consider ways to follow￾up its precedent with Tarbela, Phase II of linking major renovations, whose effects are easy to isolate to agreements, to establish plant-specific O&M funds. In the long-run, segregating O&M funding based on the additional capacity restored or added should be an exception that will not be universally applicable. USAID might fund an analysis to assess the extent to which ring-fenced funding could lead to higher circular debt, or other possible outcomes. TRANSMISSION AND DISTRIBUTION Since stakeholders identify transmission and distribution as major current or impending system limitations, especially as new generation units come on line in 2017-21, USAID should prioritize these areas. It can do so by encouraging and supporting implementation of the technical audit recommendations. Investments should prioritize the highest impact activities that will be slow to emerge 39 under the ADB’s current loan and upcoming $1.8 billion Power Transmission Investment Project, as the facility will provide funds to purchase expensive equipment and infrastructure. Addressing congestion issues will likely be a high priority. USAID’s next project could conduct studies to address key weaknesses in the transmission system that are not covered by the technical audits (e.g., load flow study to avoid fluctuating voltage). General revision of the grid code is a long process that has begun under EPP but will not be completed by the end of the project. USAID should continue supporting revision of the grid code including improvements to provide incentives for distributed generation such as wind and solar. Given the still￾modest but increasing volume of renewable energy coming to the grid in the near future, USAID should continue supporting intermittent load flow studies. As the NTDC is unbundled, USAID should support the development of market rules including possibly eliminating the requirement that all electricity be sold to the CPPA. Continue to encourage the separation of NPCC from NTDC so that an independent market operator is formed. Wholesale electricity markets will allow electricity buying and selling outside the single-buyer regime which is currently in place. Supporting the current number of DISCOs and continuing to provide live-line maintenance training represents a low-cost and needed intervention to reduce planned outages early in the follow on project. USAID also should encourage the continued use of the revitalized training facility by supporting continuous training for linemen to enhance their skills. Losses are one of the major threats to the financial sustainability of the Pakistan electricity sector. USAID technical assistance should include helping DISCOs determine the true extent of technical and commercial losses and developing plans to overcome the problem. The initial, though localized, success at PESCO as a ‘turnaround DISCO’ combined with a politically willing partner and established relationship suggests a continued focus on PESCO. Broadening the turnaround lessons to other problematic DISCOs and broader support in DISCO O&M should be a next step. GOVERNANCE, POLICY, HUMAN CAPACITY DEVELOPMENT, AND OVERALL The governance and policy arena remain clear areas needing support and areas in which USAID has experience and a comparative advantage. USAID should consider supporting areas including gas-related policies and agreements, subsidy targeting, amendments to the Electricity Act, and policies supporting improvements in DISCOs’ commercial operations. These topics are of major importance and should be finalized in the near term. Although the success of policy-level work depends on external factors and is far from certain, it is relatively inexpensive and can have considerable impact when successful. Interviewees frequently requested additional training outside the traditional areas of engineering technical capacity, such as commercial activities, finance, accounting, human resources, and IT. USAID might consider training on project management for Mangla, since it is one of the new utilities without prior project management experience and is facing a large work plan. USAID should assess the need among DISCOs, NTDC, and NPCC for broader use of and skills with the enterprise resource planning tools required to operate large organizations. At the DISCO level at the very least, USAID should support investment in a basic customer information and billing system with telemetry capabilities. This information will be critical to audit deliveries at the retail 11 kV feeder level 40 and the actual amounts billed off that feeder. Accomplishing this would be a major step in auditing the system and finding and solving the causes of loss. Training in best practices in geographic information system-supported network planning, analysis, and asset management may provide a stimulus to reform. USAID should oversee its upcoming activity to ensure buy-in from as many affected ministries as possible, paying special attention to the process owner and independently verifying significant decisions with key stakeholders as often as feasible. Given the strong link between financial sustainability and the sustainability of USAID’s investments in physical and human capacity, the agency should support NEPRA to continue linking tariffs, not just to full-cost-recovery rates, but also to energy efficiency demand-side interventions. It could also review and support business plans developed under EPP as a check that all elements of full-cost recovery have been considered. USAID could investigate providing governance-level support to the Alternative Energy Development Board. Partial risk guarantees such as Development Credit Authority guarantees could increase the availability of funding for non-conventional generation. New thermal power plants are characterized by long lead times, making it difficult for typical USAID contracts to bring energy on line by encouraging private sector investment. Hydro-power plants require even longer lead times. Wind and solar projects can come on line considerably faster. An already-notable increase in renewables suggests an avenue for USAID to encourage private-sector investment. One of the key shortcomings within the civil service system within ministries is the rapid turnover of management. When managers with little technical knowledge or experience are assigned to a highly complicated and technical sector, policy reforms can get waylaid by lack of experience and expertise among key ministry staff. A goal of future USAID interventions in the sector might be to assess areas where the GoP is willing to support changes in staffing policies in energy-related ministries, creating small technical cells within ministries, or the seconding of a technical manager from elsewhere in the public sector to key ministries for a period of time. A related area involving longevity on which USAID may have somewhat greater leverage is the secondment of technical advisors from USAID within ministries. Rotating expatriate staff in a country like Pakistan can present a challenge and prohibit developing a sufficient understanding of ministry needs to be productive. USAID should work with the next implementer to consider whether to provide locally-based advisors to pair with the expatriate advisors to have greater longevity in the program of work by embedded advisors. Given the small number of women in the energy sector, USAID should undertake an assessment and work with the GoP to expand employment opportunities for women including creating connections between power companies and universities to place women in jobs, helping the GoP monitor and track progress toward its goals, working with power plant managers to take practical steps to accommodate and increase women’s employment starting with providing appropriate facilities at targeted locations and expanding opportunities for training. In terms of general project management, USAID should require implementers to follow all government rules and regulations, such as registering in-country to avoid problems with registration. USAID also should consider mechanisms for publicizing the goals and successes of its current and future energy￾sector projects with key stakeholders including a communication element within the contract and an end-of-project workshop in-country. 41 ANNEXES 42 ANNEX 1: EVALUATION STATEMENT OF WORK ENERGY POLICY PROGRAM (EPP) END-OF-PROJECT PERFORMANCE EVALUATION EVALUATION STATEMENT OF WORK MARCH 5, 2015 This publication was produced for review by the United States Agency for International Development by [insert Author’s Name(s) here]. It was prepared by Management Systems International (MSI) under the Monitoring and Evaluation Program (MEP). 43 ENERGY POLICY PROGRAM (EPP) END-OF-PROJECT PERFORMANCE EVALUATION EVALUATION STATEMENT OF WORK Contracted under Order No. AID-391-C-13-00005 Monitoring and Evaluation Program (MEP) DISCLAIMER This study/report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of Management Systems International and do not necessarily reflect the views of USAID or the United States Government. 44 CONTENTS I. BACKGROUND INFORMATION ................................................................................................ 46 II. RATIONALE FOR EVALUATION ............................................................................................. 54 III. EVALUATION DESIGN AND METHODOLOGY .................................................................. 55 IV. TEAM COMPOSITION ............................................................................................................... 60 V. EVALUATION MANAGEMENT ................................................................................................. 60 ANNEXES ..............................................................................ERROR! BOOKMARK NOT DEFINED. Annex 1: Report Content (Adapted Agency Standard Language) ................................................................... 66 Annex 2: Table of Evaluation Questions by Data Sources, Collection, and Analysis Methodologies .......................................................................................................................................................... 68 Annex 3: Reporting Guidelines (Agency Standard Language) ......................................................................... 70 Annex 4: List of Available Documents ................................................................................................................ 71 Annex 5: Mission Strategic Framework Development Objective 1 Results Framework ............................. 72 Annex 6: Energy Policy Program Map ................................................................................................................. 73 Annex 8: List of EPP Stakeholders ....................................................................................................................... 74 Tables and Figures Table 1: Project Summary................................................................................................................................................47 Table 2: Proposed Sample for Individual Interviews ..................................................................................................56 Table 3: Tentative Evaluation Schedule ........................................................................................................................62 Table 4: Anticipated Level of Effort..............................................................................................................................63 45 ACRONYMS AEAI Advanced Engineering Associates International, Inc. AEB Area Electricity Board AJK Azad Jammu and Kashmir COR Contracting Officer’s Representative DISCO Government-Owned Power Distribution Company DO Development Objective FATA Federally Administered Tribal Areas EPP Energy Policy Program G2G Government to Government GENCO Power Generation Company GoP Government of Pakistan IFI International Financial Institutions IMF International Monetary Fund IR Intermediate Result IRG International Resources Group LNG Liquefied Natural Gas MOF Ministry of Finance MPNR Ministry of Power and Natural Resources MSF Mission Strategic Framework MSI Management Systems International MW Mega-watt MWP Ministry of Water and Power NEPRA National Electric Power Regulatory Authority NOC No Objection Certificate NPCC National Power Control Center NTDC National Transmission and Dispatch Company OAPA Office of Afghanistan and Pakistan Affairs PC Planning Commission PEPCO Pakistan Electric Power Company PESCO Peshawar Electric Supply Company PIRS Performance Indicator Reference Sheets PML-N Pakistan Muslim League-N PMP Project Monitoring Plan SOW Scope of Work SRAP Special Representative for Afghanistan and Pakistan TA Technical Assistance TO Task Order TPM Team Planning Meeting TUTAP Turkmenistan-Uzbekistan-Tajikistan-Afghanistan-Pakistan interconnection project USG United States Government USAID United States Agency for International Development WAPDA Water and Power Development Authority (WAPDA) 46 I. BACKGROUND INFORMATION A. Identifying Information about the Project/Activity The Energy Policy Program (EPP), which is implemented by Advanced Engineering Associates International, Inc. (AEAI) aims to support economic and technical studies related to power generation, transmission, fuel supplies, and load shedding. Examples include developing policies and procedures for the National Power Control Center (NPCC), developing business plans for Generation Companies (GENCOs), and providing advisory services at the Ministry of Water and Power (MWP), the Ministry of Finance (MOF), the Energy Wing of the Planning Commission (PC). In addition, it provides support to the Government of Pakistan's (GoP) policy reform activities. This includes advising the ministries of Water and Power, Finance, Petroleum and Natural Resources, the Water and Power Development Authority (WAPDA) as well as the Energy Wing of the Planning Commission on power sector issues such as shale gas development, policy for enhanced gas production, and the import of liquefied natural gas. Finally, it assists the Peshawar Electric Supply Company (PESCO) reduce transmission losses through transmission line and grid station repair. Through this work, the project aims to model the impact of a more efficient transmission system to power entities and work with the National Transmission and Dispatch Company (NTDC) to improve the capacity of their transmissions system by providing necessary technical assistance and commodity support. EPP provides monitoring and implementation support for seven ‘Signature Energy Projects:’ (a) repair, rehabilitation and maintenance of four power plant projects (three GENCO thermal power stations and one hydroelectric plant – Tarbela Dam), with total funding of $70.8 million; (b) completion of two multi-purpose dam projects (Gomal Zam Dam and Satpara Dam, with total funding of $71.0 million; and (c) a seventh power sector activity, the Mangla Dam Rehabilitation Project. The attached map in Annex 6 shows the location of the seven ‘Signature’ projects. EPP also provides technical and financial analysis support for identifying and selecting new generation and transmission projects for potential USAID funding. Examples include the Turkmenistan-Uzbekistan-Tajikistan-Afghanistan-Pakistan (TUTAP) interconnection, under which electrical power from the Central Asian Republics (CAR) could be transmitted to Pakistan through Afghanistan, and due diligence for the Diamer Bhasha, Kurram Tangi, Warsak and Mangla hydroelectric dam projects. Under this component, infrastructure projects are selected based on: (a) technical feasibility; (b) cost; and (c) overall beneficial effect toward achieving joint goals of the United States Government and the Government of Pakistan (GoP) for the power sector. EPP responds to chronic energy shortfalls that are slowing Pakistan’s march toward development and posing an existential threat to its economic and political stability. Broadly recognized as a national crisis, the shortfalls are experienced throughout the country and borne at all levels of socio-economic strata. Intermittently, for several decades, USAID has supported Pakistani efforts to resolve the crisis. Most recently, USAID has provided support under the 2009 Enhanced Partnership for Pakistan Act (EPPA – also known as the Kerry-Lugar-Bergman Act). By the end of 2014, this support has helped add and save over 1,400 megawatts (MWs) to the energy system, increase annual revenues at electricity distribution companies (DISCOs) by over $200 million each year, and bring about significant improvements in system-wide efficiency, cost-effectiveness, and cost recovery. 1. The Development Context of the Results Framework USAID/Pakistan's Mission Strategic Framework goal is "increased stability, democracy, and prosperity for the men and women of Pakistan,” which will be difficult to reach without sufficient gas and electricity available to households, businesses, and factories. To address this challenge, one of the Mission’s development objectives is “increased sustainable energy supplied to the economy." This will be accomplished 47 by increasing the available energy supply and by improving energy sector governance and management. USAID is conducting large-scale energy projects in partnership with the GoP to add MWs to the power system, to improve governance and management, and to assist the GoP with the promotion of energy policy, pricing, and regulatory reforms. TABLE I: PROJECT SUMMARY B. Development Context 1. Problem or Opportunity Addressed Pakistan has a chronic shortage of electricity, causing rolling blackouts that often last 10-12 hours per day. These blackouts constrain economic growth and development, disrupt health, education and other services, and undermine the viability of the elected government. Only 55 percent of the population has access to modern energy services, and 64 percent of public schools have no electricity. Unreliable electricity service also constrains refrigeration services that are vital to the distribution of medicines and perishable foods. This affects both public health and nutrition. Electricity shortages have been caused by high, untargeted subsidies that benefit a few, low tariff collection rates, distorted pricing, high rates of transmission and distribution losses, and widespread corruption in public entities. Combined, these factors serve as a disincentive for private investors (Independent Power Producers) to build additional capacity. The power sector’s debt – even though temporarily eased through a mid-year payment in 2013 and a recent infusion of US$550 million into Pakistan’s power sector– will grow once again, contributing to an increased budget deficit and continued borrowing from the Central Bank of Pakistan. This cycle of borrowing, called the “circular debt” has crowded-out private sector lending. To encourage additional investment in the sector, the GoP must rationalize pricing to enable power suppliers, power generation and power distribution companies to recover their operating and investment costs. The GoP must also address power line losses and power theft, both of which increase costs to legitimate users. While the GoP may find implementing such reforms difficult in the near-term given the current political climate, inaction will only exacerbate these problems over time. Title/Field Project/Activity Information Contract/Agreement Numbers AID-391-TO-12-00002 Contracting/Agreement Officer’s Representative (COR/AOR) Saeed Anwar (COR); Rabia Bukhari (alternate COR) Start Date February 12, 2012 Completion Date October 15, 2015 Location Nationwide Name of Implementing Partner (IP)  Advanced Engineering Associates International, Inc. (AEAI) USAID/Pakistan Mission Strategic Framework Linkages Development Objective 1: Increased Sustainable Energy Supplied to the Economy Intermediate Result 1.1: Increase Energy Supply Intermediate Result 1.2: Improved Energy Sector Governance Budget $80,283,410 (Technical Assistance - TA); $291,710,000(Government-to-Government oversight) 48 USAID, International Financial Institutions (IFIs) and other bilateral donors provide various forms of assistance to help the GoP develop appropriate energy pricing, regulatory and privatization policies, and implementation assistance that promotes energy sector reforms. These efforts promote expediting energy sector reforms and soliciting sufficient investment in electricity generation, as well as a more appropriate and affordable fuel supply mix that will help resolve the power crisis (as noted in the U.S.-Pakistan Strategic Dialogue and the Friends of Democratic Pakistan Task Force Report). From a management perspective, Pakistan’s power sector should be in a state of transition from government￾owned and managed to fully autonomous, where companies operate independently with regard to their ability to purchase, generate, transmit, dispatch and distribute electricity. Traditionally, the power sector has been run as a monolithic organization under WAPDA, whose Power Wing provided the line and functional control of the Distribution Department, directing the operation of eight area electricity boards (AEBs) across the country. In 1998, under power sector reforms, WAPDA was restructured to separate generation, transmission and distribution. The bulk of power generation was assigned to generation companies (GENCOs), leaving WAPDA’s responsibilities limited to hydropower and water dams only. At the same time, AEBs were converted into stock companies called electric power distribution companies (DISCOs), with all shares held by the GoP. The power sector reforms also established the regulatory agency, National Electric Power Regulatory Authority (NEPRA) and the Pakistan Electric Power Company (PEPCO), to supervise the transition to full autonomy. Sixteen years later, the “transition” continues, and autonomy remains an objective rather than a reality. Given the energy crisis outlined above, EPP contributes to the USAID/Pakistan Mission Strategic Framework’s Development Objective 1: Increased Sustainable Energy Supplied to the Economy. In particular, EPP aims to support the GoP in adding and saving megawatts (MWs), decrease transmission and distribution losses by investing in selected energy infrastructure projects and new technology, and improve governance by supporting GoP reform efforts with technical assistance. Through increasing the supply of energy and removing key governance barriers, EPP supports improvements in Pakistan’s economic development 2. Target Areas and Groups The project has worked with the key Government Ministries and Nine Distribution Companies with major focused efforts to improve the transmission network of PESCO (Peshawar Electric Supply Company). The ministries are Federal level and Islamabad based that include Ministry of Water and Power, Ministry of Petroleum and Natural Resources, Ministry of Finance, and Planning Commission. EPP also worked with National Transmission and Dispatch Company (NTDC) in Lahore and Peshawar Electric Supply Company (PESCO) in Peshawar for improving Transmission system. In addition, EPP provides monitoring services for USAID Government to Government projects to inspect and verify. The activities were designed to help the GoP entities for reducing losses and increasing revenues. Targeted population for each area differs on the size of area each entity serves. Most of energy work is national level and falls into provinces as well. 49 C. Development Hypothesis USAID/Pakistan’s development hypothesis is that by improving the management of the energy sector and increasing the availability and reliability of energy supplies, the country and its economy will stabilize and thereby afford the development of democratic institutions and processes, leading to prosperity and an improved quality of life for all citizens. Critical assumptions underlying the Results Framework are that:  The energy crisis and load shedding will last another 5-10 years keeping in view the current level of effort on the part of the GoP;  Political interests are interfering with the progress;  Policy reform is essential to tackle the issues being faced and the GoP is willing to adopt the reform agenda;  USAID support, in collaboration with other donor programs, can create the political will needed to have an impact on Development Objective 1 after 6-7 years;  The energy projects implemented by the GoP either using its own resources or through the support of donors will be completed/implemented as scheduled;  There are no major changes in GoP priorities in the post-election scenario;  The GoP demonstrates the political will to withdraw from running electricity companies; 50 See Annex 7, EPP’s M&E Plan, for the project/activity logical framework/results framework for more details. D. Intended Results EPP contributes to the Mission Strategic Framework’s Development Objective 1 – “Increased sustainable energy supplied to the economy.” The Intermediate Results applicable to EPP’s work are: IR 1.1: Increased Energy Supply IR 1.1.1: Increased Generation and Transmission Capacity IR 1.1.4: Increased Non-U.S. Government Investment in the Energy Sector IR 1.2: Improved Energy Sector Governance IR 1.2.1: Improved Policy Implementation IR 1.2.2: More Autonomous Energy Sector Entities IR 1.2.3: Improved Capacity of USAID-Supported Energy Public Sector Entities E. Approach and Implementation EPP is structured into four, interrelated components: Component I – Monitoring and Support of Project Implementation EPP proactively advises and supports USAID and counterparts in the monitoring, implementation, and compliance of Fixed Amount Reimbursement Agreements (FARA) with the GoP. This includes monitoring and supporting timely counterpart compliance with all of the terms and conditions of FARAs, and inspection and certification of deliverables. Specifically, EPP provides monitoring and implementation support for seven signature energy projects: four repair and maintenance projects (3 GENCOs and Tarbela); two multipurpose dam completion projects (Gomal Zam and Satpara); and one dam rehabilitation project (Mangla). Component II – Advice and Support of Energy Sector Policy Reform EPP actively supports energy sector reform by undertaking activities that are requested by one or more GoP entities and by the direct secondment of staff to work in the offices of the requesting entity as advisors and specialized support staff. EPP advisors support the ministries of Water and Power, Finance, Petroleum and Natural Resources as well as the Planning Commission’s Energy Wing (PC), National Transmission and Dispatch Company (NTDC), and National Power Control Center (NPCC). In achieving the objectives of this component, EPP supports generation and transmission-related studies and policy reform activities assigned by USAID with the support of the GoP. Finally, this component supports generation, transmission, and oil and gas-related studies and policy reform activities assigned by USAID. Component III – New Projects, Planning and Development Based on frequently changing priorities and schedules of USAID and U.S. Government entities, EPP undertakes due diligence of projects and activities that are candidates for U.S. Government support. Specifically, EPP conducts and prepares due diligence reports on new projects under consideration for U.S. Government support in order to provide detailed information, reduce U.S. Government risk, and set a basis for creating Project Implementation Agreements for the selected projects. 51 Component IV – New Activities The EPP’s original contract was signed in 2011. An expansion in EPP’s original scope was introduced and signed in September 2012. New Activities and the expanded scope were identified to respond to the Government of Pakistan’s efforts to address the current energy crisis in Pakistan—e.g., severe load shedding that severely affects Pakistan’s economy. Overall EPP has had seven modifications since its inception. The objective of the modifications varied from providing incremental funding every year to including/changing some language/clauses in EPP’s scope of work. As part of this expanded effort, EPP will focus on all of the new activities, as well as continued activities from all components, to support the results identified in the results framework, with the overall objective currently stated as, “Increased Energy Supplied to the Economy”. As a result of Modification Number 2 to the EPP Task Order Award, EPP supports the following efforts: 1. Support to the poorly-performing “Turn-around” DISCO (Peshawar Electric Supply Company (PESCO); 2. Expanded support for governance, that focuses on implementation of high-impact, improved governance activities that will benefit the electricity sector and end-users, including fuel mix diversification from imported power generation resources; 3. Live Line Training: a sustainable, targeted training program for selected crews of linemen on best practices and techniques for repairing live power lines; 4. Natural Gas: targeted technical support to MPNR in upstream oil and gas activities, including policy and regulatory oversight, resulting in increased supplies of domestic and imported gas (including LNG and shale gas) for power generation; 5. Support to NTDC and NPCC to mitigate unscheduled power outages by providing technical assistance, including power imports support, transmission system reviews, assets, products and services enhancements, and capacity-building services; 6. Support to GENCOs and WAPDA, consisting of technical assistance to improve finances, add MWs to the grid, and improve fuel efficiency and management operations. New activities and an expanded scope have been identified in a policy matrix developed by USAID. These policy priorities include: 1. Increasing Circular Debt; 2. Insufficient supply of Affordable Electricity affordable electricity; 3. Inefficient Power Sector Operations power sector operations and Excessive Sector Energy Losses excessive sector energy losses; 4. Poor governance and management of Public Energy Sector Entities (PPESE); and 5. Excessive Peak Demand peak demand in the summer months. 52 F. Current Status of Activities Results of EPP to date include: 1. Upstream Oil and Gas Support: EPP’s day-to-day support to MPNR, Directorate General of Petroleum Concessions, Port Qasim Authority and other upstream-related GoP entities expedited the negotiations that resulted in the timely settlement of the LNG Services Agreement (LSA) and the Implementation Agreement (IA) – key objectives of both the Government of Pakistan and the United States in mitigating Pakistan’s energy crisis 2. Commercially Sustainable Energy Sector: EPP’s policy reform support to GoP entities has helped improve the financial performance of energy sector entities by $526 million (≈6.0 percent of 2012 circular debt) 3. Cross Cutting Activities: EPP’s Cross Cutting and Capacity Building interventions have resulted in 65 GENCO employees from Jamshoro, Guddu, and Muzaffargarh being trained on best practices in thermal power plant operations and maintenance. 4. Power SIM: The Power SIM computer program, modeling the Pakistan power sector was developed for the Government. This analytical tool is being used for predicting financial impacts of policy reforms, fuel switching and efficiency enhancements in the power sector. 5. Internship Program: The internship program placed 130 interns in 17 public energy sector entities throughout all provinces of the country. Over 20 percent of these interns have been women. Thirty￾five percent of graduates from EPP’s Internship Program are employed full-time, 11 percent of those are female. 6. Monitoring and Implementation Support: Effective monitoring and implementation support to GENCOs and WAPDA for the $291.71 million obligated for power plant completion and rehabilitation projects (see Component I, above) has resulted in 933 MW of power restored or added. 7. Economic and Technical Studies: These studies have been conducted to facilitate GoP policy reforms in the energy sector. They have supported the newly elected PML-N government in formulating the new 2013 National Energy Policy; supported prioritization of gas allocations to the power sector; convinced the GoP to begin importing LNG in order to enhance the diversification of the country’s fuel mix for power generation; and, helped inform the government of existing domestic gas resources, including shale gas. EPP continues to provide consultants to the GoP to recommend ways for increasing domestic gas production. 8. Linemen Training: EPP provided training to 38 linemen from distribution companies in Peshawar, Hyderabad, Faisalabad and Gujranwala on live line maintenance of high and extra-high voltage transmission lines. 9. Shahibagh Grid Station: Forty-eight MW of power have been restored through the rehabilitation of this grid station’s power transformer and its three capacitor banks. 10. Transmission: To date, 321.1 MW of transmission throughput capacity have been added to the NTDC and PESCO networks as a result of EPP’s assistance. Improvements and upgrades to the transmission system by reducing transmission losses and improving reliability have resulted in this restored capacity. Areas identified as requiring enhancement of EPP services. They have been communicated to the implementer, which has agreed to their implementation and, in some areas, demonstrated that these improvements are already underway: 53 1. More frequent and comprehensive reports to be submitted to USAID on plans for meetings with representatives from GoP agencies and donors, in particular with the current main clients: MWP and MPNR. 2. More comprehensive and timely summaries on plans for travel by EPP staff to be submitted to USAID. 3. More frequent updates on progress and plans in various activities demonstrating that EPP delivers “diverse services to diverse GoP clients in the energy sector” is being properly planned and documented. 4. More of a global approach to capacity building of GoP energy sector staff is needed. This is to ensure the sustainability of the development resources invested in the sector, and it should go beyond the focused but dispersed initiatives in capacity building, which have been possible to date. 5. Implement more comprehensive capacity building in hydropower operations and maintenance, addressing both the power plant and dam structure, but still giving attention to the operation and maintenance of thermal plants, based on the fact that power generation and electricity supply to users is still primarily dependent on thermal units, with hydropower units closely behind. More comprehensive due diligence assessments of new generation and transmission project opportunities are needed to accelerate the implementation of capital projects in the sector and gas supply infrastructure in the nation. There are several challenges and constraints that EPP faces. Among these constraints are the lack of capacity and willingness of the counterparts to support real reform, and stakeholder engagement. 1. Stakeholder Management: EPP advises various ministries, authorities, commissions, centers and other GoP entities. Each of these audiences has competing visions for what the future state of Pakistan’s power sector should look like. 2. Political Will: There is a general lack of political will to implement structural reforms. Those officials capable and best placed to implement reforms generally benefit from the status quo. 3. Transparency: There is a general lack of transparency in GoP institutions. 4. Leadership: The GoP is perceived to lack the leadership and will to curb both energy theft and non￾payment of bills. Of particular concern are the geographic areas of the federal and provincial governments; Federally Administered Tribal Areas (FATA); and, Azad Jammu and Kashmir (AJK). Additionally, some of the security services, private consumers, and the politically well-connected are also perceived to benefit from this challenge. 5. Circular Debt: Even though “circular debt” was eased during the summer of 2013 (by using a tranche of IMF money plus a wave of rupees printing), it is building once again. There appears to be an absence of political will within the government to address the fundamental drivers of circular debt. 6. Decision Making: Seven ministries and over thirty organizations have to approve a power sector project before construction can begin. Lack of a centralized decision-making entity delays project approval and implementation. Concentrated decision-making has resulted in some key counterparts managing more than one entity. Individuals rather than institutions are responsible for making critical decisions. Those decisions are often not practical and lack institutional support, and are therefore, not implemented. 7. Counterpart Senior Management: Changes in counterpart senior management and decision-making staff in ministries and counterpart organizations take place too frequently to allow for the continuity 54 of the vision needed to implement any long-term improvement strategies in the public sector energy entities. 8. Security: Deteriorating security throughout the country hinders working with some provincial authorities and power sector companies based in Khyber Pakhtunkhwa and Balochistan. It is becoming increasingly difficult to obtain No Objection Certificates (NOCs) to visit project sites. 9. Visas for Expats: Obtaining Pakistani visas for expatriate project staff remains a challenge. 10. Lack of Understanding: The power sector is perceived to be beholden to a group of ministries that do not understand their role to develop and implement policy in their respective domains, and do not direct challenges to the proper authorities. 11. Change Management: Effecting change during a period when the current PML-N government is still settling in after the 2013 elections is difficult, as the new government pursues a series of short-term interventions to ensure popular credibility, as well as compliance with quarterly IMF targets and milestones, rather than pursuing a longer-term strategic vision. II. RATIONALE FOR EVALUATION This evaluation will be conducted to review the performance of the USAID-funded Energy Policy Program (EPP) implemented by AEAI. The evaluation will focus on assessing the program’s effectiveness in achieving its planned results, along with identifying the factors that have led to this success or lack thereof. The evaluation will inform USAID’s decisions regarding future programs. A. Purpose and Use of the Evaluation This external evaluation is taking place during the last year of the Energy Policy Program’s implementation. The evaluation will serve a dual purpose: (I) to learn to what extent the project’s planned results have been achieved and how sustainable they are; and (II) to inform the design of USAID’s future support of energy sector reforms in Pakistan. The information from this evaluation will assist the USAID/Pakistan Mission in decision making related to (a) the effectiveness of the EPP activities in supporting the GoP’s energy sector; (b) the type of approach the Mission should adopt in any future assistance to the GoP’s energy sector reforms; and (c) the type and scope of possible future interventions in the energy sector. B. Audience and Intended Use The primary audience for the evaluation includes: (i) the USAID/Pakistan Mission, particularly the Energy Office Team, (ii) U.S. Embassy Pakistan ASSIST Office (ASSIST), (iii) U.S. State Department Special Representative for Afghanistan and Pakistan (SRAP), (iv) the USAID Office of Afghanistan and Pakistan Affairs (OAPA), and (v) the implementing partner AEAI, Inc. An Executive Summary and recommendations will be provided to the GoP. USAID and other stakeholders will use the evaluation report to inform future policy and programs. 55 C. Evaluation Questions USAID has developed the following two questions for the evaluation: 1. How and to what extent has the project achieved planned results? Explanation: The question addresses the effectiveness of EPP, overall and specifically its various components. The answer to the question should explore what has worked and how, what has not worked as anticipated and why, and highlight any unintended outcomes. It should also examine instances where EPP has changed its approach to overcome obstacles and the results of these changes, and identify issues (political and operational) that have not been addressed and why. The question should include EPP’s effectiveness in improving the “governance” of public sector power generation and transmission companies. The answer to the question should also contribute to practical recommendations and lessons learned for making future programs more effective. This question should also provide recommendations about which project activities to continue, revise, or drop in future energy programming. 1.1. What is the perceived effect of the 1,300 MW added to the grid through USAID energy projects? Explanation: The sub-question will specifically focus on examining the EPP stakeholders’ perception of the effect—if any—of the 1,300 MWs added to the grid through EPP. The answer to this question will explore ways in which the increased energy supply, i.e., the addition/restoration of 1,300 MWs, has improved/changed/affected GoP’s perception of the utility of EPP viz-a-viz USAID’s support to energy sector in Pakistan. 2. What is the likelihood that the results EPP has achieved are sustainable beyond the life of the project? Explanation: This question should generate conclusions about which EPP results are likely to be sustainable beyond USAID support. It should also explore factors that contribute to sustainability or unsustainability. It should generate recommendations for how to enhance prospects for sustainable results and identify any additional actions EPP or USAID can take that would enhance prospects for sustainability in the future, e.g., continued advocacy after EPP ends or follow-on activities. III. EVALUATION DESIGN AND METHODOLOGY A. Data Collection Methods The study will employ qualitative methods design for data collection and analysis to produce clear qualitative evidence to answer the evaluation questions. The evaluation team will draw these data from both primary and secondary sources. The principle source of secondary data will be the available program documents (mentioned above) and the primary data will come from qualitative interviews. The study will use the following data collection methods. Document review – Document review will be an on-going process before and during the field work. The study team will collect and review all relevant documents from USAID, AEAI. Inc, and other sources identified during the study. The relevant documents may include, but are not limited to, original EPP contract, quarterly and annual reports, work plans, program modification documents, M&E reports, and technical assessments and studies. The study team will review technical assessments and studies, and program modification documents along with various program background and performance documents as required to 56 understand the project, the development needs it is designed to address, the basis for the evaluation questions above, and the evolutionary history of the program. Individual Interviews – The study team will gather information from key stakeholders supported directly by EPP. The individual interview protocols prepared by the study team will particularly focus on exploring how effective EPP has been and how sustainable the results it achieved will be. In addition, the interviews will include questions that will explore instances where EPP has changed its approach to overcome obstacles and the results of these changes, and identify issues (political and operational) that have not been addressed and why. Interview participants will include representatives from WAPDA, GENCOs, Ministry of Water and Power, Ministry of Petroleum and Natural Resources, Ministry of Finance, and Planning Commission. In addition, representatives from AEAI, Inc. and USAID including The Energy Office staff will be interviewed. Table 1 below provides an illustrative breakdown of individual interviews by organization, location and EPP component. TABLE 2: PROPOSED SAMPLE FOR INDIVIDUAL INTERVIEWS34 Relevant Stakeholders Number of Interviewees Project Component Project Location 1. USAID ENR, OPM, OAA and OFM Offices 4 Project Overview, Component 3 Islamabad 2. DCOP, AEAI 1 Project Overview Islamabad 3. Additional/Joint Secretary, Ministry of Water and Power 1 Project Overview, Components 2 and 4 Islamabad 4. Member Power, WAPDA 1 Project Overview Lahore 5. CEOs, Chief Engineer/Project Directors, and Finance/Procurement Managers of Jamshoro, Muzaffargarh and Guddu GENCOs 6 (select a sample of 2 GENCOs from the total of 3) Project Overview, Components 1 and 4 Jamshoro, Jacobabad, Muzaffargarh 6. 7. Chief Engineers/Project Directors for Tarbela, Gomal Zam, Satpara and Mangla Dams 4 Components 1 and 4 Haripur, South Waziristan, Skardu, Mirpur (AJK) 8. Joint Secretary, Ministry of Petroleum and Natural Resources (MPNR) 1 Components 2 and 4 Islamabad 9. Director Energy Wing, Planning Commission 1 Component 2 Islamabad 34 Annex 8 contains a more detailed list of potential interviewees by program activities, job titles, and point of contact. 57 Relevant Stakeholders Number of Interviewees Project Component Project Location 10. Managing Director, Project/Activity Manager, and Training Manager, National Transmission and Dispatch Company (NTDC) 3 Components 2 and 4 Lahore 11. CEO, Project/Activity Manager and Training Manager, National Power Control Center (NPCC) 3 Components 2 and 4 Islamabad 12.Joint Secretary, Ministry of Finance 1 Component 2 Islamabad 13.CEOs of DISCOs 5 (select a sample of 5 from a total DISCOs of 9) Component 4 Faisalabad, Lahore, Gujranwala, Multan, Hyderabad, Islamabad, Peshawar, Quetta and Sukkur 14.Director of Directorate General Petroleum Concessions (DGPC) 1 Component 4 Islamabad Total Number of Interviewees 32 B. Data Analysis Methods The evaluation team will use the document review and interview data to conduct the analysis in response to the questions identified above. The team will assemble in Islamabad to analyze the data and develop findings, conclusions and recommendations. The evaluation team will use a structured and systematic approach to analyzing the qualitative data. In addition, the data collection and analysis will employ several layers of triangulation (methods, sources, and investigator) to ensure the reliability and validity of results. The study team will analyze the qualitative data as follows:  Summarize notes from individual interviews;  Code individual interview notes/data according to themes relevant to the study questions;  Prepare tally sheets identifying the themes that emerge in the individual interviews to facilitate systematic and rigorous data analysis aimed at identifying key study findings; and  Prepare a detailed outline summarizing key findings, based on all the data analysis, and conclusions for each study question and overall recommendations. After completing the data analysis, the study team will debrief USAID/Pakistan on the findings, conclusions, and recommendations. After this, the expat team members will depart for their home stations where the Team Leader will draft the report. Other team members may provide assistance remotely. 58 C. Methodological Strengths and Limitations The evaluation methodology relies on triangulation of sources and methods to ensure the validity and reliability of results. The scope of this evaluation is limited to answering the evaluation questions to the extent that they can be operationally defined and data are available. The methods proposed for collecting and analyzing data are potentially subject to selection bias, which occurs when the subjects of interviews are not representative of the population of interest. In this case, selection bias is most likely to be limited as majority of the stakeholder the Energy Office identified have been selected for qualitative interviews. D. Existing Data and Information Sources A summary of the documents available to the evaluation team is listed below. A complete list of documents that will be reviewed by the evaluation team will be provided prior to the first evaluation meeting. Some baseline data is available for selected indicators. Program Documents:  EPP’s contract  EPP’s work plan  EPP M&E Plan including PIRS  Annual and Quarterly reports  Technical assessments, studies, and training reports GoP Documents:  Copies of key policies and regulations  G2G Agreements  Request letters from GoP for services EPP’s Work Plan is available both at the Mission and at the AEAI, Inc. Office in Islamabad (Sector G-6/3, Street 88, House #4). EPP’s performance indicators are described in EPP’s M&E plan (attached Annex 7). EPP’s M&E Plan includes key performance indicators for each key process area, along with indicator definitions, rationales, data sources, collection frequencies and targets. EPP has created Performance Indicator Reference Sheets (PIRS) for each indicator. The PIRS is a summary resource that describes each indicator in detail and includes information on indicator definitions, units of measurement, data sources, data collection methods, collection and reporting frequency, persons responsible for data collection, data analysis methods, data quality and data safety procedures, and performance targets. EPP has established a Monitoring and Evaluation (M&E) system that tracks and reports the results of selected performance indicators. The M&E unit has baseline data, quarterly, annual and special reports as well as audit reports. 59 The consultants will review the following documents: a) Work Plan b) Quarterly Reports c) Annual Reports d) MSF, M&E Plans and other related documents e) Project performance data f) Project-generated assessments and studies E. Evaluation Process The evaluation process consists of five main stages: Stage 1: Preparation and planning by the consultant’s staff. Stage 2: Initial review of priority documents by the evaluation team. Working from their home stations, the evaluation team members will review all project background documents. Stage 3: Team work in Islamabad. Upon deployment in Islamabad, the evaluation team will participate in a team planning meeting facilitated by the consultant and undertake specific preparatory tasks as a team. A five-day Team Planning Meeting (TPM) will be held in Islamabad, Pakistan before the evaluation begins In addition, the evaluation team will:  Clarify team members’ roles and responsibilities;  Establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion;  Review and finalize the assignment timeline and share with USAID;  Develop/finalize data collection methods, instruments, tools and guidelines;  Develop and present for USAID review and approval, a Data Analysis Plan that details how interviews will be conducted and analyzed; what procedures will be used to analyze the qualitative data from key stakeholder interviews; to reach conclusions about the effectiveness and sustainability of the interventions to date;  Review and clarify any logistical and administrative procedures for the assignment  Develop a preliminary draft outline of the team’s report; and  Assign drafting responsibilities for the final report. Stage 4: Fieldwork. The evaluation team will begin its fieldwork after it completes its team work in Islamabad. Team members will conduct key informant interviews. Stage 5: Data analysis and report writing. 60 IV. TEAM COMPOSITION A. Evaluation Team Positions and Skills The evaluation team will consist of:  Two international experts: One of the international experts will be the team lead and primary coordinator with USAID. S/he should be a senior-level evaluator with at least fifteen years of experience in areas such as international development, large-scale program management, evaluation of USAID projects and a working knowledge of power generation and transmission issues preferably in Pakistan’s energy sector. The second international consultant should be an expert in project management and policy-making. Both the international experts must be fluent in English and have strong writing skills.  Three Pakistani experts: The local independent experts should have experience in Pakistan’s power sector and/or large-scale USAID programs related to infrastructure and monitoring and evaluation. Their combined experience and expertise should focus on hydroelectric and thermal energy, environment, and finance and/or procurement. They should have extensive knowledge of the Pakistani energy sector. They should also be proficient in English as well as Urdu. A statement of potential bias or conflict of interest (or lack thereof) is required of each team member. B. Stakeholder Participation (USAID, Implementing Partner, beneficiaries, host government, other) From USAID, the Office of Energy will be the lead for the evaluation with PMU in a facilitation role. All other actors – EPP, MWP, MOF, MPNR, PC, NTDC, NPCC, GENCOs, WAPDA, PESCO, DISCOs – will be a focus of the evaluation. V. EVALUATION MANAGEMENT The evaluation team will officially report to MSI, who is responsible for all direct coordination with the USAID/Pakistan, through the Contract Officer’s Representative. From a technical management perspective, the evaluation team will work closely with following staff members of Energy Office: Michael Curtis, Energy Office Director Timothy Moore, Deputy Energy Director Saeed Anwar, COR for EPP The Energy Office will provide guidance for the participation of key partners. In order to maintain objectivity, key decisions about the evaluation will involve USAID/Pakistan’s Performance Management Unit. 61 A. Logistics USAID/Pakistan The Program Office’s Performance Management Unit (PMU) will facilitate the preparation of the evaluation SOW in accordance with USAID standards and good practices, review the instruments and the draft report, and provide technical inputs on the contractual matters. The energy office and EPP staff will extend support in conducting meetings with different GoP officials and will provide technical input to the evaluation. Energy Office will also provide different source of information as and when required. Energy Policy Program AEAI will provide all relevant information required for the evaluation and facilitate meetings and interviews which the evaluation team may require with their staff and beneficiary organizations. Beneficiary Organizations The staff from selected beneficiary organizations is expected to cooperate with the evaluation team by giving time for meetings and interviews and providing relevant documents. MSI MSI will provide support for travel, lodging and other arrangements related to evaluation team’s work. B. Scheduling The estimated time period for undertaking this evaluation is approximately 60 working days, of which about 40 days should be spent in Pakistan. The ideal arrival time for the international consultants is May 2015; however, the arrival date will be finalized between USAID and the organization conducting the evaluation. The evaluation team is required to work six days a week. The team is required to travel to selected provinces where program activities are being implemented. Approximately 40 percent of the consultants’ time will be spent outside Islamabad to conduct interviews with plant officials, project staff, government officials, and partner organization representatives. The evaluation team will prepare a debriefing and presentation of the findings, conclusions, and recommendations, which it will deliver to USAID staff before they depart from Pakistan. Comments from USAID will be incorporated before the submission of the final draft. The evaluation should finish by early August, 2015. The complete process, including finalization of the SOW, review of documents, developing interview tools, data collection and analysis, and report writing and finalization will require approximately 13 weeks. 62 TABLE 3: TENTATIVE EVALUATION SCHEDULE Activity W1 May W2 May W3 May W4 May W1 June W2 June W3 June W4 June W1 July W2 July W3 July W4 July W1 Aug Planning, document review Team Planning Workshop Data collection – Fieldwork Data analysis and initial debriefing Report writing Internal review by MSI and editing and branding USAID review and comments Home office review and final report submission to USAID 63 C. Budgeting TABLE 4: ANTICIPATED LEVEL OF EFFORT Tasks Team Leader (Expat￾STTA) Project Management/ Policy Expert (Expat￾STTA) Hydro￾electricity Expert (CCN￾STTA) Thermal Energy Expert (CCN￾STTA) Finance/ Procurement Expert (CCN￾STTA) Director Evaluation (LTTA) Evaluation Manager (LTTA) Technical Director (LTTA) Senior Advisor (LTTA) Review of Documents 3 3 3 3 3 2 2 1 Drafting of instruments 3 3 3 3 3 2 2 2 2 International Travel 3 3 - - - Team Planning Workshop 5 5 5 5 5 5 5 - 3 Field Work 12 12 12 12 12 3 12 - 1 Data analysis 17 17 17 17 17 8 17 - 5 Initial Findings Debriefing 1 1 1 1 1 1 1 - 1 Report writing 8 6 2 2 2 3 3 - 1 Internal report review and revisions 3 2 - - - 3 3 3 3 Total 55 52 44 44 44 27 45 5 17 64 VI. Evaluation Deliverables 1. Team Planning Meeting (TPM): A two-day Team Planning Meeting (TPM) will be held in Islamabad, Pakistan before the evaluation begins. This meeting will allow USAID to present the team with the purpose, expectations, and agenda of the assignment. In addition, the team will:  Clarify team members’ roles and responsibilities;  Establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion;  Review and develop final evaluation questions (work out realistic expectations of the team within each of the topic areas during meetings with AEAI, MWP, and USAID);  Review and finalize the assignment timeline and share with USAID;  Develop data collection methods, instruments, tools and guidelines;  Review and clarify any logistical and administrative procedures for the assignment;  Develop a preliminary draft outline of the team’s report; and  Assign drafting responsibilities for the final report. 2. Evaluation Work Plan: During the TPM, the evaluation team will prepare a detailed Work Plan which will include the methodologies to be used in the evaluation. The Work Plan will be submitted to the COR at USAID/Pakistan for approval no later than the sixth day of work. USAID will share the revised work plan with GoP for comments, as needed, and will revise accordingly. The initial work plan will include (a) the overall evaluation design, including the proposed methodology, data collection and analysis plan, and data collection instruments; (b) a list of the team members indicating their primary contact details while in-country, including the e-mail address and mobile phone number for the team leader; and, (c) the team’s proposed schedule for the evaluation. The revised work plan shall include the list of potential interviewees, sites to be visited, and evaluation tools. 3. Methodology Plan: A written Methodology Plan (evaluation design/operational work plan) will be prepared during the TPM and discussed with USAID prior to implementation. 4. Discussion of Preliminary Draft Evaluation Report: The Team will submit a rough draft of the report to the USAID COR and Energy Office team, who will provide preliminary comments prior to final Mission debriefing. This will facilitate preparation of a more final draft report that will be left with the Mission upon the evaluation team’s departure. 5. Debriefing with USAID: The team will present the major findings of the evaluation to USAID/Pakistan by means of a PowerPoint presentation after submission of the draft report and before the team’s departure from the country (if the evaluation team is based in Pakistan, this ‘departure’ obviously does not apply). The debriefing will include a discussion of achievements and activities only, with no recommendations for possible modifications to project approaches, results, or activities. The team will consider USAID comments and revise the draft accordingly, as appropriate. 6. Debriefing with AEAI, Inc.: The team will present the major findings of the evaluation to AEAI through a PowerPoint presentation. The debriefing will include a discussion of achievements and activities only, with no recommendations for possible modifications to project approaches, results, or activities. The team will consider AEAI’s comments and revise the draft report accordingly, as appropriate. 65 7. Draft Evaluation Report: The content of the draft evaluation report is outlined in Annex 1, below, and all formatting shall be consistent with the USAID branding guidelines. The focus of the report is to answer the evaluation questions and may include factors the team considers to have a bearing on the objectives of the evaluation. Any such factors can be included in the report only after consultation with USAID. The draft evaluation report will be submitted by the evaluation team leader to PMU 24 hours in advance of the exit briefing for review and comments by USAID. USAID’s PMU and Energy Office will have fourteen calendar days in which to review and comment and PMU shall submit all comments to the evaluation team leader. 8. Final Evaluation Report: The final report will incorporate final comments provided by the PMU. USAID comments are due within 10 days after the receipt of the initial draft. The final report should be submitted to the PMU within three days of receipt of comments by the evaluation team leader. All project data and records will be submitted in full and shall be in electronic form in easily readable format; organized and fully document for use by those not fully familiar with the project or evaluation; and owned by USAID and made available to the public barring rare exceptions. 9. Briefer – two page summary of evaluation in English and Urdu 10. Submission of data to USAID: Per ADS 579 - USAID Development Data –all data collected for this evaluation will be submitted to USAID in electronic format within 30 days of completion. 66 ANNEXES SOW Annex 1: Report Content (Adapted Agency Standard Language) The evaluation report will follow standard guidelines as laid out in Appendix 1 of USAID’S Evaluation Policy and operationalized in ADS 203.3.1.8 (Documenting Evaluations), reproduced in Annex 3. The evaluation report will follow the structure given below (the section titles and order are illustrative):  Title Page  Table of Contents  Table of Tables and Figures  List of Acronyms  Acknowledgements or preface (optional)  Program Summary  Map showing the location of program activities  Executive Summary: This section will be 3-5 pages in length and summarize key points (project purpose and background, key evaluation questions, methods, findings, etc.)  Introductory Chapter  The Development Problem and USAID’s Response (1-3 pages): This section will describe the development problem USAID wants to address. This will include USAID’s response to the problem, the development hypothesis and theory of change, results framework, and project implementation (including the current status of the project or activity)  Purpose of the Evaluation and Evaluation Questions (1-2 pages): This section will include the purpose of the Study and state all questions  Evaluation Design, Methodology, and Limitations (1-3 pages): A written design which includes key questions, methods, main features of data collection instruments; an explanation of why these methods were chosen, with additional information in the annex as necessary; limitations of the methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.), and how these have been accounted for; and data analysis plan with discussion relevant to the analysis  Findings and Conclusions: This section will include the findings and conclusions related to each evaluation question. If there are a large number of findings, there will be a synthesis or summary of findings for each question that establishes the connection with the conclusions that follow. The conclusion must answer each evaluation question based on the evidence provided in the findings.  Recommendations: Based on the conclusions, this section must include actionable statements that can be implemented into the existing program or included into future program design. Recommendations are only valid when they specify who does what, and relate to activities over which the USAID program has control.  References 67  Annexes:  Evaluation Statement of Work  Evaluation Methods and Limitations  Table of evaluation question by data sources, collection and analysis methodologies  Data Collection Instruments (all survey instruments, questionnaires, discussion guides, checklists, etc.)  Bibliography of Documents Reviewed  List of individuals and agencies contacted and places visited  Meeting notes of all key meetings with stakeholders.  Disclosure of Any Conflicts of Interest  Statement of Differences (only if applicable)  Evaluation Team Bios 68 SOW Annex 2: Table of Evaluation Questions by Data Sources, Collection, and Analysis Methodologies Evaluation Question Type of Answer/Evidence Data Collection Data Analysis Methods Method Sources Sampling 1. How and to what extent has the project achieved planned results? Explanation: The question addresses the effectiveness of EPP, overall and specifically its various components. The answer to the question should explore what has worked and how, what has not worked as anticipated and why, and highlight any unintended outcomes. It should also examine instances where EPP has changed its approach to overcome obstacles and the results of these changes, and identify issues (political and operational) that have not been addressed and why. The question should include EPP’s effectiveness in improving the “governance” of public sector power generation and transmission companies. The answer to the question should also contribute to practical recommendations and lessons learned for making future programs more effective. This question should also provide recommendations about which project activities to continue, revise, or drop in future energy programming. Descriptive  Description of project outcomes  Comparative analysis of project outcomes relative to project planned results Analytical Assessing how the projects achieved its outcomes and/or why did not.  Key stakeholder/expert interviews  Relevant project staff and USAID staff interviews Document Review  Project work plans, reports, training materials, EPP policies, agreements and due diligence studies  Review of GoP energy policy documents  Key stakeholder interviews with officials relevant entities such as, MWP WAPDA Member Power/Water, NTDC MD, PESCO CEO, GENCO CEO,MPNR, MOF Document Review  Results framework  Review project quarterly, annual reports, work Plan)  Capacity building and business plans developed by EPP for partner organizations  Various training materials/manuals developed by EPP for partner organizations  Training Reports (live line maintenance, PowerSIM, GENCO O&M, Internship program.)  Purposive sampling for key stakeholder interviews  USAID and project staff  Identification of trends and themes across data sources  Comparative analysis triangulating across data sources and between interview data and document review. 69 Evaluation Question Type of Answer/Evidence Data Collection Data Analysis Methods Method Sources Sampling 2. What is the likelihood that the results EPP has achieved are sustainable beyond the life of the project? Explanation: This question should generate conclusions about which EPP results are likely to be sustainable beyond USAID support. It should also explore factors that contribute to sustainability or unsustainability. It should generate recommendations for how to enhance prospects for sustainable results and identify any additional actions EPP or USAID can take that would enhance prospects for sustainability in the future, e.g., continued advocacy after EPP ends or follow-on activities. Analytical Assessing prospects of sustainability by examining: i. implementation and outcomes of new government policies; ii. Trainee utilization of new skills and knowledge iii. Utilization of the research  Key stakeholder/expert interviews Document Review  GoP policies and reports training materials, EPP policies, agreements.  Key informant interviews Key stakeholder interviews with officials relevant entities such as, MWP WAPDA Member Power/Water, NTDC MD, PESCO CEO, GENCO CEO,MPNR, MOF Document Review  Capacity building plans developed by EPP for partner organizations  Various training materials/manuals developed by EPP for partner organizations  Review project records(quarterly , annual reports, Work Plan)  Training Reports (live line maintenance, PowerSIM, GENCO O&M, Internship program, etc.)  G2G distribution  Purposive sampling for key stakeholder interviews including trainees  Identification of trends and themes across data sources  Comparative analysis triangulating across data sources and between interview data and document 70 SOW Annex 3: Reporting Guidelines (Agency Standard Language) According to ADS 203.3.1.8 (Documenting Evaluations), evaluation reports must meet the following criteria: 1. Evaluation reports must represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the project, what did not work, and why. 2. Evaluation reports must address all evaluation questions included in the scope of work. The evaluation report should include the evaluation statement of work as an annex. The technical officer (who is the COR when the evaluation is conducted by a contractor) must agree upon, in writing, all modifications to the statement of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline. 3. Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an annex in the final report. 4. When evaluation findings address outcomes and impact, they must be assessed on males and females. 5. Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). 6. Evaluation findings must be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. 7. Sources of information must be properly identified and listed in an annex. Recommendations must be supported by a specific set of findings and should be action-oriented, practical and specific, with defined responsibility for the action. 71 SOW Annex 4: List of Available Documents A summary of the documents available to the evaluation team is listed below. A complete list of documents that will be reviewed by the evaluation team will be provided prior to the first evaluation meeting. Some baseline data is available for selected indicators. Program Documents:  EPP contract  EPP work plan  EPP M&E Plan (PMP) including PIRS  Annual and quarterly reports  Technical assessments and studies GoP Documents:  Copies of key policies and regulations  G2G Agreements  Request letters from GoP for services The EPP Work Plan is available both at the Mission and at the AEAI, Inc. Office in Islamabad (Sector G￾6/3, Street 88, House #4). EPP’s performance indicators are described in the Project Management Plan (PMP) or draft M&E plan (attached Annex 7). The PMP table includes key performance indicators for each key process area, along with indicator definitions, rationales, data sources, collection frequencies and targets. EPP has created Performance Indicator Reference Sheets (PIRS) for each indicator in the PMP. The PIRS is a summary resource that describes each indicator in detail and includes information on indicator definitions, units of measurement, data sources, data collection methods, collection and reporting frequency, persons responsible for data coaction, data analysis methods, data quality and data safety procedures, and performance targets. EPP has established a Monitoring and Evaluation (M&E) system that tracks and reports the results of selected performance indicators. The M&E unit has baseline data, quarterly, annual and special reports as well as audit reports. 72 SOW Annex 5: Mission Strategic Framework Development Objective 1 Results Framework DO 1: Increased Sustainable Energy Supplied to the Economy a) gigawatt-hours (GW∙h) of energy sold b) hours of unplanned load shedding c) tariff differential subsidy IR 1.1.1: Increased Generation and Transmission Capacity a) megawatts (MW) of electrical power added or saved as a result of United States Government supported construction, rehabilitation, and other generation and transmission improvements b) efficiency of thermal power plants (British thermal units of input heat energy per kilowatt-hour of electrical output energy (Btu/kW∙h)) c) number of United States Government supported installations and operations and maintenance improvements of generation plants and transmission networks d) number of transmission bottlenecks resolved IR 1.2.1: Improved Policy Implementation a) number of key policies and regulations in development stages of analysis, drafting, stakeholder consultation, legislative review, approval, or implementation as a result of United States Government assistance b) TBD IR 1.1.2: Improved Efficiency of Consumption and the Distribution Systems a) megawatts (MW) of electrical power saved as a result of United States Government support to distribution companies b) number of installations and operations and maintenance improvements IR 1.1.3: Increased Financial Sustainability of Power Supply a) revenue at distribution companies b) number of days in the distribution company billing and collections cycle c) number of days for fuel adjustment process IR 1.2.2: More Autonomous Energy Sector Entities a) number of policies following international best practices developed and implemented b) number of board recommendations following international best practices implemented by public sector entities IR 1.2.3: Improved Capacity of USAID-Supported Energy Public-Sector Entities a) a) number of best practice-driven systems created, improved, and implemented b) b) percent change in the number of linemen injuries and deaths IR 1.2.4: Increased constructive civil society engagement in the energy sector a) number of civil society organizations receiving United States Government assistance engaged in advocacy interventions (energy disaggregate of the democracy and governance indicator) b) number of public forums resulting from United States Government assistance in which government officials and citizens interact IR 1.1: Increased Energy Supply a) number of beneficiaries with improved energy services due to United States Government assistance (4.4.1-31) b) gigawatt-hours (GW∙h) of energy availability c) power (megawatts) available to meet power sector demand as result of USG assistance IR 1.2: Improved Energy Sector Governance a) number of positive responses from citizens surveyed on transparency in the energy sector in Pakistan b) percent change in the gross annual accumulation of circular debt IR 1.1.4: Increased Non-USG Investment in the Energy Sector a) public and private funds leveraged by the United States Government for energy infrastructure projects (alternative F indicator 4.4.1-32) 73 SOW Annex 6: Energy Policy Program Map 74 SOW Annex 8: List of EPP Stakeholders Project Component Activity Relevant Institutions Stakeholder Groups within Institutions Number of People from Stakeholder Groups Point(s) of Contact Broad Overview of the Project  USAID  AEAI  GoP Key Ministries  ENR, OPM, OAA, OFM offices  DCOP  MWP, Joint Secretary  WAPDA Member Power  GENCOs CEO  Five  One  One  One  Three  Rabia Bukhari  Talha Javed  Zirgham Ishaq  Badar ul Munir  Mr. Iftikhar Aziz (GENCO-I)  Mr. Ghulam Rassol Akhund (GENCO-II)  Mr. Muhammad Shoaib Rasheed (GENCO-III) Broad Overview of Component-I  WAPDA  GENCOs (Jamshoro, Muzaffargarh & Guddu)  Chief Engineers or Project Director for Tarbela, Gomal Zam, Satpara, Mangla  Chief Executive Officers for each GENCO  Four  Three  Mr. Sher Afzal Khan (Chief Engineer Tarbela - Power)  Mr. Javed Akhtar (Project Manager Mangla )  Mr. Johar Ali (Project Director Satpara)  Mr. Sikander Ali Chandio (Project Director Gomal Zam)  Mr. Iftikhar Aziz (GENCO-I)  Mr. Ghulam Rassol Akhund (GENCO-II)  Mr. Muhammad Shoaib Rasheed (GENCO-III) 75 Project Component Activity Relevant Institutions Stakeholder Groups within Institutions Number of People from Stakeholder Groups Point(s) of Contact I - Monitoring and Support of Project Implementation  Supports USAID and counterparts in the monitoring, implementation, and compliance of Fixed Amount Reimbursement Agreements (FARA) with the GoP. Same as above  Monitors and supports timely counterpart compliance with all of the terms and conditions of FARAs, and inspection and certification of deliverables. Same as above  Provides monitoring and implementation support for four repair and maintenance projects (3 GENCOs and Tarbela); two multipurpose dam completion projects (Gomal Zam and Satpara); and, one dam rehabilitation project (Mangla).  WAPDA  GENCOs (Jamshoro, Muzaffargarh & Guddu)  Chief Engineers or Project Director for Tarbela, Gomal Zam, Satpara, Mangla  Chief Executive Officers for each GENCO  Four  Three  Mr. Sher Afzal Khan (Chief Engineer Tarbela - Power)  Mr. Javed Akhtar (Project Manager Mangla )  Mr. Johar Ali (Project Director Satpara) Mr. Sikander Ali Chandio (Project Director Gomal Zam)  Mr. Iftikhar Aziz (GENCO-I)  Mr. Ghulam Rassol Akhund (GENCO-II)  Mr. Muhammad Shoaib Rasheed (GENCO-III) 76 Project Component Activity Relevant Institutions Stakeholder Groups within Institutions Number of People from Stakeholder Groups Point(s) of Contact Broad Overview of Component-II  Ministry of Water and Power  Ministry of Petroleum and Natural Resources  Joint Secretary MWP  Advisor, MPNR  One or Two  One  Zirgham Ishaq  Zahid Muzaffar II - Advice and Support of Energy Sector Policy Reform  Supports energy sector reform by undertaking activities that are requested by one or more GoP entities and by the direct secondment of staff to work in the offices of the requesting entity as advisors and specialized support staff. Same as above Same as above Same as above  EPP advisors support the ministries of Water and Power, Finance, Petroleum and Natural Resources as well as the Planning Commission’s Energy Wing (PC), National Transmission and Dispatch Company (NTDC), and National Power Control Center (NPCC).  Planning Commission (PC)  MWP  MPNR  NTDC  NPCC  Ministry of Finance  Member (Energy)  Joint Secretary MWP  Advisor, MPNR MD NTDC  GM NPCC  Additional Secretary  One  One or Two  One  One  One  One  Syed Akhtar Ali  Zirgham Ishaq  Zahid Muzaffar  Arshad Choudhary  Aziz ur Rehmaan  Azra Mujtaba  In achieving the objectives of this component, EPP supports generation and transmission￾related studies and policy reform activities assigned by USAID with the support of the GoP.  Finally, this component supports generation, transmission, and oil and gas-related studies and policy reform activities assigned by USAID. 77 Project Component Activity Relevant Institutions Stakeholder Groups within Institutions Number of People from Stakeholder Groups Point(s) of Contact Broad Overview of Component-III  USAID  AEAI  Specifically, EPP conducts and prepares due diligence reports on new projects under consideration for USG support in order to provide detailed information, reduce USG risk, and set a basis for creating Project Implementation Agreements for the selected projects.  USAID  USAID Energy office Broad Overview of Component-IV  PESCO  NTDC & MPNR  PESCO CEO  Advisor, MPNR  NTDC MD  One  One  One  PESCO – Hasan Fazil  Zahid Muzaffar (Advisor, MPNR)  Arshad Choudhary (MD NTDC)  ?? IV - New Activities  Supports the poorly-performing “Turn￾around” DISCO (Peshawar Electric Supply Company (PESCO);  PESCO  CEO  One  PESCO – Hasan Fazil  Provides expanded support for governance, that focuses on implementation of high-impact, improved governance activities that will benefit the electricity sector and end-users, including fuel mix diversification from imported power generation resources;  MWP  Joint Secretary MWP  One or Two  Zirgham Ishaq 78 Project Component Activity Relevant Institutions Stakeholder Groups within Institutions Number of People from Stakeholder Groups Point(s) of Contact  Live Line Training: a sustainable, targeted training program for selected crews of linemen on best practices and techniques for repairing live power lines;  9 DISCOs  SE GSO  Nine  IESCO – Ayaz Hussain  LESCO – Mehmood Alam  GEPCO – Saleem  FESCO – Basharat  SEPCO – Nazeer Soomro  HESCO – Abdul Haq Memon  QESCO – Syed Salahuddin  PESCO – Abdul Waheed  MEPCO – Khalid Javed  Natural Gas: Targeted technical support to Ministry of Petroleum and Natural Resources in upstream oil and gas activities, including policy and regulatory oversight, resulting in increased supplies of domestic and imported gas (including Liquid Natural Gas and shale gas) for power generation;  Ministry of Petroleum  Directorate General Petroleum Concessions (DGPC) Director  Advisor, MPNR  One  One  Saeed Ullah Shah (DGPC)  Zahid Muzaffar (Advisor, MPNR)  Supports NTDC and NPCC to mitigate unscheduled power outages by providing technical assistance, including power imports support, transmission system reviews, assets, products and services enhancements, and capacity-building services;  NTDC  NPCC  MD NTDC and GM NPCC  Two  Two  Arshad Choudhary (MD NTDC)  Aziz Ur Rehmaan (GM NPCC) 79 Project Component Activity Relevant Institutions Stakeholder Groups within Institutions Number of People from Stakeholder Groups Point(s) of Contact  Supports GENCOs and WAPDA, consisting of technical assistance to improve finances, add MWs to the grid, and improve fuel efficiency and management operations.  Jamshoro, Muzaffargarh and Guddu Thermal Station Tarbela, Mangla, Gomal Zam, Satpara, Mangla  CEO GENCOs  Chief Engineer or Project Directors each WAPDA project  Three- Four  Three  Mr. Iftikhar Aziz (GENCO-I)  Mr. Ghulam Rassol Akhund (GENCO-II)  Mr. Muhammad Shoaib Rasheed (GENCO-III)  Mr. Sher Afzal Khan (Chief Engineer Tarbela- Power)  Mr. Javed Akhtar (Project Manager Mangla )  Mr. Johar Ali (Project Director Satpara)  Mr. Sikander Ali Chandio (Project Director Gomal Zam) 80 ANNEX 2: TABLE OF EVALUATION QUESTION BY DATA SOURCES, COLLECTION AND ANALYSIS METHODOLOGIES Evaluation Question Type of Answer/Evidence Data Collection Data Analysis Methods Method Sources Sampling 1. How and to what extent has the project achieved planned results? Explanation: The question addresses the effectiveness of EPP, overall and specifically its various components. The answer to the question should explore what has worked and how, what has not worked as anticipated and why, and highlight any unintended outcomes. It should also examine instances where EPP has changed its approach to overcome obstacles and the results of these changes, and identify issues (political and operational) that have not been addressed and why. The question should include EPP’s effectiveness in improving the “governance” of public sector power generation and transmission companies. The answer to the question should also contribute to practical recommendations and lessons learned for making future programs more effective. This question should also provide recommendations about which project activities to continue, revise, or drop in future energy programming. Descriptive  Description of project outcomes  Comparative analysis of project outcomes relative to project planned results Analytical  Assessing how the projects achieved its outcomes and/or why did not.  Key stakeholder/expert interviews  Relevant project staff and USAID staff interviews Document Review  Project work plans, reports, training materials, EPP policies, agreements and due diligence studies  Review of GoP energy policy documents  Key stakeholder interviews with officials relevant entities such as, MWP WAPDA Member Power/Water, NTDC MD, PESCO CEO, GENCO CEO,MPNR, MoF Document Review  Results framework  Review project quarterly, annual reports, work Plan)  Capacity building and business plans developed by EPP for partner organizations  Various training materials/manuals developed by EPP for partner organizations  Training Reports (live line maintenance, PowerSIM, GENCO O&M, Internship program.)  Purposive sampling for key stakeholder interviews  USAID and project staff  Identification of trends and themes across data sources  Comparative analysis triangulating across data sources and between interview data and document review. 81 Evaluation Question Type of Answer/Evidence Data Collection Data Analysis Methods Method Sources Sampling 2. What is the likelihood that the results EPP has achieved are sustainable beyond the life of the project? Explanation: This question should generate conclusions about which EPP results are likely to be sustainable beyond USAID support. It should also explore factors that contribute to sustainability or unsustainability. It should generate recommendations for how to enhance prospects for sustainable results and identify any additional actions EPP or USAID can take that would enhance prospects for sustainability in the future, e.g., continued advocacy after EPP ends or follow-on activities. Analytical Assessing prospects of sustainability by examining: iv. implementation and outcomes of new government policies; v. Trainee utilization of new skills and knowledge vi. Utilization of the research  Key stakeholder /expert interviews Document Review  GoP policies and reports training materials, EPP policies, agreements.  Key informant interviews Key stakeholder interviews with officials relevant entities such as, MWP WAPDA Member Power/Water, NTDC MD, PESCO CEO, GENCO CEO,MPNR, MoF Document Review  Capacity building plans developed by EPP for partner organizations  Various training materials/manuals developed by EPP for partner organizations  Review project records(quarterly , annual reports, Work Plan)  Training Reports (live line maintenance, Power SIM, GENCO O&M, Internship program, etc.)  G2G distribution  Purposive sampling for key stakeholder interviews including trainees  Identification of trends and themes across data sources  Comparative analysis triangulating across data sources and between interview data and document 82 ANNEX 3: DATA COLLECTION INSTRUMENTS USAID Energy Policy Program (EPP) Evaluation Interview Protocol for AEAI Staff Involved with Generation Introduction I am a consultant with Management Systems International (MSI), a U.S. based company that has been contracted to evaluate USAID funded development projects in Pakistan. MSI is carrying out an evaluation of USAID’s Energy Policy Program (EPP). This interview is being conducted to gain further knowledge and insight into the various activities implemented by the EPP contractor, Advanced Engineering Associates International (AEAI). As a key stakeholder, your input to our assessment is extremely valuable and will enhance our understanding and analysis of the program. Your answers will be used by MSI for data analysis purposes only and will be kept strictly confidential. No informants will be identified by name in the report. Would it be possible for us to record our conversation to support our notes? CHECK Yes- O No - O The recordings will be deleted at the end of the evaluation and will not be shared outside of MSI under any conditions. Can I confirm how much time you have available? Given the short time available and the importance of getting your response to the information we need across a number of areas, in case my question was unclear and leads the discussion in the wrong direction, I apologize if I might interrupt to clarify my question. ___________________________________ Organization Respondent Name(s) Title(s) 83 # and (Sub)Question Prompts / Probes Identification and Design 1. Could you describe your role within the EPP project? a. How long have you been involved with EPP, and has your role changed over time? b. Please describe your involvement in the design of EPP activities for your component. 2. What were the key considerations that were the basis of the design of EPP? a. How were decisions made regarding the type of assistance EPP would provide? 3. How could the program(s) have been otherwise designed? 4. What issues or challenges that resulted from USAID’s decision making processes were there in EPP project design? a. In implementation? Preparation and Approval 5. In which cases were feasibility studies, or technical or environmental studies NOT undertaken prior to the start of EPP assistance in generation? Looking for validation of the technical aspects of the projects, the financial costs and expected results Implementation 6. What were the major risk factors? a. How they were dealt with? Alternatively: What risk assessment was done at the start of the project? 7. Which components and sub-activities were the most difficult to implement? 8. Describe the coordination between EPP and different organizations involved in the program like WAPDA, GENCOs, HYDELs, NTDC 9. What external bottlenecks were faced during the implementation phase? Internal? 10. What other development finance institutions besides USAID is assisting in this work program? 11. Describe the quality control plans concerning procurement, construction and final product Were timely progress reports produced outlining progress to date, delays, issues and other factors effecting the project? 12. How did the approach to the program change over the course of the work? a. What benefits / problems resulted? Looking for underlying issues out of project control that needed to be overcome. Effectiveness 13. Data Requests by plant a. nameplate capacities b. Monthly production capacity, heat rate (thermal), available MW at the unit level over last five yrs? c. Monthly unit-level and plant-level peak load profiles over last 5 yrs d. Annual outages over the last five years. average duration. Reasons Measuring reliability of the investment and capacity building. What types of outages were most affected? 14. Are GENCOS carrying out Annual Heat Rate tests? 15. Regarding the extent to which the work program with EPP affected output a. To what extent did MWh (production) and Peak Load (MW) improve? 16. [Transmission] What were the transmission bottlenecks resolved through EPP? 84 # and (Sub)Question Prompts / Probes 17. Did the work program with EPP reduce or shorten outages? If so, by how much? Number and duration before and after. 18. How and to what extent has EPP improved the Accounts Payable position of GENCOs i.e payments to oil/ gas companies? 19. Regarding training by the Program a. How were candidates chosen? b. What do you hear about plans to continue training? 20. Have other elements of the companies or ministries not directly related to the EPP activities benefitted from the training and capacity built during the EPP program? a. Have other people who have not been directly involved with EPP training and capacity building benefitted from these efforts? 21. Was the overall EPP work program successful in achieving its stated objectives? a. Could you provide examples of outcomes that were unanticipated, both positive and negative? Were there any unaccounted for benefits or shortcomings not accounted from the original feasibility study. 22. Speaking of the EPP project broadly, what do you think is the broader effect of the additional MW added due to EPP? a. Do you think the EPP’s contributions to the energy supply will be sustained? Why or why not? What do you think is the additional amount of MW added to the system nationally? 23. Fuel availability will affect GENCO capacity to reach maximum MWs. Import of LNG by 2015 will decrease costs and increase energy supply. Will that make any significant impact on the availability of the plants? 24. Hydrology affects seasonal water flows for hydels which can affect maximum MWs. Has the EPP’s work helped in dealing with seasonality? 25. What is the status of Business plans for NTDC/GENCOs? 26. What is the status of PPAs for GENCOS? 27. How was the involvement of plan personnel in the maintenance or rehabilitation activities of contractors at Hydros/Gencos carried out? a. What was the involvement of plant personnel in the program? b. Were any skills transferred to the plants engineers and technical staff on the best international maintenance practices this way? 28. To what extent has the staff and management taken ownership of the rehabilitation activities at plants/grid stations/discos? 29. Speaking of system capacity, rather than human capacity, what should be done to ensure capacity is maintained or increased? 30. What efforts are being made by EPP to help ensure that companies are able to maintain or expand EPP’s benefits without further donor financial support? a. By GoP? b. What are the constraints, internal and external, that would require them to continue to rely on donor assistance? 31. Based on reflection, can you think of anything that you wish you had done during EPP implementation to improve and sustain the program’s outcomes? a. In your view, would you have changed emphasis between reliability and enhancing transformer capacity? 32. What advice would you give USAID if it were to invest in a future program to help advance energy policy? 85 USAID Energy Policy Program (EPP) Evaluation Interview Protocol for AEAI Staff Involved with Generation Introduction I am a consultant with Management Systems International (MSI), a U.S. based company that has been contracted to evaluate USAID funded development projects in Pakistan. MSI is carrying out an evaluation of USAID’s Energy Policy Program (EPP). This interview is being conducted to gain further knowledge and insight into the various activities implemented by the EPP contractor, Advanced Engineering Associates International (AEAI). As a key stakeholder, your input to our assessment is extremely valuable and will enhance our understanding and analysis of the program. Your answers will be used by MSI for data analysis purposes only and will be kept strictly confidential. No informants will be identified by name in the report. Would it be possible for us to record our conversation to support our notes? CHECK Yes- O No - O The recordings will be deleted at the end of the evaluation and will not be shared outside of MSI under any conditions. Can I confirm how much time you have available? Given the short time available and the importance of getting your response to the information we need across a number of areas, in case my question was unclear and leads the discussion in the wrong direction, I can clarify my question. ___________________________________ Organization Respondent Name(s) Title(s) 86 # and (Sub)Question Prompts / Probes Identification and Design 1. Could you describe your role within the EPP project? a. How long have you been involved with EPP, and has your role changed over time? b. Please describe your involvement in the design of EPP activities for your component. 2. What were the key considerations that were the basis of the design of EPP? a. How were decisions made regarding the type of assistance EPP would provide? 3. How could the program(s) have been otherwise designed? 4. What issues or challenges that resulted from USAID’s decision making processes were there in EPP project design? a. In implementation? Preparation and Approval 5. In which cases were feasibility studies, or technical or environmental studies NOT undertaken prior to the start of EPP assistance in generation? Looking for validation of the technical aspects of the projects, the financial costs and expected results Implementation 6. What were the major risk factors? a. How they were dealt with? Alternatively: What risk assessment was done at the start of the project? 7. Which components and sub-activities were the most difficult to implement? 8. Describe the coordination between EPP and different organizations involved in the program like WAPDA, GENCOs, HYDELs, NTDC 9. What external bottlenecks were faced during the implementation phase? Internal? 10. What other development finance institutions besides USAID are assisting in this work program? 11. Describe the quality control plans concerning procurement, construction and final product Were timely progress reports produced outlining progress to date, delays, issues and other factors effecting the project? 12. How did the approach to the program change over the course of the work? a. What benefits / problems resulted? Looking for underlying issues out of project control that needed to be overcome. Effectiveness 13. Data Requests by plant a. nameplate capacities b. Monthly production capacity, heat rate (thermal), available MW at the unit level over last five yrs? c. Monthly unit-level and plant-level peak load profiles over last 5 yrs d. Annual outages over the last five years. Average duration. Reasons Measuring reliability of the investment and capacity building. What types of outages were most affected? 14. Are GENCOS carrying out Annual Heat Rate tests? 15. Regarding the extent to which the work program with EPP affected output a. To what extent did MWh (production) and Peak Load (MW) improve? 16. [Trans] What were the transmission bottlenecks resolved through EPP? 17. Did the work program with EPP reduce or shorten outages? If so, by how much? Number and duration before and after. \ 18. How and to what extent has EPP improved the Accounts Payable position of GENCOs i.e payments to oil/ gas companies? 87 # and (Sub)Question Prompts / Probes 19. Regarding training by the Program c. How were candidates chosen? d. What do you hear about plans to continue training? 20. Have other elements of the companies or ministries not directly related to the EPP activities benefitted from the training and capacity built during the EPP program? a. Have other people who have not been directly involved with EPP training and capacity building benefitted from these efforts? 21. Was the overall EPP work program successful in achieving its stated objectives? a. Could you provide examples of outcomes that were unanticipated, both positive and negative? Were there any unaccounted for benefits or shortcomings not accounted from the original feasibility study. 22. Speaking of the EPP project broadly, what do you think is the broader effect of the additional MW added due to EPP? a. Do you think the EPP’s contributions to the energy supply will be sustained? Why or why not? What do you think is the additional amount of MW added to the system nationally? 23. Fuel availability will affect GENCO capacity to reach maximum MWs. Import of LNG by 2015 will decrease costs and increase energy supply. Will that make any significant impact on the availability of the plants? 24. Hydrology affects seasonal water flows for hydels which can affect maximum MWs. Has the EPP’s work helped in dealing with seasonality? 25. What is the status of Business plans for NTDC/GENCOs? 26. What is the status of PPAs for GENCOS? 27. How was the involvement of plan personnel in the maintenance or rehabilitation activities of contractors at Hydros/Gencos carried out? a. What was the involvement of plant personnel in the program? b. Were any skills transferred to the plants engineers and technical staff on the best international maintenance practices this way? 28. To what extent has the staff and management taken ownership of the rehabilItation activities at plants/grid stations/discos? 29. Speaking of system capacity, rather than human capacity, what should be done to ensure capacity is maintained or increased? 30. What efforts are being made by EPP to help ensure that companies are able to maintain or expand EPP’s benefits without further donor financial support? a. By GoP? b. What are the constraints, internal and external, that would require them to continue to rely on donor assistance? 31. Based on reflection, can you think of anything that you wish you had done during EPP implementation to improve and sustain the program’s outcomes? b. In your view, would you have changed emphasis between reliability and enhancing transformer capacity? 32. What advice would you give USAID if it were to invest in a future program to help advance energy policy? 88 USAID Energy Policy Program (EPP) Evaluation Protocol for Interview with EPP (July 7, 2015) Introduction: I am a consultant with Management Systems International (MSI), a U.S. based company that has been contracted to evaluate USAID funded development projects in Pakistan. MSI is carrying out an evaluation of USAID’s Energy Policy Program (EPP). This interview is being conducted to gain further knowledge and insight into the various activities implemented by the EPP contractor, Advanced Engineering Associates International (AEAI). As a key stakeholder, your input to our assessment is extremely valuable and will enhance our understanding and analysis of the program. Your answers will be used by MSI for data analysis purposes only and will be kept strictly confidential. No informants will be identified by name in the report. Would it be possible for us to record our conversation to support our notes? CHECK Yes- O No - O The recordings will be deleted at the end of the evaluation and will not be shared outside of MSI under any conditions. Can I confirm how much time you have available? Given the short time available and the importance of getting your response to the information we need across a number of areas, in case my question was unclear and leads the discussion in the wrong direction, I can clarify my question. ___________________________________ Organization Respondent Name(s) Title(s) 89 Questions Prompts/Probes Identification 1. Could you describe your role within the EPP project? a. How long have you been involved with EPP, and has your role changed over time? b. Please describe your involvement in the design of EPP activities for your component. 2. Do you feel you had a strategic role in Pakistan energy reform? What do you think was the most important intervention and why? - Activities/responsibilities on a day-to-day basis - As the COR what is your level of input in - Identifying projects - Initiating projects - Expert input into program design Preparation/ Implementation 3. Is your (EPP) a tactical intervention in a large strategic plan of the USAID? a. Is this strategic plan a coordinated one with other international development agencies? -Is the development of policy related to a technical study, input from a think-tank, stakeholder recommendation? 4. Is the AEAI work-plan for the EPP program coordinated with PDP? To what extent are the synergies captured? 5. Does AEAI plan & submit projects to USAID for funding? What is the mechanism for receiving feedback from USAID? The role of EPP experts in shaping the program. 6. Please list the advisory positions and the output that these advisors made or contributed to GoP policy making? What was the timeline of these interventions? How far the advisers stayed the course to their original SoW? 7. What role do EPP-USAID advisors play in developing EPP programs significantly in developing the plans with the GoP? a. Do you receive direct feedback from the ministries on EPP advisors? - Making decisions about which EPP activities will address the ministry’s needs Describe your role in providing support to the ministry Effectiveness and Sustainability 8. To what extent do you believe EPP has met USAID’s broader goals from EPP in the policy realm? 9. Could you describe what have been the priorities for policy reform? Does AEAI receive any feedback on effectiveness of policy reform and prioritize the agenda in any manner? Collaboration with stakeholders. Seminars, meetings, workshops prior to launching an initiative. Needs assessment 10. In your opinion, are the implemented sets of policy reforms in the energy sector adequate? a. What are the remaining key areas for policy reform within the country? Specific areas requiring attention e.g. transmission & distribution infrastructure investment, fast￾tracked privatization, tariff deregulation etc. Have you been asked by USAID to prepare a “lessons learnt” summary. 11. Did the approach of EPP support change over the last three years? a. If yes, why and how? Please explain Has the level of support increased or decreased over this period? 12. As a result of EPP support, what changes, if any, was the Pakistan Power Sector able to initiate or enforce. Please provide details. a. Have there been any administrative orders issued as a result of the EEP support on policy development? Examples of reforms Examples of policy changes What else? 13. Could you describe some of the key policy level inputs from the EPP program? Please give examples. State major examples that led to policy input from the EPP provided support 90 Questions Prompts/Probes 14. Have other elements of the ministries not directly related to the EPP activities benefitted from the training and capacity built during the EPP program? a. Have other people within the ministries? 15. Speaking of the EPP project broadly, what do you think is the broader effect of the additional MW added due to EPP? What do you think is the additional amount of MW added to the system nationally? 16. Are the measures taken to implement and institutionalize policies developed with the support of EPP by the Pakistani government been adequate? What steps can be taken to improve the co-operation? Has legislative action been sufficient to institutionalize the set of EPP policy interventions? Need to establish a USAID working group that directly liaises between the implementing partner and GoP? 17. Do you feel the USAID steering of your project was mutually supportive? Alternate: Do you feel the guidance received from USAID aided project effectiveness? 18. Can you provide examples of major policy changes within the GoP that can be attributed to EPP’s Intervention? 19. Were any policies put in place by the GoP that may be negatively impacting EPP interventions? Any policy that undermine USAID interventions. 20. Do you try to measure the effect of policy differently on women, children, men? By poverty level? 21. What is the EPP legacy? What reforms are you leaving as EPP legacy? Kindly provide a few examples. Recommendations 22. What changes would you have made to the EPP work program? 23. What recommendations would you have for similar future policy improvement programs? Using USAID resources 91 USAID Energy Policy Program (EPP) Evaluation Interview Protocol for Stakeholders Involved with Generation Introduction I am a consultant with Management Systems International (MSI), a U.S. based company that has been contracted to evaluate USAID funded development projects in Pakistan. MSI is carrying out an evaluation of USAID’s Energy Policy Program (EPP). This interview is being conducted to gain further knowledge and insight into the various activities implemented by the EPP contractor, Advanced Engineering Associates International (AEAI). As a key stakeholder, your input to our assessment is extremely valuable and will enhance our understanding and analysis of the program. Your answers will be used by MSI for data analysis purposes only and will be kept strictly confidential. No informants will be identified by name in the report. Would it be possible for us to record our conversation to support our notes? CHECK Yes- O No - O The recordings will be deleted at the end of the evaluation and will not be shared outside of MSI under any conditions. Can I confirm how much time you have available? Given the short time available and the importance of getting your response to the information we need across a number of areas, in case my question was unclear and leads the discussion in the wrong direction, I can clarify my question. ___________________________________ Organization Respondent Name(s) Title(s) 92 # and (Sub)Question Prompts / Probes Identification 1. How long have you been in your organization and how long involved with the EPP project? a. What is your role with the EPP project and has your role changed over time? - Activities/responsibilities on a day-to￾day basis - How long have you served at this position? Level within the overall policy making hierarchy in the organization 2. What were the broad issues identified that your utility / institution wanted to address with EPP's assistance? a. Who decided the type of assistance EPP would provide How was this issue identified and agreed upon for further study? Preparation and Approval 3. Was there a feasibility study or technical or environmental study undertaken prior to the start of EPP assistance? a. If yes, please document source b. If no, please explain justification Looking for validation of the technical aspects of the Project, the financial costs and expected results 4. What were the major findings from the studies? a. Who approved moving ahead with project assistance after the studies were completed? Looking for indicators of good governance. Implementation and Effectiveness 5. Could you summarize the assistance provided to your organization by EPP? a. What were the objectives of EPP’s assistance? b. What were the investments? c. Describe the Project Management Structure (including who supervises the overall project in your facility as well as the individual components) - What was EPP's proposed role in solving the issue addressed earlier? - Just a broad understanding of how the Project will work and the benefits of it. 6c. Do you have a PMU? 6. TRAINING / CAP BUILD What activities did EPP provide to strengthen the technical capacity of employees? a. Besides training, what other technical support did EPP provide? - Including seminar, trainings & knowledge transfer EPP provided Includes types of hardware invested and indirect support through capacity building 7. TRAINING / CAP BUILD How many men and women were trained by the Program? e. How were they chosen? f. How many remain in their positions after receiving training? g. How many were men/women? 8. What did staff from your plant learn from EPP? 9. TRAINING / CAP BUILD Have other elements of the utility/organization/company not directly related to the EPP activities benefitted from the training and capacity built during the EPP program? a. Have other people who have not been directly involved with EPP training and capacity building benefitted from these efforts? 93 # and (Sub)Question Prompts / Probes 10. OVERSIGHT & CAP BUILD Were plant staff involved in overseeing the maintenance or rehabilitation activities of contractors? a. What was the involvement of plant personnel in the program? b. Were any skills transferred to the plants engineers and technical staff on the best international maintenance practices this way? c. To what extent will you be able to follow international best practice for operation and maintenance of the plant? d. for rehabilitation? e. How prepared is your utility to carry out annual heat rate tests? 11. [GENCOs] Have you developed PPAs? a. How will that be useful? b. Where are you in the process of developing business plans? c. How useful has been development of business plans? Please provide examples 12. Describe the quality control plan concerning procurement, construction and final product Were timely progress reports produced outlining progress to date, delays, issues and other factors effecting the project? 13. Did you encounter any significant delays in procurement processes? 14. [GENCOs] To what extent do you expect the import of LNG to make affect the availability of the plants? Data, Effectiveness Outcomes, Sustainability 15. Data Requests (2010-2015): e. What is your nameplate capacity? f. What was the monthly production capacity, heat rate (thermal), and available MW at the unit level for the units on which EPP worked? g. What was your monthly unit-level and plant-level peak load profile? h. How many annual outages? average duration? Reasons? Measuring reliability of the investment and capacity building. What types of outages were most affected? 16. SUCCESS OVERALL Regarding the extent to which the work program with EPP affected output a. To what extent did MWh (production) and Peak Load (MW) improve? [#] b. Did the EPP work program reduce or shorten outages? Looking for awareness of results. c. Was the overall EPP work program successful in achieving its stated objectives? d. Could you provide examples of outcomes that were unanticipated, either positive or negative? Were there any unaccounted for benefits or shortcomings not accounted from the original feasibility study. 94 # and (Sub)Question Prompts / Probes Improvements, Changes in approach, Recommendations 17. What should be done to ensure (physical) capacity is maintained or increased? a. What efforts are being made to help ensure that you are able to maintain or expand EPP’s benefits without further donor financial support? b. What are the constraints, internal and external, that would require you to continue to rely on donor assistance? 18. Based on reflection, can you think of anything that you wish you had done during EPP implementation to improve and sustain the program’s outcomes? c. In your view, would you have changed emphasis between reliability and enhancing transformer capacity? 19. Did the approach to the program change over the course of the work? Looking for underlying issues out of project control that needed to be overcome. 20. What other development finance institutions besides USAID is assisting you in the work program in which EPP works with you? 21. Speaking of the EPP project broadly, what do you think is the additional amount of MW added to the system nationally by EPP? a. What do you think is the broader effect of the additional MW added due to EPP? b. Do you think the EPP’s contributions to the energy supply will be sustained? Why or why not? 22. What advice would you give USAID if it were to invest in a smaller future program to help advance energy policy? 95 USAID Energy Policy Program (EPP) Evaluation Interview Protocol for Stakeholders Involved with Policy Development I am a consultant with Management Systems International (MSI), a U.S. based company that has been contracted to evaluate USAID funded development projects in Pakistan. MSI is carrying out an evaluation of USAID’s Energy Policy Program (EPP). This interview is being conducted to gain further knowledge and insight into the various activities implemented by the EPP contractor, Advanced Engineering Associates International (AEAI). As a key stakeholder, your input to our assessment is extremely valuable and will enhance our understanding and analysis of the program. Your answers will be used by MSI for data analysis purposes only and will be kept strictly confidential. No informants will be identified by name in the report. Would it be possible for us to record our conversation to support our notes? CHECK Yes- O No - O The recordings will be deleted at the end of the evaluation and will not be shared outside of MSI under any conditions. Can I confirm how much time you have available? Given the short time available and the importance of getting your response to the information we need across a number of areas, in case my question was unclear and leads the discussion in the wrong direction, I can clarify my question. ___________________________________ Organization Respondent Name(s) Title(s) 96 Question or Sub-question Prompts or Probes Identification 1. Could you describe your role within your organization and your involvement in EPP? a. How long have you been involved with EPP? - Activities/responsibilities on a day-to-day basis - How long have you served at this position? - Level within the overall policy making hierarchy in the organization Preparation / Implementing EPP-supported Program 2. What are the energy related policy objectives of your organization? a. How did you develop those? - Was the development of policy related to a technical study, input from a think-tank, stakeholder recommendation? - Look for broad consensus to speak to sustainability 3. How did the EPP program in your organization originate? - Did EPP approach you or were they asked to assist by your organization? 4. What role, if any, did your ministry play in shaping EPP assistance to your ministry? b. When was an EPP advisor seconded to your ministry? - Making decisions about which EPP activities will address the ministry’s needs - Describing EPP advisor’s role in providing support to the ministry 5. Was an official from your ministry assigned to work with the advisor? If no, why not? a. Please describe the EPP advisor’s day to day support provided to your ministry. b. Who supervises this work? - Who did the advisor work with on a day to day basis? - Is the EPP assistance housed in your ministry? - Policy papers developed, assessments and other studies conducted - What else? 6. Could you summarize the work undertaken by your organization with the support of EPP? a. What are the contributions EPP has made to the development and implementation of policy? - E.g, have there been administrative orders issued resulting from EEP support on policy development? - Policy, reforms, capacity building, what else? Development of LNG policy - 6a. Look for number of people trained, including women and men / gender. - Quality of expertise provided and oversight by the ministry 7. Did the approach of EPP support change over the last three years? a. If yes, why and how? Please explain Has the level of support increased or decreased over this period? Effectiveness and Sustainability 8. Please describe studies that EPP conducted for your ministry? a. If no such support has been provided thus far, why not? Policy papers, assessments, studies, etc. 9. Could you explain step by step how your organization develops and implements a policy? Who works on it? How collaborative is the process? 10. What role, if any, do you or ministry’s officials play in designing and conducting these studies/developing policy documents? b. Besides USAID, are there other donor organizations assisting your organization related to energy policy development? If yes, in your opinion, is there adequate coordination among donors supporting the energy sector in Pakistan? 97 Question or Sub-question Prompts or Probes 11. Please explain ways in which the ministry was able to use the support from the EPP advisor. - Examples - Contributions EPP advisor has made to the development and implementation of policy, Mentoring ministry officials, providing technical support in designing studies, and developing policy papers. - Was the support as per expectations? 12. Could you describe some of the EPP support that you consider as an important policy input? Please give examples. State major examples that led to policy input from the EPP provided support 13. Regarding the SIM Power Model, how have you used this model? a. Do you believe the model provides adequate information to assist policy making or modification? a. How effective was the training for the SIM Power Model? Why? How? 14. Speaking of the EPP project broadly, what do you think is the broader effect of the additional MW of power added or restored due to EPP? a. Do you think the EPP’s contributions to the energy supply will be sustained? Why or why not? EQ1.2 [Need answered!] What do you think is the additional amount of MW added to the system nationally? 15. Could you describe what measures your ministry has taken to implement and institutionalize policies developed with the support of EPP? a. What stakeholders did you involve, if any, in the design and implementation of the policies? Is the policy enacted through legislation? Is there an independent agency managing it, or that this is planned? [Speaks to financial sustainability] Were women’s groups consulted? Groups representing the poor? 16. Do you try to measure the effects of policies on different groups like women, children, men or by income level? Recommendations 17. Reflecting back on EPP assistance, what changes do you wish you could have made to improve the assistance effort? [If short on time, skip to 23.] 18. What advice would you give USAID if it were to invest in a future program to help advance energy policy? 98 USAID Energy Policy Program (EPP) Evaluation Interview Protocol for Stakeholders Involved with Transmission and/or Distribution Introduction: I am a consultant with Management Systems International (MSI), a U.S. based company that has been contracted to evaluate USAID funded development projects in Pakistan. MSI is carrying out an evaluation of USAID’s Energy Policy Program (EPP). This interview is being conducted to gain further knowledge and insight into the various activities implemented by the EPP contractor, Advanced Engineering Associates International (AEAI). As a key stakeholder, your input to our assessment is extremely valuable and will enhance our understanding and analysis of the program. Your answers will be used by MSI for data analysis purposes only and will be kept strictly confidential. No informants will be identified by name in the report. Would it be possible for us to record our conversation to support our notes? CHECK Yes- O No - O The recordings will be deleted at the end of the evaluation and will not be shared outside of MSI under any conditions. Can I confirm how much time you have available? Given the short time available and the importance of getting your response to the information we need across a number of areas, in case my question was unclear and leads the discussion in the wrong direction, I can clarify my question. ___________________________________ Organization Respondent Name(s) Title(s) 99 (Sub)Question Prompts Identification 1. How long have you been in your organization and how long involved with the EPP project? a. What is your role with the EPP project and has your role changed over time? - Activities/responsibilities on a day-to-day basis - How long have you served at this position? - Level within the overall policy making hierarchy in the organization 2. What were the issues identified that your utility wanted to address with EPP's assistance? b. Who decided the type of assistance EPP would provide? How was this issue identified and agreed upon? [Looking for inclusion] Preparation and Approval 3. Was there a feasibility study or technical or environmental study undertaken prior to the start of EPP assistance? c. If yes, please document source d. If no, please explain justification Looking for validation of the technical aspects of the Project, the financial costs and expected results 4. What were the major findings from the studies? a. Who approved moving ahead with project assistance after the studies were completed? Looking for good governance indicators Implementation and Effectiveness 5. Could you summarize the work program undertaken by your organization with support of EPP? a. What were the objectives of EPP’s assistance b. What were the investments c. Describe the Project Management Structure (including who supervises the overall project in your facility as well as the individual components) - What was EPP's proposed role in solving the issue addressed earlier? - Just a broad understanding of how the Project will work and the benefits of it. - 6c. Do you have a PMU? 6. TRAINING / CAP BUILD What activities did EPP provide to strengthen the technical capacity of employees? a. Besides training, what other technical support did EPP provide? - This would list the seminar, trainings and knowledge transfer EPP provided - Includes indirect support through capacity building 7. TRAINING / CAP BUILD How many men and women were trained by the Program? h. How were they chosen? i. How many remain in their positions after receiving training? [Look for inclusion of stakeholders in deciding, whether believe “right ones” were chosen, retention] 8. What did staff from your plant learn from EPP? 9. What is the total number of staff in the plant (working at least 30 hours per week) who are male? Females? a. What might increase female staff employment? 10. TRAINING / CAP BUILD Have other elements of the utility/organization/company not directly related to the EPP activities benefitted from the training and capacity built during the EPP program? a. Have other people who have not been directly involved with EPP training and capacity building benefitted from these efforts? 100 (Sub)Question Prompts 11. OVERSIGHT & CAP BUILD Were plant staff involved in overseeing the maintenance or rehabilitation activities of contractors? a. What was the involvement of plant personnel in the program? b. Were any skills transferred to the plants engineers and technical staff on the best international maintenance practices this way? 12. Describe the quality control plan concerning procurement, construction and final product a. Did you encounter any significant delays in procurement processes? Were timely progress reports produced outlining progress to date, delays, issues and other factors effecting the project? 13. Did the approach to the program change over the course of the work? How? Looking for the underlying issues out of the project's control that needed to be overcome. Effectiveness, Sustainability, Recommendations 14. What were the transmission bottlenecks resolved through EPP? 15. What future outcomes will develop from the technical audit of the grid stations? a. Within the 220/500 kV national system b. At the 32/66/133 (PESCO) 16. PESCO: What impact has the specialized equipment and vehicles donated to PESCO had in improving overall operations of the distribution network? 17. Concerning the 132/66 kV live line voltage training, how many people were trained, what was the feedback from the trainees and trainers, how has the training helped in increasing the availability of the system? 18. NPCC: What is impact of the capacity building given to NPCC? Asked to NPCC 19. Data Requests, Physical i. How many Power/Auto transformers were added to the system? j. At what grid stations? k. At what voltage levels? l. Capacity of transformers added? PESCO ONLY BELOW m. How many circuit breakers were replaced at the 132/66/33kV level? n. How many capacitor banks were installed at the 132/66/33 kV level? 20. To what extent has the transformer investment helped in reducing voltage drops, system losses and extending the life of the system and its assets. a. Within the 220/500 kV national system b. At the 132/66/33 (PESCO) 21. To what extent has the circuit breaker investment helped in increasing the capacity of the station and safety concerns. 22. PESCO: To what extent has the telemetering system has help in streamlining the billing and operational practices at PESCO. 101 (Sub)Question Prompts 23. PESCO: To what extent has the capacity building in testing transformer oil improved the efficiency of the 136/66/32 system. 24. To what extent has the capacitors helped in increasing the capacity of the station, and safety concerns. 25. To what extent has the investment in cooling fans improved the performance of the transformers? 26. PESCO: Under the EPP, how many and what type of studies were undertaken at PESCO? Did the studies result in implementation of any improvements? 27. What are the other technical benefits of transmission reinforcement through EPP? a. Within the 220/500 kV national system b. At the 132/66/33 (PESCO) 28. Did the work program with EPP reduce or shorten transmission outages? a. Within the 220/500 kV national system b. At the 132/66/33 (PESCO) Measuring reliability of the investment and capacity building. What types of outages were most affected? 29. Did the work program successfully decrease losses and increase output? a. Within the 220/500 kV national system b. At the 132/66/33 (PESCO) What extent did MWh (production) and Peak Load (MW) improve 30. Was the overall EPP work program successful in achieving its stated objectives? 31. Could you provide examples of outcomes that were unanticipated, both positive and negative? unaccounted for benefits or shortcomings not accounted from the original feasibility study 32. What do you think is the additional amount of MW added to the system nationally due to EPP's efforts? 33. What do you think is the broader effect of the additional MW added due to EPP? 34. Speaking of system capacity, what should be done to ensure capacity is maintained or increased? Within the areas of Technical, financial, political 35. How was the needs and areas for capacity building/training determined? Looking for a good cross section across all departments 36. Does the company have any future activities planned that are an extension of the EPP Program? a. Where is the funding expected to come from? b. Does the future plans coincide with the country's policy towards energy security, private investment and environment concerns? c. Do you anticipate any external issues with future activities? - Maintenance plans, procurement plans, - Plant's resources, government, banks, public-private partnership, donors 37. What efforts are being made to help ensure that you are able to maintain or expand EPP’s benefits without further donor financial support? a. What are the constraints, internal and external, that would require you to continue to rely on donor assistance? Recommendations, Changes 38. Based on reflection, can you think of anything that you wish you had done during EPP implementation to improve and sustain the program’s outcomes? Experience 102 (Sub)Question Prompts a. In your view, would you have changed emphasis between reliability and enhancing transformer capacity? Experience 39. What other development finance institutions besides USAID is assisting you in this work program? 40. For future programs of this type using donor external resources, what recommendations would you have? 103 ANNEX 4: BIBLIOGRAPHY OF DOCUMENTS REVIEWED Dam, S. (2007-2015). Monthly Energy Generation Record. Dam, T. (2006-12, 2015). Tarbela Dam Stakeholder Data. Dr. Kevin Warr, M. M. (2011). EVALUATION OF DRUM AND WENEXA. India: USAID. Energy. (n.d.). Circular Debt Report. Energy. (n.d.). GENCO Business Plans. Energy. (n.d.). Heat Rate Test Report. Energy. (n.d.). PowerSIM User Manual. EPP. (2012). Monthly Report. EPP. (2012, 13, 14). Annual reports. EPP. (2012, 13, 15, 15). Quarterly Report. EPP. (2013,14 & 15). Work Plans. EPP. (2014). Approved EDF-EMMP of EPPs LNG Activities. EPP. (2014). Approved EDF-EMMP of EPPs Transmission Activities. EPP. (2014). Monitoring & Evaluation-Learning Outcomes. EPP Project. EPP. (2014-15). Trainings Evaluation Reports. EPP. (2015). EPP Projects Overview. EPP. (n.d.). Acknowledgment Letters. EPP. (n.d.). Draft Electricity Act 2013-EPP Comments. EPP. (n.d.). Draft Policy Framework for Private Sector Transmission Line Projects. EPP. (n.d.). Energy Policy Program Performance Monitoring Plan. EPP. (n.d.). ENERGY Policy Project RFramework and PIRS. EPP. (n.d.). EPP advisors and Terms of References (TORs). EPP. (n.d.). EPP Contract and Modifications. EPP. (n.d.). Government of Pakistan (GoP) Request Letters. EPP. (n.d.). Government to Government (G2G) Agreements. EPP. (n.d.). MSI EPP Evaluation Due Diligence Table -EPP updates. EPP. (n.d.). Muzzafargarh - Generation & Outage information. GoP. (n.d.). GoP MWP Private Power & Infrastructure Structure Board. 2015. 104 GUDDU. (n.d.). Generation Data Before and After Rehabilitation Under USAID EPP Program. MSI. (2012). USAID Pakistan Gender Analysis & Gender Assessment Of Stabilization Programming. USAID . MSI. (2015). PERFORM Evaluation Checklist. Islamabad. NTDC. (2012-13). Power System Statistics. Pakistan, G. R. (2015). Opinion Poll Energy Economy. Islamabad: Gallup Pakistan. Rahman, A. A. (2010). Final Report for Pre Award Survey of Water and Power Development Authority (WAPDA). Grant Thornton International. Saeed Ahmad, S. A. (n.d.). Executive Exchange on Best Practices in Market Operations. NPCC, NTDC. USAID. (2013). Pakistan: Gender Overview. KSC Research Series. 105 ANNEX 5: LIST OF INDIVIDUALS AND AGENCIES CONTACTED AND PLACES VISITED TABLE 5: LIST OF INDIVIDUALS AND AGENCIES CONTACTED AND PLACES VISITED Sr. # Relevant Stakeholders* Project Location Interviewees Name Interviewees Designation Original Plan Actual Institutions Individuals Institutions Individuals Interview mode: face to face 1 USAID Energy Office Islamabad 1. Mr. Saeed Anwar EPP Contracting Officer Representative 1 6 1 6 2. Ms. Rabia Bukhari Project Management Specialist 3. Mr. Sami Abbasi Office of Acquisition & Assistance 4. Mr. Kamal Ayub Office of Acquisition & Assistance 5. Mr. Atif Haider Office of financial management 6. Mr. Nadeem Habib PDP Contracting Officer Representative 2 AEAI Islamabad 1. Mr. Talha Javed Chief of Party 1 5 1 7 2. Mr. Rufus Kellam Transmission Manager 3. Mr. Akbar Yusuf Director Policy Reforms 4. Mr. Tariq Qazi Senior Advisor Transmission 5. Mr. Ghullam Abbas Senior Advisor Hydro 6. Mr. Saleem Paracha Senior Energy Expert 7. Mr. Ghulam Abbas Malik Senior Energy Expert 3 Ministry of Water and Power Islamabad 1. Mr. Zargham Khan Joint Secretary 2. Mr. Muhammad Farhan Additional Secretary 1 2 1 3 3. Mr. Saulat Section Officer 4 Ministry of Petroleum and Natural Resources (MPNR) Islamabad 1. Mr. Mobin Saulat Managing Director Interstate Gas 1 2 1 1 5 Director Energy Wing, Planning Commission Islamabad 1. Mr. Syed Akhtar Ali Member Energy, Planning Commission 1 1 1 1 6 National Power Control Center (NPCC) Islamabad 1. Mr. Saeed Ahmed Deputy Director Regional Control Center 2. Mr. Saleh Muhammad Chief Engineer Operational Planning 1 3 1 3 3. Mr. Muzammil Director Regional Control Center (North) 106 Sr. # Relevant Stakeholders* Project Location Interviewees Name Interviewees Designation Original Plan Actual Institutions Individuals Institutions Individuals 7 Ministry of Finance Islamabad 1. Mr. Noor Ahmed Senior Joint Secretary, Finance Division 1 1 1 2 2. Mr. Muhammad Anwar Sheikh Joint Secretary 3. Mr. Tassaduq Hussain Jadoon Deputy Secretary (Corporate Finance) 8 EPP Advisors seconded to MWP, NTDC, PESCO, MPNR, MoFP, Islamabad 1. Mr. Muhammad Razi Abbas (Project Development Advisor)-MWP 1 5 1 3 2. Mr. Imran Haider Janjua (Project Development Advisor)-MoF 3. Mr. Shahid Ahmad (IT Specialist)-PC 9 Directorate General of Petroleum Concessions (DGPC) Islamabad 1. Mr. Saeed Ullah Shah Director General 2. Mr. Hassan Mehmood Senior Economist 1 1 1 3 3. Mr. Kashif Ali Abbasi Deputy Director (Technical) 10 IESCO Islamabad DISCO 1. Mr. Ayaz Hussain Superintending Engineer Grid System Operations 1 2 1 1 11 Member Power, WAPDA Lahore 1. Mr. Badar Ul Munir Member Power 1 1 1 1 12 National Transmission and Dispatch Company (NTDC) Lahore 1. Mr. Arshad Chaudhry Managing Director 2. Mr. Rana Wajahat 1 3 1 3 General Manager Grid System Construction 3. Mr. R. S. Rehan General Manager Power Planning 13 LESCO Lahore DISCO 1. Mr. Rao Zameer Ud Din Chief Executive Officer 1 2 1 2 2. Mr. Mehmood Alam Superintending Engineer Grid System Operation 14 Muzaffargarh (GENCO) Muzaffargarh 1. Mr. Masood Ahmad Finance Director 1 3 1 2 2. Mr. Mohammad Akram Additional Director I&C 15 Jamshoro (GENCO) Jamshoro, 1. Mr. Iftikhar Aziz Siddiqui Chief Executive Officer 2. Mr. Rustam ali Ghouri Project Manager 1 3 1 3 3. Mr. Shamsul Arfeen Finance Director 16 Tarbela Dam Haripur 1. Mr. Sher Afzal Khan Chief Engineer 1 2 1 2 2. Mr. Muhammad Farooq Project Engineer 17 Mangla Dam Mirpur (AJK) 1. Mr. Javed Akhtar Project Director 1 2 1 2 2. Mr. Abdul Wahab Project Engineer 107 Sr. # Relevant Stakeholders* Project Location Interviewees Name Interviewees Designation Original Plan Actual Institutions Individuals Institutions Individuals 18 PESCO Peshawar 1. Mr. Hassan Fazil Chief Executive Officer 1 5 1 10 2. Mr. Amjad Khan Chief Engineer Project Management Unit 3. Mr. Niaz Muhammad Khan Chief Engineer Transmission & Grids 4. Mr. Abdul Waheed Superintending Engineer Grid System Operations 5. Mr. Anwar Ul Haq Yousafzai Finance Director 6. Mr. Mian Zahid Quayyum Executive Engineer (EXN)Protection and Instrumentation 7. Mr. Murad Ali Executive Engineer (EXN) Mardan 8. Mr. Zarif Khan Executive Engineer (EXN) Peshawar 9. Mr. Kashif Director Planning 10. Mr. Fida Muhammad Khan GM Technical 19 MEPCO Multan DISCO 1. Mr. Khalid Javed Superintending Engineer Grid System Operations 1 2 1 1 20 GENCO Holding Company 1. Mr. Muhammad Imran Chief Executive Officer 1 1 1 1 21 FESCO Faisalabad 1. Mr. Basharat Superintending Engineer Grid System Operations 1 2 1 2 2. Mr. Rizwan Ali Rao SDO Live Line 22 Asian Development Bank Islamabad 1. Mr. Ehtesham Zafar Khattak Senior Project Officer (Energy) 1 1 23 Alternate Energy Development Board Islamabad 1. Mr. Syed Aqeel Hussain Jafri Director Policy 1 2 2. Mr. Salman Ishaque Malik Director (Bioi-energy) Interview mode: Skype Conversation 24 Gomal Zam Dam South Waziristan 1. Mr. Abdullah Khan Executive Engineer (EXN) 1 2 1 1 25 Satpara Dam Skardu 1. Mr. Muhammad Ijlal Project Director 1 2 1 1 108 Sr. # Relevant Stakeholders* Project Location Interviewees Name Interviewees Designation Original Plan Actual Institutions Individuals Institutions Individuals 26 Guddu Power Plant 1. Mr. Muhammad Khalid Alvi Chief Executive Officer 1 1 1 1 Interview mode: Email 27 Port Qasim Authority Karachi 1. Mr. Asad Abbas Deputy Manager and PQA Engineer for LNG project 1 1 28 Sui Southern Gas Company Ltd. 1. Mr. Nadeem Sarwat Technical Advisor LNG 1 1 29 Pakistan State Oil Karachi 1. Mr. Babar H. Chaudhary GM Corporate Affairs and LNG Team Leader 1 1 Total 24 59 29 68 * Due to the nature of the power sector in Pakistan, the evaluation team adopted the definition of stakeholder generally as the institutional level. When meeting with multiple respondents within one organization, they generally each understood or could speak to some elements of the training but not others. Within hierarchical organizations, many insisted they be interviewed with their supervisor present in part for that reason and in part because they would not have to withhold information until obtaining later permission. Few internal inconsistencies were found within organizations, and, after follow-up to resolve differences, they turned out to be due to lack of knowledge at the level of a person speaking independently. 109 ANNEX 6: EVALUATION TEAM BIOS 1. Team Leader (Expat) shapmail@yahoo.com Marc Shapiro has a PhD in Political Science and a B.A in Political Science and Economics. He has many years of solid experience in leading performance and impact evaluations in Africa, Middle East, and Asia. He has a strong methodological background in both quantitative and qualitative methods including experimental and quasi-experimental designs, survey research, focus groups and semi￾structured interviews. His vast cross-sectoral experience includes climate change, infrastructure, water, energy, knowledge management, capacity building, and environment. His work encompasses contracts with some of the major donors/international organizations including USAID and the UN. 2. Energy Expert (Expat) gbschaeffer@hotmail.com George B. Schaeffer has an MBA and a PhD in Finance. He has strong knowledge of the energy sector in Pakistan. His major areas of work include financial analysis, accounting, structured finance, project evaluation, historical risk assessment, and privatization proposal development. His energy sector experience also focuses on policy and financial aspects of electricity and public utilities. He has worked with International Resources Groups (IRG) and on USAID funded projects including the Power Distribution Project in Pakistan. He is well-versed in Pakistan energy sector regulations. Dr. Schaeffer served as a senior financial advisor within the Empower Project Pakistan in 2009 and to NEPRA for six months in 2011. 3. Energy Expert (Expat) asubbiah@mclean-group.com Anand Subbiah has a Masters’ in Business Administration (Mergers and Acquisitions Finance). He served as researcher in Industrial Energy Systems for three years at Center for Energy and Environmental Studies at Princeton University. Mr. Subbiah has 25 years of experience in energy sector both as an implementer and evaluator. His major work areas include energy market assessment, carbon finance, planning and policy, utility restructuring, tariffs and rate unbundling, energy efficiency project evaluation, and capacity development and training. He has worked for major donors/international organizations including the World Bank, USAID and the Asian Development Bank. Mr. Subbiah has worked in many countries including Afghanistan, Pakistan, and Bangladesh. He will support the evaluation team with data analysis and debriefing on findings and conclusions. 4. Hydro/Thermal Energy Expert (Local) atherzaheer@yahoo.com Zaheer Ahmed Athar has degrees in mechanical and environmental engineering. He has strong knowledge of and work experience (35 years) in the energy sector in different parts of Pakistan. He has worked in the energy sector in various capacities, e.g., engineer, auditor, manager, and evaluator. He also worked on rehabilitation of a thermal plant, received training on gas turbines and gas compressors, served as energy sector procurement manager, worked as auditor, and assessment and evaluation. Mr. Athar previously worked with MSI as a consultant on the Power Distribution Project (PDP) evaluation. 110 5. Energy Expert –Project Management and Policy (Local) husain_babur@yahoo.com Husain Babur has an MBA, an MA in Economics and an Advanced MA certificate in Applied Economics of Law and Regulations. He has strong (30 years) experience in public and private sector development in the areas of energy, infrastructure, and finance in Pakistan. Mr. Babur served as General Director with National Electric Power Regulatory Authority for ten years. Husain steered the privatization of the Karachi Electric Supply Company (KESC). He is a skilled financial analyst, has prepared complex rates of return revenue models, and developed performance based multi-year tariffs for wind and hydro plants. He previously worked with MSI on the review of the Load Data Improvement Project data under MEP. 6. Energy Procurement/Finance Expert fhaye@outlook.com Faisal Haye has a MA in accounting and a BA/BBA in economics and finance. He is an energy sector financial analyst and has worked on economic analysis of impact of donor-driven programs on the energy sector in Pakistan. He has also worked with two DISCOs (Islamabad and Lahore) on preparing demand forecasts and assessing the commercial impact of energy sector interventions on distribution companies. He understands energy transmission and generation, and has completed financial assessments of the impact of transmission and distribution losses on energy sector. One of his tasks when working with PDP was the development of an integrated national power system model (hydro and thermal). With Deloitte, he worked on developing financial models related to Tarbela and Jamshoro GENCO. He understands the energy sector well from the financial point of view. 111 ANNEX 7: KEY FACTORS AFFECTING SUSTAINABILITY The following factors, drawn from across the set of findings and conclusions in the text and presented in bullet format, represent a list of the key factors affecting sustainability of results from the EPP intervention. Positively  The Electricity Reforms from 1994 to 1998 started the movement from a monolithic, centrally-planned system toward more independent generation, transmission and distribution entities. After halting when the GoP breached the PPA agreements demanding lower prices, given the demands by the IMF to cut the public sector spending, reform is collecting momentum again. This includes registration of CPPA as a corporate entity.  In the last three years, increases in energy prices have been achieved. Of the generation plants examined under EPP, the thermal power plants receive enough to cover basic O & M with some excess to fund capital replacement. This represents an improvement.  The cycle of borrowing known as ‘circular debt’ has slowed.  Economic fuel supply will be critical for future power sector development. EPP supported the foundation for the first imports of natural gas, a lower cost and lower GHG-emitting fuel supply.  Under EPP, the initial framework has been established for exploration of shale energy, a source of gas that has contributed to decreasing costs for electricity generation in the United States.  The reduction in the rate of transmissions losses system-wide by 2 percent between 2011 and 2014 contributes positively to the financial sustainability of the system.  Upcoming generation projects listed by the Private Power and Infrastructure Board as in development or at the feasibility stage lean heavily toward hydro-power, which have three important factors; flood control, water for irrigation to improve food supply, and inexpensive marginal costs for power. Negatively Debt and Governance  The circular debt, now at 2.5 billion rupees, continues to grow, even if at lower rates. This crowds-out budgeting for long-term capital repair and replacement, as discussed by stakeholders and evidences that the overall governance of the sector is still lacking, mainly at the distribution level.  Generation, distribution, and transmission companies remain not operating fully independently with regard to purchase, generation, transmission, dispatch and distribution of electricity reducing the ability to take steps to prevent further circular debt accumulation.  Business plans have not received the desired importance. If implemented, they will make the utilities more commercially viable. However, financial constraints limit the ability to make substantial difference in managing the utilities. Fuel Supply, Pricing, and Budgeting  Energy pricing, within generation even and especially for hydro-power, is insufficient to offset losses and to enable power suppliers, power generators and power distribution companies recover their operating and investment costs including long-term capital replacement cost. 112  Low levels of natural gas reserves, declining by 2 percent a year, inhibit thermal generation expansion based on gas. LNG import might offset the reserves though at a higher price.  The fuel supply mix relies too much on expensive fuels like heavy fuel oil, which is more expensive and produces greater greenhouse gas emissions.  Recent progress in broadening fuel supply mix will or would rely on low-quality, high￾emission coal for which supply network is not established, increasing fuel costs in parallel with GoP planning.  Competition among gas fuel users among the energy, transportation, and agricultural sectors limits availability of gas for generation of electricity.  Energy sector budgets remain insufficient to scale up piloted EPP interventions for the reasons indicated above. Transmission and Distribution  As a sector, transmission losses remain high with 24 percent of all electricity generation is not compensated for, fully 18-19 percent higher than the norm for a fully functioning system. This reduces the ability to pay for continuation of USAID-funded achievements.  Based on published statistics, power line losses, power theft and collections at the distribution level are problematic. PESCO remains one of the two most extreme cases.  The pressing need for maintenance and rehabilitation of 132 kV transmission wires, as found at PESCO, contributes to both unsustainability and unplanned outages.  The need for maintenance and rehabilitation of the 220 kV transmission wires, as found at NTDC, also contributes to both unsustainability and unplanned outages.  Although the investment within PESCO by USAID was effective in improving the system from a technical and operation standpoint, it has done nothing to curb the critical area of losses. PESCO is not yet seen as serving as a model of a turnaround DISCO and is not expected yet to be unable to scale up piloted improvements. Human Resources  There remains a lack of appropriate positive and negative incentives, expertise, and continuity in management of public entities. Appropriate human resource processes and approaches are said to be lacking.  The rapid rotation of civil service employees among agencies leads to lack of technical knowledge to continue both typical processes and broader sectoral reform.  Technical professions working in the ministries lack technical knowledge from working in the field and understanding energy as a system. 113 ANNEX 8: PRIVATE POWER AND INFRASTRUCTURE BOARD UPCOMING IPPS TABLE 6: PRIVATE POWER & INFRASTRUCTURE BOARD UPCOMING IPPS Sr. No. Project Sponsor/ Company Name Location Fuel Capacity (MW) Expected COD/ Remarks Year 2017 1 Patrind Hydropower Project Star Hydropower Limited Kunhar River, KP/AJK Hydel 147 Apr-17 - Under Construction Sub Total (2017) 147 Year 2019 2 Gulpur Hydropower project Mira Power Ltd Poonch River/Gulpur, AJK Hydel 100 Oct-19 FC in Process 3 Grange Holdings Power Project Grange Power Limited Arifwala, Punjab Coal 163 2019 4 Sehra Hydropower Project Farab Energy & Water Project, Iran Poonch River, AJK Hydel 130 Dec-19 PPA under negotiation. EPC tariff under NTDC/NEPRA approval Sub Total (2019) 393 Year 2020 5 Karot Hydropower Project Karot Power Company Pvt Ltd Jhelum River, Distt. Rawalpindi, Punjab Hydel 720 Dec-2020 -LOS Issued. Land acquisition in process Sub Total (2020) 720 Year 2021 6 Suki Kinari Hydropower Project S.K Hydro Pvt Ltd Kunhar River/Mansehra, KP Hydel 870 Sep-2021 - FC in process Sub Total (2021) 870 Year 2022 7 Azad Pattan Hydropower Project Alamgir Power Pvt Ltd Jehlum River/Sudhnoti, AJK Hydel 640 Jun-2022 - FS level Tariff determination by NEPRA completed on 16-Oct-2014 8 Chakothi-Hattian Hydropower Project Suhail Jute Mills Ltd Muzaffarabad, AJK Hydel 500 Dec-2022 - FS level Tariff determination by NEPRA completed on 16-Oct-2014 Sub Total (2022) 1140 Year 2023 9 Kohala Hydropower Project China International Water & Electric Company Jhelum River/Kohala, AJK Hydel 1100 Dec-2023 - FS tariff determined/approved by NEPRA Sub Total (2023) 1,100 Project at Feasibility Study Stage 10 Madian Hydropower Project Cherat Cement PAK Swat River, KP Hydel 157 Dec-22: FS Complete. Progress Slow due to Law & Order situation in Swat area 114 Sr. No. Project Sponsor/ Company Name Location Fuel Capacity (MW) Expected COD/ Remarks 11 Asrit-Kedam Hydropower Project Younas Brothers Group Near Kalam/Swat River, KP Hydel 215 Dec-22: FS complete. Progress Slow due to Law & Order situation in Swat area 12 Kaigah Hydropower Project Telecom Valley Pvt Ltd Kaigah/Indus River, KP Hydel 548 Feasibility Study completed. Sponsors are in process of submission of FS level tariff to NEPRA - Dec 2024 13 Mahl Hydropower Project - Jhelum River, AJK/Punjab Hydel 590 Dec-23: LOI issued. FS in progress Sub Total (Project at Feasibility Study Stage) 1,510 Other Projects in Process (Gas and Hydel) 14 Kandra Power Project Kandra Power Company Limited Near Sukkur, Sindh Gas 120 Dec-2022: Gas availability not confirmed 15 Lower Palas Valley Hydropower Project PEDO, WAPDA, K￾Water and Daewoo of Korea Kohistan Valley, KP Hydel 665 COD:Dec-2022 Project under implementation in PPP mode 16 Lower Spat Gah Hydropower Project PEDO, WAPDA, Komipo and Pasco of Korea Kohistan Valley, KP Hydel 496 COD:Dec-2022 Project under implementation in PPP mode 17 Rajdhani Hydropower Project - Poonch River, AJK Hydel 132 Dec-24: Proposals received are Under Evaluation 18 Athmuqam Hydropower Project - Neelum River, AJK Hydel 350 To be advertised shortly 19 Neckeherdim Hydropower Project - YarKun River, Chitral Valley, KP Hydel 80 Project advertised and No Proposal received 20 Turtonas-Uzghor Hydropower Project - Golen Gol River, Chitral Valley, KP Hydel 58 Proposals received are Under Evaluation Sub Total Others(Gas/Hydel) 1,901 MW Other Projects in Process (Coal) 21 1400 MW Thar Coal based Power Project Shanghai Electric Thar Block-I, Sindh Coal 1,400 Issuance of NTP and LOI in process 22 660 MW Thar Coal based Power Project Engro Powergen Limited (EPGL) Thar Block-II, Sindh Coal 660 IA and PPA Signed on 4th May 2015 23 1320 MW Imported coal based Power Project Sinohydro Resources Limited, China / Al Mirqab Capital, Qatar Port Qasim, Karachi Coal 1,320 LOS Issued. IA Signed on 18th April 2015 24 6600 MW Imported coal based power projects at Gadani Pakistan Power Park, Gadani Balochistan Coal 6,600 115 Sr. No. Project Sponsor/ Company Name Location Fuel Capacity (MW) Expected COD/ Remarks 25 1320 MW Imported coal based Power Project at HUB Balochistan Hub Power Company HUB, Balochistan Coal 1,320 Issuance of NTP and LOI in process 26 300 MW Local Coal based Power Project China Machinery Engineering Corporation Salt Rang Pind Dadan Khan, Punjab Coal 300 LOI validity extended till 19-August-2015 27 660 MW Imported coal based Power Projects at Port Qasim Lucky Electric Power Company Limited Port Qasim Karachi Coal 660 LOS Issued on 08-Jun￾2015 28 350 MW Imported coal based Power Projects at Port Qasim Karachi Siddiqsons Limited Port Qasim Karachi Coal 350 LOI Issued; Tariff determination in Process 29 1320 MW Imported coal based Power Projects at Qadarabad Dist Sahiwal Huaneng Shandong Ruyi (Pakistan) Energy (Pvt) Limited Qadarabad, District Sahiwal Coal 1,320 LOS Issued on 17th April 2015 Sub Total Coal 13,930MW GRAND TOTAL 21,711MW Source: (http://www.ppib.gov.pk/N_upcoming_ipps.htm) 116 ANNEX 9: ANALYSIS OF POWER PLANTS ADMINISTRATIVE DATA TABLE 7: INSTALLED CAPACITY AND EFFECTIVE CAPABILITY (MW) Public Sector Thermal and Hydros In Pakistan Year Public sector thermal Public sector Hydros Public sector thermal plus Hydros Installed capacity Effective capability Installed capacity Effective capability Installed capacity Effective capability Summer Winter Summer Winter 2011-12 4841 3635 6516 6770 3797 11357 10405 7432 Percent effective of installed 75% 104% 33% 92% 65% 2012-13 4841 3635 6733 6906 3933 11574 10541 7568 Percent effective of installed 75% 106% 35% 93% 67% 2013-14 5458 4347 6902 7046 4001 12360 11393 8348 Percent effective of installed 80% 108% 35% 100% 74% Percent increase from 2011-12 9.5% 12.3% Note: Fiscal years run from MONTH to MONTH. 117 TABLE 8: RUNNING HOURS SHEET AT JAMSHORO TPS January 2014 Description Unit no 1 Unit no 2 Unit no 3 Unit no 4 Forced outage hours 0 408 0 0 Planned outage hours 96 0 147 0 Total outage hours 96 408 147 0 Outage hours as percent of time available 15% 63% 23% 0% Stand by time ( hours) 122 0 0 230 Standby hours as percent of time available 19% 0% 0% 36% Running hours 524 335 597 514 Running hours as percent of time available 81% 100% 100% 69% Time available for running 646 335 597 744 Period hours (total hours in the month) 744 744 744 744 Notes: Unit No 1 was Available for Operation for 646 Hours but was Run for only 524 Hours Unit No 4 was Available for Operation for 744 Hours but was Run for 514 Hours Only Source - E- Form Statistical Data of Jamshoro Tps for January 2014 118 TABLE 9: PESCO INTERVENTIONS Activity Status of Activity Completion Date Investment Intervention Description Sector Impact Significance within EPP Work Program Stakeholder of Perceived Value Month/Year ( US$ ) (Material, Not, Undetermined) (High, Medium, Low, NA)* Transformer (213 oil) samples analyzed and reported Completed Nov-14 Reduce losses Improve efficiency Material High Oil Testing Training Completed Nov-14 Reduce losses Improve efficiency Material High Real-Time Digital Simulator Completed Feb-15 Operations and Planning Better resource application Material Medium Reactive Power Compensation Study Cancelled Transmission Technical Audit Completed Sep-14 Conducted detailed assessments at each grid station and overhead transmission line Develop rehabilitation plans and Improve operational efficiency Material High Purchased 38 132 kV SF6 Circuit Breakers (CBs) Completed Jun-15 0.22 Reliability and sustainability Increase system performance Material High Telemetering Completed Nov-15 0.06 Installed 124 meters and 56 DCUs Governance Material High Three capacitor banks at Peshawar University, Shahibagh and Chakdarra grid stations Completed Mar-13 0.1 Reliability and sustainability Less outages Material High Shahibagh Grid Station 30/37.5 MVA Power Transformer Repaired Completed May-15 0.3 Improved throughput Less outages Undetermined High Tower repairs Completed Jun-13 0.31 Completed all 15 priority tower repairs Safety and reliability Material High 119 Activity Status of Activity Completion Date Investment Intervention Description Sector Impact Significance within EPP Work Program Stakeholder of Perceived Value Month/Year ( US$ ) (Material, Not, Undetermined) (High, Medium, Low, NA)* Jamrud Grid Station 20/26 MVA transformer Repaired Completed Aug-14 0.15 Improved throughput Less outages Material High Three 132/11.5kV, 40MVA Power Transformers for Jamrud, Hattar and D.I. Khan grid station Completed May-15 2.1 Improved throughput Less outages Material High Installed 348 cooling fans on 47 grid stations Completed Mar-15 0.17 Improved throughput 125 MVA Material High Gadoon Amazai 132 kV Grid Station repaired 20/26MVA power transformer Completed Apr-15 0.16 Improved throughput 13 MVA Material High Procuring two oil filtration plants Ongoing Aug-15 0.3 Reduce Losses Improve Efficiency Undetermined NA Bannu 48 MVA capacitor bank was rehabilitated Completed Dec-14 0.05 Improved Throughput Less outages Material High Fully equipped Protection & Instrumentation testing vans Completed Nov-15 0.5 Fully equipped testing vans for PESCO’s P&I department were handed over to PESCO Efficiency Material High 82 Grid Station technical audits for Pesco Completed Dec-14 Develop O & M and rehabilitation plans Efficiency and increase throughput Material High Procurement of 132kV circuit breakers (20) Completed Sep-15 0.43 Improved Throughput Less outages Material High 120 Activity Status of Activity Completion Date Investment Intervention Description Sector Impact Significance within EPP Work Program Stakeholder of Perceived Value Month/Year ( US$ ) (Material, Not, Undetermined) (High, Medium, Low, NA)* Transmission and Grid Equipment Ongoing Aug-15 0.76 Partial delivery of transmission and grid equipment Efficiency and increase throughput Undetermined NA Cooling Fans batch 2 (129 cooling fans) Completed Feb-15 0.07 Increased capacity 64.2 MVA Material High Circuit breakers, capacitor cells, CT for rehabilitation of capacitor banks at Tall, Bannu and Jahangira GS Completed Apr-15 0.29 Increased capacity Efficiency and increase throughput Material High Augmentation and replacement at two Grid stations (Khawaza Khela and Gadoon Amazai) Completed 0.04 Increased capacity Efficiency and increase throughput Material High Supply, Delivery and installation of 2 new 40 MVA transformers Ongoing Aug-15 1.4 Increased capacity 28 MVA Undetermined NA Provision of 4 bucket trucks Ongoing Jul-15 0.23 Improve workforce efficiency Safety, efficiency and OM Undetermined NA Provision of 2 crane mounted trucks Ongoing Sep-15 0.44 Improve workforce efficiency Safety, efficiency and OM Undetermined NA Grounding sets and PPEs for PESCO Grid stations Completed Jun-15 0.12 Increased capacity Safety and grid protection Material High *Note: If the stakeholder did not discuss the line item specifically as a material part of the EPP-developed intervention, this cell is listed as "NA." Source of Program. EPP Quarterly Reports and interviews with AEAI staff 121 TABLE 10: NTDC SUPPORT Activity Status of Activity Completion Date Intervention Description Sector Impact Significance within EPP Work Program Stakeholder Evaluation of Perceived Value Month/Year (Material, Not, Undetermined) (High, Medium, Low, NA)* 500 and 220 kV NTDC Grid Station technical audits Completed Jun-15 NTDC grid stations (45/48) have been audited Identify losses and line congestion Material High NTDC Business Plan Ongoing Aug-05 Business plan for commercial operating capacity Operational Efficiency Undetermined NA NTDC Performance Contract. Completed Feb-15 Contract outlines performance evaluation parameters based on commercial, financial, human resource, and corporate management categories. Improve operational efficiency Material Medium Standard Operating Procedures Completed Mar-12 Better Operations Improve operational efficiency Material Medium NTDC Human Resources Review. Completed Dec-14 Review of proposals for human resources services vendors review of proposals for human resources services vendors Not Material Low Intermittent Supply Study: Ongoing Aug-15 RE expansion More RE Generation Undetermined NA Contingency Plan for NPCC Completed Mar-15 Operational efficiency Improved Liability Material Medium Embedded advisor to NTDC/NPCC Completed Jun-15 Support reform and development Improved transmission system Material High Integration of Renewable Energy ongoing Aug-15 RE expansion More RE Generation Undetermined NA 122 Activity Status of Activity Completion Date Intervention Description Sector Impact Significance within EPP Work Program Stakeholder Evaluation of Perceived Value Month/Year (Material, Not, Undetermined) (High, Medium, Low, NA)* Enterprise Resource Plan Cancelled Real-Time Digital Simulator Completed Sep-15 Training Improve Planning Material Medium SAN Training Complete Feb-15 Increase computer storage Efficiency Material Medium Intermittent renewable resources integration. Ongoing Aug-15 RE expansion More RE Generation Undetermined NA Load flow study Cancelled Transmission License completed Feb-15 Revised draft license that would recognize the removal of CPPA from NTDC Sector reform Material High PSSCE Training (Software) completed Aug-12 Software training Improve operational efficiency Non Material Low *Note: If the stakeholder did not discuss the line item specifically as a material part of the EPP-developed intervention, this cell is listed as "NA." Source of Program. EPP Quarterly Reports and interviews with AEAI staff 123 TABLE 11: MONITORING AND SUPPORT OF TRANSMISSION PROJECT IMPLEMENTATION Status of Activity Completion Date Investment Intervention Description Sector Impact Significance within EPP Work Program Stakeholder Evaluation of Perceived Value Month/Year (Million USD) (Material, Not, Undetermined) (High, Medium, Low, NA)* Ongoing Sep-15 $1.80 Planning Efficiency Undetermined NA Completed Jun-15 $0.57 Procurement Enhancement Efficiency Material High Completed Feb-15 $0.27 Increase Computer Storage Efficiency Material Medium Completed Jul-14 Decision making and capacity Efficiency within planning Planning Material Medium Cancelled Perform Due Diligence, RFP Ongoing Sep-15 $2.50 180 MVA of Capacity Increase Throughput of Electricity Undetermined NA Completed Apr-15 $0.10 Operational Efficiency Material Medium Ongoing Aug-15 $0.70 180 MVA of Capacity Increase Throughput of Electricity Undetermined NA Ongoing Oct-15 No Cost 320 MVA of Capacity Increase Throughput of Electricity Undetermined NA Completed Nov-13 $0.31 Enhance Metering Governance Material Medium Ongoing Sep-15 $2.25 Improved Efficiency governance Undetermined NA Ongoing Aug-15 $0.40 Repair Increase Throughput of Electricity Undetermined NA 124 TABLE 12: DUE DILIGENCE Activity Planned Investment (Million USD) USAID Component Capacity MW Completion Date Month/Year Intervention Description Sector Impact Stage reached (1 or 2) Outcome / USAID Decision Diamer Basha Multipurpose Dam 11.2 billion Not started 4,500 Not yet started Step 1 Due Diligence Generation and Storage 1 Not funding 84 MW Kurram Tangi $700 84 Not yet started AEAI did due diligence of complete Kurram Tangi project. But USAID decided to finance only one component of the the complete project which is Kaitu weir. Increase generation and irrigation 2 Not funding MW Kaitu Weir (componemet I of Kurram Tangi) $147 81 19 Dec-15 Signing of Activity Agreement is in process with WAPDA. Increase generation and irrigation 2 Co- Financing Refurbishment/Reha bilitation of 1,000 MW Mangla Power Station $400 150 90 Dec-15 Developed Environmental Monitoring and Mitigation Plan, Due Diligence for the whole project Costing 397 Million USD, USAID agreed for unit 5 and 6 costing 150 Million USD, Status Activity agreement and PIL signed. SOW for USAID financing Unit 5 & 6 and auxiliary of all units Generation 2 Co- financing Jinnah $225 96 Jun-13 Rapid Environmental Analysis , Due Diligence Generation 1 Not funding Alai Khwar $208 121 Jun-15 Rapid Environmental Analysis , Due Diligence Generation 1 Not funding Khan Khwar $95 72 Dec-15 Rapid Environmental Analysis , Due Diligence Generation 1 Not funding 125 Activity Planned Investment (Million USD) USAID Component Capacity MW Completion Date Month/Year Intervention Description Sector Impact Stage reached (1 or 2) Outcome / USAID Decision Duber Khwar $290 130 Mar-15 Rapid Environmental Analysis , Due Diligence Generation 1 Not funding Golan Gol $313 36 106 Dec-17 Submitted the Golan Gol due diligence report to USAID, SOW for USAID financing is E&M for powerhouse, Generation 2 Co- financing Naltar-V hydel power project $22 Project Identified Generation Not funding Neelum Jehlum hydropower project $3,000 969 Dec-15 Project Identified Generation Not funding Nandipur Thermal Power Generation Project 329-574 Mar-Jun-15 Project Identified Generation Not funding Bunji Hydropower Project $6,800 7,100 Not yet started Project Identified Generation Not funding 132 kV transmission line connecting Skardu and Gilgit $32 Not yet started Project Identified Dispersal Not funding Partrind Hydropower Project TL $8 7 147 Dec-15 In process of Stage 2 Generation 2 90% Funding Lal Sohanra power Solar plant TL $32 1,000 Stage 1 (not accepted) Increase RE Generation 1 Not funding Jhimpir and Gharo wind power plants TL $43 43 734 In process of activity agreement and project Implementation Bill Increase RE Generation 2 Financing *Note: If the stakeholder did not discuss the line item specifically as a material part of the EPP-developed intervention, this cell is listed as "NA." Source: AEAI 126 TABLE 13: TRAINING AND CAPACITY BUILDING Activity Number Trained Status of Activity Completion Date Intervention Description Survey Results Did not meet expectations Needs improvement Satisfactory Very good Outstanding Men Woman Month/Year Design Center 19 11 Completed Jun-15 Organized a seven￾week classroom based training on contemporary software such as AUTOCAD, Power line, PLS Tower, PLS Pole, SAPS, ARCGIS, SAGSEC, Caisson, and Bentley STAAD-Pro, Bentley Sub-Station and Microsoft Project 2% 9% 33% 48% 8% Internship 76 14 Completed Mar-15 Introduction of prospective students to electricity Post evaluation survey will be taken in August Live Line Training 1 67 0 On-going Aug-15 Organized Line training across all DISCOs Uniformly rated excellent by the participants LNG Certification Ongoing Sep-15 Intensive two week course for training towards Certification No available data Commercial Operations Exchange 7 2 Completed Apr-15 Training included business process improvement, financial management, utility commercial management practices; 0% 6% 30% 59% 5% 127 Activity Number Trained Status of Activity Completion Date Intervention Description Survey Results Did not meet expectations Needs improvement Satisfactory Very good Outstanding Men Woman Month/Year Grid Operations Exchange 5 4 Completed Oct-14 Best practices in transmission operations, maintenance, and planning, including the integration and dispatch of renewable energy into the grid 0% 8% 33% 52% 8% Market Operation Exchange 7 1 Completed Jan-15 to best practices in transmission operations, maintenance, and planning, including the integration and dispatch of renewable energy into the grid, 0% 5% 27% 52% 16% O & M Hydro 56 15 Completed Oct2014, Nov 2014, Dec 2014 successfully delivered first of the three training courses 0% 7% 36% 49% 7% O & M Thermal 60 5 Completed Oct 2014, Nov 2014, Dec 2014 Three training sessions on best practices in hydro operations and maintenance (O&M) 5% 8% 10% 47% 29% Power SIM 14 2 Completed Dec 2013 Jan 2014, Jan 2015 Conducted training sessions for MWP, MPNR and PC 3% 11% 23% 36% 29% 128 Activity Number Trained Status of Activity Completion Date Intervention Description Survey Results Did not meet expectations Needs improvement Satisfactory Very good Outstanding Men Woman Month/Year Storage Area Network (SAN) 6 1 Completed Sep-14 The training technical covered overview of SAN, upgrading, administrating, monitoring, preventive maintenance of Chassis and performance statistics. 1% 1% 15% 60% 23% Power System Analysis using PSS-E software Not Available Survey Averages 0% 8% 26% 51% 17% 1 Actually tally sheets where in Urdu so the summary of comments was used to evaluate. Uniformly it was reported the training was successful and more was needed. Source of Program. EPP Quarterly Reports and interviews with AEAI staff 129 TABLE 14: PLANNING COMMISSION Activity Status of Activity Completion Date Intervention Description Sector Impact Significance within Sector Stakeholder Evaluations Perceived Value Month/Year (Material, Not, Undetermined) (High, Medium, Low, NA)* Natural gas allocation among major users Completed Nov-13 Measure supply and demand of gas Planning Material Medium Impact of the present and growing natural gas and electricity rationing Completed Dec-13 Load management Planning Not Material Low Circular Debt Completed Mar-13 Jointly prepared by EPP and PDP on the request of GoP to USAID. The report identified the root causes of power sector circular debt. Being used by GoP and NEPRA for policy formulation. Material to MoF, not PC NA Model Petroleum Concession Agreements Completed Jul-14 Allows investors to contract for drilling Increase energy capacity Material to MNPR, not PC NA 5 MW Hunza Proposed Projects Ongoing Sep-15 WAPDA proposed projects Increased generation addition and water control Undetermined NA 10 MW Hanzil Proposed Projects Ongoing Oct-15 WAPDA proposed projects Increased generation addition and water control Undetermined NA 20 MW Tormik-II Proposed Projects Ongoing Nov-15 WAPDA proposed projects Increased generation addition and water control Undetermined NA 30 MW Ghowari Ghanche Proposed Projects Ongoing Dec-15 WAPDA proposed projects Increased generation addition and water control Undetermined NA ECC summary on Import of LNG Implementation Completed May-15 Summary of the importation of LNG Increased generation Not Material Low 130 Activity Status of Activity Completion Date Intervention Description Sector Impact Significance within Sector Stakeholder Evaluations Perceived Value Month/Year (Material, Not, Undetermined) (High, Medium, Low, NA)* Energy Chapter for the 11th, 5 Year Development Plan Completed Mar-15 Power SIM forecasting and recommendations concern logic, consistency, and flow. Forecast demand High High Power SIM Training Completed Aug-14 Power SIM has been adopted at the Planning Commission Planning and system expansion HIgh High Implementation of E￾Governance. Ongoing Sep-15 Servers are in process of upgradation and will be followed by training and implementation of E￾Governance System Office Efficiency Undetermined NA Long-Range Energy Alternatives Planning (LEAP) System Cancelled NA Imbedded Advisor Ongoing Sep-15 Capacity Undetermined NA *Note: If the stakeholder did not discuss the line item specifically as a material part of the EPP-developed intervention, this cell is listed as "NA." Source of Program. EPP Quarterly Reports and interviews with AEAI staff 131 TABLE 15: MINISTRY OF FINANCE Activity Status of Activity Completion Date Month/Year Intervention Description Sector Impact Significance within Sector (Material, Not, Undetermined) Stakeholder Evaluations Perceived Value (High, Medium, Low, NA)* Private Sector Hydel Power Generation Fund Completed Jun-12 The proposed funds will provide financing for small and medium size Hydro projects. Will help to generate clean and cheaper energy Material High Time of Use Meter Policy Completed Sep-12 Proposal to stop use of TOU by commercial consumers. (not supported by MoF) The continuation of the policy will help to increase the revenue of DISCO and reduction of load shedding Not material Low Bin Qasim Power Station (K-Electric) supply to CPPA Completed Oct-12 Policy proposal provided purchase of power generated by BQPS by CPPA instead of K-Electric. The proposal was not supported by MoF as the Bin Qasim Power plant was a low efficiency plant and its costly generation would have increased the pool price of CPPA. The additional burden of costly generation was avoided to the consumer of public sector DISCOs Not material Low Proposal for allocation of gas to fertilizer plants Completed Oct-12 Policy provided for allocation of more gas to fertilizer sector instead of power sector. The proposal not supported by MoF. MoF has some concerns, as this would further reduce the gas supply to power sector, adding up to tariff differential subsidy (TDS) Material High 132 Activity Status of Activity Completion Date Month/Year Intervention Description Sector Impact Significance within Sector (Material, Not, Undetermined) Stakeholder Evaluations Perceived Value (High, Medium, Low, NA)* Policy proposals for Tube Wells Consumers Subsidy to Balochistan. Completed Nov-12 Preparation of policy proposal for Balochistan tube well consumers and new formula for sharing of the cost of electricity between Balochistan consumers and GoP/GOB. Deduction in subsidy burden of GoP Not material Low Grant of Distribution License to TESCO by NEPRA Completed Nov-12 Proposal provided grant of DISCO status to TESCO. TESCO granted Distribution License under NEPRA which will bring the entity under NEPRA purview and allow the entity to work independently. Material High MWP proposal to allow IPPs to sell the power to the Bulk Purchaser Completed Dec-12 Policy proposal to allow IPPs to sell power to Bulk Purchaser. The policy to sell power to Bulk Power Consumers (BPCs) is a step forward towards multi buyer model. Material Medium Circular Debt Report Completed Mar-13 Jointly prepared by EPP and PDP on the request of GoP to USAID. The report identified the root causes of power sector circular debt. Being used by GoP and NEPRA for policy formulation. Material High Tariff Differential Subsidy claim of HESCO Completed June, 2013 To work out the actual claim of subsidy submitted by HESCO on the basis of audited accounts. Helped to work out actual additional subsidy claims of HESCO Material High 133 Activity Status of Activity Completion Date Month/Year Intervention Description Sector Impact Significance within Sector (Material, Not, Undetermined) Stakeholder Evaluations Perceived Value (High, Medium, Low, NA)* Settlement of Power Sector Circular Debt Stock Completed June-July 2013 The policy decision to settle the power sector circular debt stock amounting to Rs.480 billion as on 30th June, 2014. The policy intervention helped to improve the financial position of Power Generators and Fuel Suppliers which added up about 1500 MW generation in the National Grid. Material High Transaction Advisory Services Agreement for Turkmenistan￾Afghanistan-Pakistan￾India Pipeline (TAPI) Project. Completed Nov., 2013 Approval of the draft for Transaction Advisory Services Agreement for TAPI project The approval of the draft TASA for TAPI project will facilitate the execution of the project through World Bank. Not material Low Solar Power Generation Projects Draft Security Documents Completed June, 2014 The policy provided more incentives and security for the investment by the private sector in solar power generation and has been supported by MoF. This will help to attract private sector investment in solar power generation project. Material High Governance framework and supply chain for LNG import Completed Feb., 2015 The policy provided for import of LNG from Qatar under G to G arrangement through PSO, SSGCL and SNGPL and prepared in consultation with MoF Will help to provide clean and cheaper fuel for power generation in the country Not material Low Draft Supplemental Agreement pricing mechanism for old gas fields Completed Feb., 2015 The policy framework provided for better pricing mechanism for old gas fields in line with the incentives being provided for new gas fields. To attract more investment in the gas sector Not material Low 134 Activity Status of Activity Completion Date Month/Year Intervention Description Sector Impact Significance within Sector (Material, Not, Undetermined) Stakeholder Evaluations Perceived Value (High, Medium, Low, NA)* Required funding for Neelum Jehlum Hydel project to be sent to MWP Completed Feb., 2015 Scrutinization of additional fund requirement for Neelum Jhelum Hydel Power Project. Funds arranged through local and foreign banks. Material High Draft Power Generation Policy 2015 Completed March, 2015 Policy provides security package for investment in power sector which includes Fiscal and Financial Incentives. The policy will help to attract the investment in power sector. Not material Low Pakistan Petroleum Production Rules 1949 to the grant extension in the existing leases Completed March, 2015 The policy provided for allowing extension in the lease period of the investor who was granted leases under the old policy. The policy provided security / confidence to old investor in Petroleum Sector which will help to make more investment in the energy sector. Not material Low NEPRA Equalization Surcharge Case to Lahore High Court Completed June, 2015 The stay order issued by Lahore High Court was got vacated by the Supreme Court and legal cover was provided to impose the surcharge. This will help to increase the revenue of the DISCOs and will also lead to reduction in circular debt Material High Gas Supply Payment Mechanism for PSO, SSGCL, SNGPL, GENCOs and IPPs Ongoing July-Sept., 2015 Mechanism provides for opening of SBLC by all stake holders or opening of special account for payment of LNG. The approval of the mechanism will provide comfort to LNG supplier. Undetermined NA Low Income Targeted Policy proposals Ongoing Nov-15 Formulation of policy to provide subsidy only to low income groups The targeted subsidy will help to reduce the quantum of electricity subsidy being provided by GoP Undetermined NA 135 Activity Status of Activity Completion Date Month/Year Intervention Description Sector Impact Significance within Sector (Material, Not, Undetermined) Stakeholder Evaluations Perceived Value (High, Medium, Low, NA)* Diagnostic Analysis of the Regulatory Framework for NEPRA. Ongoing Nov-15 The governing law of NEPRA is being reviewed by MoF as per direction of Council of Common Interest (CCI) The diagnostic analysis/ review to be submitted to CCI will help to strengthen the NEPRA oversight mechanism and to improve professional expertise. Undetermined NA DISCO Power Purchase Agreement from on-site IPPs based on interim gas supply Ongoing Nov-15 Policy to facilitate the IPPs who will be interested to invest for power generation through supply of gas on interim gas supply. The supply of gas under this policy will be for a short period up to 2-3 years. The policy formulation is linked with the independent certification of gas reserves of such fields. Undetermined NA Energy Purchase Agreement and Implementation Agreement for IPP Biomass Ongoing Dec-15 The revised EPA and IA standard documents will bring more incentives for the IPPs investing in Biomass/ Bagasse. This will help to attract more investment by private sector in Biomass/ Bagasse Power projects. Undetermined NA Pakistan Energy Efficiency & Conservation Bill 2013 Completed Mar-15 The draft law provided to deal with the policy issues relating to energy efficiency and to streamline the institutional framework required for energy efficiency and energy conservation. The bill provides a uniform law applicable throughout the country for promotion of energy efficiency/conservation and establishment of institution and procedures to ensure effective conservation of energy efficiency and conservation. Material High 136 Activity Status of Activity Completion Date Month/Year Intervention Description Sector Impact Significance within Sector (Material, Not, Undetermined) Stakeholder Evaluations Perceived Value (High, Medium, Low, NA)* Comments on Policy to attract Private Investment in Transmission Line Projects Completed Mar-15 Policy provided for private sector investment in Transmission Sector which was supported by MoF. This would require about US$ 8 billion investment which could not be arranged by NTDC. Will help to construct the high voltage transmission lines for power evacuation and transmission from Diamir-Bhasha, Dasu and other hydro projects, which are in pipeline. Material Medium Comments on the summary for ECC￾Draft Energy Purchase Agreement (EPA) & Draft Implementation Agreement (IA) for Biomass/Bagasse projects in IPP mode Completed Jan-15 The standard Implementation Agreement (IA) and Energy Purchase Agreement (EPA) for Biomass/Bagasse project in IPP mode will help to reduce the time of negotiation for investment in Biomass/Bagasse project. The approval of these documents will standardize the IA & EPA for Biomass/Bagasse project which will bring transparency and clarity for the investor in this sector. Material Medium Incentives on recovery of outstanding amounts and anti-theft campaign in DISCOs approved by the ECC Completed Feb-15 The policy will provide incentives for long outstanding dues and anti￾theft campaign in DISCOs. (June 2015) The policy will help to increase the revenue of DISCOs and reduction in power electricity theft. Not material Low Guidelines for Rationalization of Power Tariff approved by the ECC Completed Apr-15 Policy provided for rationalization of tariff and reduction of subsidy. the policy decision will help to increase the revenue of DISCOs and reduce the burden of GoP Not material Low 137 Activity Status of Activity Completion Date Month/Year Intervention Description Sector Impact Significance within Sector (Material, Not, Undetermined) Stakeholder Evaluations Perceived Value (High, Medium, Low, NA)* Diagnostic Analysis / Review of Regulatory Framework of NEPRA Ongoing Sep-15 Policy provided for rationalization of tariff and reduction of subsidy. the policy decision will help to increase the revenue of DISCOs and reduce the burden of GoP Under terminated NA *Note: If the stakeholder did not discuss the line item specifically as a material part of the EPP-developed intervention, this cell is listed as "NA." Source of Program. EPP Quarterly Reports and interviews with AEAI staff 1, 2 Individual activities rated by Tassaduq Hussain Jadoon, Depurty Secretary (Corporate Finance) 138 TABLE 16: MINISTRY OF PETROLEUM AND NATURAL RESOURCES Activity Status of Activity Completion Date Month/Year Intervention Description Sector Impact Significance within Sector (Material, Not, Undetermined) Stakeholder Evaluations of Perceived Value (High, Medium, Low, NA)* Shale Gas/Oil Resource Assessment Completed Jun-15 Data evaluation of 124 public and private domain wells New energy sources Material Medium Shale Gas/Oil Technology, Infrastructure, Environment and Well Head Costs (Shale Gas/Oil Guidance Document) Ongoing Sep-15 Part of above New energy sources Undetermined NA Downstream Oil Policy Completed Nov-12 Report Submitted to GoP New energy sources Material High Tender Process for Integrated LNG Project Completed Apr-13 Procurement of Gas Regasification Plant New energy sources Material High Tender Process for Retrofit LNG Project Completed Apr-13 Procurement of Gas Infrastructure New energy sources Material High SOWs for LNG terminal construction oversight Completed Apr-14 Procurement New energy sources Material Medium Technical Support to Port Qasim Authority (PQA) for LNG terminal and LNG Specific Services Ongoing Sep-15 Develop port for LNG terminal New energy sources Undetermined NA Commercial and Legal Support for LNG Procurement Ongoing Sep-15 Transaction Advice for procurement of LNG from International Markets New energy sources Undetermined NA LNG Service Agreement (LSA) was signed between SSGC and EETL for Fast Track LNG Import Project Completed Apr-14 Tender contract concluded New energy sources Material High Implementation Agreement (IA) signed between PQA and EETL for Fast Track LNG Import Project Completed Jun-14 Port Agreement to have the LNG Terminal New energy sources Material High 139 Activity Status of Activity Completion Date Month/Year Intervention Description Sector Impact Significance within Sector (Material, Not, Undetermined) Stakeholder Evaluations of Perceived Value (High, Medium, Low, NA)* Full Bridge Mission Simulation (FBMS) study Completed Mar-15 Training Pilot Operators New energy sources Material High Training PQA Pilots for handling Q￾Flex LNG vessels Completed Apr-15 Training Tugboat Operators New energy sources Material High Framework for utilization of stranded gas fields Completed 2014 Draft framework for utilization of stranded gas fields forwarded to MPNR New energy sources Non Material Low Model Supplemental Agreement for PCA (Onshore) Completed Aug-14 Modified Petroleum Concession Agreements to reflect now policy New energy sources Non Material Low Director General Petroleum Concessions (DGPC) Manpower Plan Reorg. Study Completed Mar-15 Human Resources Operational efficiency Material Low Model Supplemental Agreement for PSA (Offshore) Completed Nov-14 Submitted to DGPC for GoP approvals New energy sources Material High Master Sales Purchase Agreement (MSPA) Ongoing Jul-15 Draft Master Sales Purchase Agreement New energy sources Undetermined NA LNG Sales Purchase Agreement (SPA) with Qatargas Ongoing International gas contract with Qatar New energy sources Undetermined NA Direct and Option Agreements under the LNG Services Agreement Conditions Subsequent Completed May-15 Material High 140 Activity Status of Activity Completion Date Month/Year Intervention Description Sector Impact Significance within Sector (Material, Not, Undetermined) Stakeholder Evaluations of Perceived Value (High, Medium, Low, NA)* Support to Port Qasim Authority (PQA). Ongoing Sep-15 Full Mission Bridge Simulation (FMBS) study at Siport21 facilities in Madrid, Spain Capacity building for Port Pilots Undetermined NA Upstream Oil and Gas Implementation Unit Legal assistance. Ongoing Sep-15 Support to DGPC for executing supplemental agreements Energy Reform Undetermined NA Upstream Oil and Gas 18th Constitutional Amendment. Completed Feb-15 Submitted to USAID and DGPC Energy Reform Material High Upstream Advisory Support to DGPC Ongoing Sep-15 Advisory Support to DGPC in commercial, legal, regulatory and financial matters Improve transparency in energy sector Undetermined NA Flare Gas Guidelines Completed May-14 Submitted to DGPC for necessary GoP approvals Environmental Mitigation Assistance Material Medium Formulation of Standard Operating Procedures (SOPs) for effective work flow of the DGPC (Regulator) Completed Apr-15 Submitted to DGPC for implementation Material Medium Recommendation to improve 2012 Petroleum Policy, Low BTU Gas Pricing Policy, Tight Gas Policy and Marginal/Stranded Gas fields Guidelines Completed Apr-15 Submitted to DGPC for implementation Energy Reform Material Medium 141 Activity Status of Activity Completion Date Month/Year Intervention Description Sector Impact Significance within Sector (Material, Not, Undetermined) Stakeholder Evaluations of Perceived Value (High, Medium, Low, NA)* New Rules or way forward for implementing Low BTU Gas Pricing Policy, Tight Gas Policy and Marginal/Stranded Gas fields Guidelines Completed Jul-15 Submitted to DGPC for implementation Energy Reform Material high Potential improvements in current E&P Rules (1986, 2001, 2003 offshore, 2009 and 2013) Ongoing Sep-15 Add transparency to Exploration of domestic energy resources Improve transparency in energy sector Undetermined NA Model Supplemental Agreements for Petroleum Concessions Agreements (PCAs) to avail incentives under Low Btu Gas Pricing Policy, Tight Gas Policy, Marginal/Stranded Gas Fields Guidelines and higher 2012 Policy gas price for 10% incremental gas production from existing fields Completed Jul-15 Submitted to DGPC for implementation New energy sources Material High Improvement of Model Petroleum Concession Agreement and Production Sharing Agreement Ongoing Sep-15 In the process of bringing it to international standards. Improve transparency in energy sector Undetermined NA Model Multi Client Agreement for Survey, Studies and other exploratory work. Ongoing Sep-15 Development of domestic resources New energy sources Undetermined NA Review the existing sub-surface ownership rights under the Constitution. Ongoing Sep-15 Add transparency to Exploration of domestic energy resources New energy sources Undetermined NA 142 Activity Status of Activity Completion Date Month/Year Intervention Description Sector Impact Significance within Sector (Material, Not, Undetermined) Stakeholder Evaluations of Perceived Value (High, Medium, Low, NA)* Technical Study: Uniform methodology for estimation of oil & gas reserves (currently based on different methodologies). Ongoing Sep-15 Add transparency to Exploration of domestic energy resources New energy sources Undetermined NA *Note: If the stakeholder did not discuss the line item specifically as a material part of the EPP-developed intervention, this cell is listed as "NA." Source of Program. EPP Quarterly Reports and interviews with AEAI staff 143 TABLE 17: MINISTRY OF WATER AND POWER Activity Status of Activity Completion date Month/Year Intervention Description Sector Impact Significance within Sector (Material, Not, Undetermined) Stakeholder Evaluation of Perceived Value (High, Medium, Low, NA)* Smart meters financial model Completed Dec-10 Establishing transparency Improve efficiency Not material Low Energy financing mechanisms memo Completed Oct-11 Improving Efficiency Improve efficiency Not material Low Key performance indicators for DISCOS Completed Jul-12 Monitoring Performance Transparency of Power Sector Material Medium Review of the agreements between the GENCOs and NTDC Ongoing Oct-15 Establish transparency between GENCOs and NTDC Reform and transparency of power sector Undetermined NA Power Purchase Agreement between JPCL & CPPA-G Ongoing Sep-15 Establish transparency between GENCOs and NTDC Reform and transparency of power sector Undetermined NA Power Purchase Agreement between NPGCL & CPPA-G Dependent on above Sep-15 Establish transparency between GENCOs and NTDC Reform and transparency of power sector Undetermined NA Fuel supply agreement between GENCOs and PSO Dependent on above Sep-15 Establish transparency between GENCOs and NTDC Reform and transparency of power sector Undetermined NA Business Transfer Agreement Ongoing Aug-15 Revised original unbundling agreements Reform and transparency of power sector Material High 144 Activity Status of Activity Completion date Month/Year Intervention Description Sector Impact Significance within Sector (Material, Not, Undetermined) Stakeholder Evaluation of Perceived Value (High, Medium, Low, NA)* Draft National Power and Subsidy Policy Guidelines, 2014 Completed Jan-14 Establish transparency within the subsidies of electricity Reform and transparency of subsidies Material Medium ECC regarding Allocation of Gas from SARA and Suri Fields to GENCO-II Completed Aug-14 Efficiency of the energy sector Reform and transparency of power sector Material Medium Draft Policy Framework for Private Power Transmission Projects, Ongoing Aug-15 Reform of Power Sector Reform and transparency of power sector Undetermined NA Power Purchase Agreement (PPA) between WAPDA and NTDC Ongoing Aug-15 Establish transparency between WAPDA and NTDC Reform and transparency of power sector Undetermined NA Power SIM Completed Jul-14 Decision making and capacity Efficiency within planning Material Medium Performance Contract between GoP and NTDC Delivered to MWP Dec-14 Improve operations of NTDC Reform and transparency of power sector Undetermined NA Economic Impact of Shutting Down Jamshoro and Muzaffargarh Complete Dec-14 Decisions making based on economic advice Improve overall sector economics Material Medium Policy Analysis on LNG and Gas Allocation Complete Dec-14 Decisions making based on economic advice Improve overall sector economics Not Material Low 145 Activity Status of Activity Completion date Month/Year Intervention Description Sector Impact Significance within Sector (Material, Not, Undetermined) Stakeholder Evaluation of Perceived Value (High, Medium, Low, NA)* Performance Contract between individual GENCOs and the MWP Delivered to MWP Jan-15 Targets were established for Key Performance Indicators (KPIs) against which the performance of GENCOs will be monitored by MWP. Improve efficiency Not Material Low Energy Efficiency and Conservation Bill, 2014 Completed Mar-13 Setting standards for appliances to improve energy efficiency Improve efficiency Not Material Low Electricity Act, 2013 Prepared first and second drafts Ongoing stakeholder process Revising the Bill to reflect Pakistan’s electricity sector Transparency within the legal framework Undetermined NA Circular Debt Paper Complete Jan-15 Examination of sources of economic shortcomings with power sector Power Sector Reform Not Material Low Embedded advisor Continues Sep-15 Internal support Power Sector Reform Material Medium Gas Flaring Policy Draft Flare Gas Policy was sent by DGPC to PPEPCA Dec-12 Contain emission’s from burning flare gas Environmental Risk Mitigation Not Material Low *Note: If the stakeholder did not discuss the line item specifically as a material part of the EPP-developed intervention, this cell is listed as "NA." Source of Program. EPP Quarterly Reports and interviews with AEAI staff 1, 2, Individual activities rated by Muhammad Farhan (Section Officer) and M. Saulat, (Section Officer) 146 TABLE 18: GENERATION TECHNICAL ASSISTANCE Activity Status of Activity Completion Date Month/Year Intervention Description Sector Impact Significance within EPP Work Program (Material, Not, Undetermined) Stakeholder Evaluation of Perceived Value (High, Medium, Low, NA)* Conduct Heat Rate Test for GENCOs Completed Apr-15 Measure Efficiency of Generators Fuel saving of U.S. $75 Million Material High Purchase Heat Rate Testing Equipment Completed Apr-15 Measure Efficiency of Generators Fuel saving of U.S. $75 Million Material High Flow Meters Completed May-15 Measure Efficiency of Generators Fuel saving of U.S. $75 Million Material High Environmental Monitoring and Mitigation Plan (EMMP) Ongoing Sep-15 Examine Environment Impacts of Energy Development Prevent Environmental Damage Undetermined NA Mangla Environmental Monitoring and Audit Report to USAID. EPP submitted Environmental Monitoring Sep-15 Measure Environmental Impacts of Rehabilitation Prevent Environmental Damage Undetermined NA Generation Company Performance Contract January, EPP submitted final contracts to MWP Sep-15 These performance contracts set targets and established key performance indicators for MWP to monitor the GENCOs Improved Generation Efficiency Undetermined NA JPCL Power Purchase Agreement Ongoing Sep-15 Revised PPA that would recognize the complete legal removal of CPPA from within NTDC. Structural Reform Undetermined NA 147 Activity Status of Activity Completion Date Month/Year Intervention Description Sector Impact Significance within EPP Work Program (Material, Not, Undetermined) Stakeholder Evaluation of Perceived Value (High, Medium, Low, NA)* NPCL Power Purchase Agreement Ongoing Sep-15 Revised PPA that would recognize the complete legal removal of CPPA from within NTDC. Structural Reform Undetermined NA JPCL Fuel Supply Agreement Ongoing Sep-15 Revised PPA that would recognize the complete legal removal of CPPA from within NTDC. Structural Reform Undetermined NA *Note: If the stakeholder did not discuss the line item specifically as a material part of the EPP-developed intervention, this cell is listed as "NA." Source of Program. EPP Quarterly Reports and interviews with AEAI staff 148 TABLE 19: IMPORT OF POWER Activity Status of Activity Completion Date Month/Year Intervention Description Sector Impact Significance within EPP Program (High, Medium, Low, NA)* EPP transmission experts visited Kabul Completed Jul-12 Increase low cost power Improved supply Not Material CAR and Afghanistan imported Power Scope of Work Completed Nov-12 Increase low cost power Improved supply Not Material *Note: If the stakeholder did not discuss the line item specifically as a material part of the EPP-developed intervention, this cell is listed as "NA." Source: Quarterly Reports and AEAI Interviews 149 TABLE 20: JAMSHORO POWER PLANT MAXIMUM LOAD ATTAINED DURING JAN. 2014, BY UNIT Max Load Attained Capacity of Unit Month Unit#1 187 250 January Unit#2 175 200 January Unit#3 185 200 January Unit#4 175 200 January Total 722 850 January Source: e-form statistical data of Jamshoro Thermal Power Station (JTPS) for Jan 2014 TABLE 21: LOAD AT JAMSHORO UNITS BEFORE AND AFTER REHABILITATION Units Date Load Before Rehab Date Load After Rehab Unit #1 NA NA Unit#2 2/8/2012 100 20/03/2013 170 Unit#3 30/07/12 100 10/3/2013 170 Unit#4 3/10/2012 100 23/03/2013 170 Total, USAID-funded units 300 510 Source: Daily Log Sheet of Jamshoro Thermal Power Station (JTPS) 150 ANNEX 10: ASSIGNMENT WORK PLAN Performance Evaluation of the Energy Policy Program (EPP) Assignment Work Plan EVL.001 Date: June 5, 2015 151 TABLE OF CONTENTS 1. Assignment Work Plan....................................................................................................... 152 2. AWP Purpose ..................................................................................................................... 152 3. Deliverables........................................................................................................................ 152 4. Anticipated Schedule of Activities ..................................................................................... 153 5. Anticipated Level of Effort ................................................................................................ 154 6. Cost Estimate ..................................................................................................................... 155 7. COR Approval .................................................................................................................... 155 152 1. Assignment Work Plan 2. AWP Purpose This external evaluation is taking place during the last year of the Energy Policy Program’s implementation. The evaluation will serve a dual purpose: (I) to learn to what extent the project’s planned results have been achieved and how sustainable they are; and (II) to inform the design of USAID’s future support of energy sector reforms in Pakistan. The information from this evaluation will assist the USAID/Pakistan Mission in decision-making related to (a) the effectiveness of the EPP activities in supporting the Government of Pakistan’s (GoP’s) energy sector; (b) the type of approach the Mission should adopt in any future assistance to the GoP’s energy sector reforms; and (c) the type and scope of possible future interventions in the energy sector. The Energy Policy Program (EPP) is a multi-year, United States Agency for International Development (USAID)- funded initiative to increase power generation and to improve transmission capacity and reliability. EPP works with selected energy enterprises to assist the GoP sector reform efforts. The program supports the joint goals of the United States Government and the GoP in reforming the power sector and is designed to address Pakistan’s chronic electricity shortage. For more details, please see the attached statement of work (SOW). 3. Deliverables The evaluation will produce the following deliverables:  Team Planning Workshop  Instruments of Individual Interviews  Evaluation Work Plan Assignment Work Plan Number EVL.001 AWP Title Performance Evaluation of the Energy Policy Project (EPP) USAID AWP Manager Saeed Anwar, EPP COR USAID/Pakistan Unit Energy Office MSI/PERFORM Learning Facilitator TBD MSI/PERFORM Technical Manager Jehanzaib Khan Start Date June 1, 2015 End Date August 28, 2015 Total AWP Cost Estimate 153  Methodology Plan  Findings-Conclusion-Recommendations Workshop with USAID  Draft Evaluation Report  Final Evaluation Report  Two-Page Summary of the Evaluation For more information about deliverables, please see the attached SOW. 4. Anticipated Schedule of Activities The anticipated schedule of activities may change based on team members obtaining visas for travel to Pakistan. Activity Location Start Date End Date Review of Documents Home of Record 15-Jun-15 19-Jun-15 Draft of instruments Islamabad 15-Jun-15 24-Jun-15 Draft of initial chapter Islamabad 15-Jun-15 27-Jun-15 Team Planning Workshop Islamabad 22-Jun-15 27-Jun-15 Field Work – Individual Interviews (Team A) Islamabad 29-Jun-15 30-Jun-15 Field Work – Individual Interviews (Team A) Haripur 1-Jul-15 1-Jul-15 Field Work – Individual Interviews (Team A) Mirpur AJK 2-Jul-15 3-Jul-15 Field Work – Individual Interviews (Team A) Islamabad 6-Jul-15 10-Jul-15 Field Work – Individual Interviews (Team B) Islamabad 29-Jun-15 30-Jun-15 Field Work – Individual Interviews (Team B) Peshawar 1-Jul-15 3-Jul-15 Field Work – Individual Interviews (Team B) Jamshoro/Karachi 6-Jul-15 6-Jul-15 Field Work – Individual Interviews (Team B) Islamabad 7-Jul-15 10-Jul-15 Field Work – Individual Interviews (Team C) Islamabad 29-Jun-15 29-Jun-15 Field Work – Individual Interviews (Team C) Lahore 1-Jul-15 3-Jul-15 Field Work – Individual Interviews (Team C) Multan 6-Jul-15 6-Jul-15 Field Work – Individual Interviews (Team C) Muzaffargarh 7-Jul-15 7-Jul-15 Field Work – Individual Interviews (Team C) Islamabad 8-Jul-15 10-Jul-15 Data Analysis Islamabad 13-Jul-15 27-Jul-15 Findings Conclusions and Recommendations Workshop with USAID/Pakistan Islamabad 28-Jul-15 28-Jul-15 Report writing, internal review and revisions, editing and branding Islamabad/International 29-Jul-15 17-Aug-15 USAID/Pakistan Draft Report Review Islamabad 18-Aug-15 24-Aug-15 Final report revisions & submission (and 1-2 page Briefer) Islamabad/International 25-Aug-15 31-Aug-15 154 5. Anticipated Level of Effort Tasks Team Leader (Expat￾STTA) Project Management/ Policy Expert (Expat-STTA) Energy Sector Expert (Expat￾STTA) Hydro/ Thermal Energy Expert (CCN-STTA) Hydro/ Thermal/ Management Advisory Services and Policy (CCN-STTA) Energy Finance and Procurement Expert (CCN￾STTA) Evaluation Manager (STTA) Technical Director/ Advisor (HO Review) Senior Advisor (LTTA) Review of Documents 3 3 2 3 3 3 2 1 Drafting of instruments 3 3 1 3 3 3 2 2 2 International Travel 3 3 3 - - - Team Planning Workshop 5 5 - 5 5 5 5 - 3 Field Work 12 12 - 12 12 12 3 - 1 Data analysis 13 13 13 11 11 11 11 - 5 Initial Findings Debriefing 1 1 1 1 1 1 1 - 1 Report writing 10 6 - 2 2 2 3 - 1 Internal report review and revisions 5 2 - - - - 3 3 3 Total 55 48 20 37 37 37 30 5 17 155 6. Cost Estimate 7. COR Approval [COR to either sign below or indicate “approve” on a return email]. __________________________________________ ________________ Contracting Officer’s Representative (COR) Date Sadia Naseer Khan, or designate 156 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523